Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(WIP) Fix Vagrantfile #757

Closed
wants to merge 9 commits into from
Closed

Conversation

scls19fr
Copy link
Contributor

Closes #756

@scls19fr
Copy link
Contributor Author

When trying to compile XCSoar with this Vagrantfile from a Windows 10 install I get the following rror

$ make TARGET=UNIX
  GET     output/download/boost_1_77_0.tar.bz2
download https://boostorg.jfrog.io/artifactory/main/release/1.77.0/source/boost_1_77_0.tar.bz2
Traceback (most recent call last):
  File "/xcsoar-host-src/./build/download.py", line 15, in <module>
    download_and_verify(*sys.argv[1:])
  File "/xcsoar-host-src/build/python/build/download.py", line 40, in download_and_verify
    os.link(tmp.name, path)
PermissionError: [Errno 1] Operation not permitted: '/xcsoar-host-src/output/download/tmphdxkaw0n' -> './output/download/boost_1_77_0.tar.bz2'
make: *** [build/libboost.mk:14: output/download/boost_1_77_0.tar.bz2] Error 1

also try using

$ sudo make TARGET=UNIX

but it raises same error

@scls19fr
Copy link
Contributor Author

scls19fr commented Oct 20, 2021

My guess (but I may be wrong) is that shared folder (shared between VM and guest OS) is in a NTFS partition (because of Windows 10). And hard link may not be supported.

os.link(tmp.name, path)

I wonder if only source code shouldn't be in the shared folder but all the build process should be outside of this folder.
Any opinion?

@lordfolken
Copy link
Contributor

lordfolken commented Oct 20, 2021

Cross linking from windows is always difficult. Git for example checks out the source on windows with CRLF formatted files. Using then this git to compile inside a shared folder can lead to lots of errors.

This here seems a permission error. So I think a sudo chown -R user:group could help.
My tests from a linux host where successful.

I fixed your commit message.
The make command i did not include, as the original purpose of the vagrant box was to provide an ide to the end user.
There are some scripts that are added to the container(docker) image in https://github.com/XCSoar/XCSoar/tree/master/ide/docker/bin that would probably be useful to have in /usr/local/bin/ as they compile xcsoar with best optimization and most targets.

I would be happy to merge this as it is, if you agree.

@lordfolken
Copy link
Contributor

One noteable difference between this and the container: xcsoar is in /opt/xcsoar inside the container.

@scls19fr
Copy link
Contributor Author

Ok let's merge this.
I will try in an other PR to fix the problem with Windows

@scls19fr
Copy link
Contributor Author

scls19fr commented Oct 20, 2021

Currently trying to change

config.vm.synced_folder "../..", "/xcsoar-host-src"

to

config.vm.synced_folder "../..", "/xcsoar-host-src", type: "rsync", rsync__auto: true, rsync__exclude: ['./build']

When running

> vagrant up
Bringing machine 'XCSoarDevEnv' up with 'virtualbox' provider...
==> XCSoarDevEnv: Checking if box 'generic/debian11' version '3.4.2' is up to date...
==> XCSoarDevEnv: Setting the name of the VM: xcsoar-2110202017
==> XCSoarDevEnv: Clearing any previously set forwarded ports...
==> XCSoarDevEnv: Clearing any previously set network interfaces...
==> XCSoarDevEnv: Preparing network interfaces based on configuration...
    XCSoarDevEnv: Adapter 1: nat
==> XCSoarDevEnv: Forwarding ports...
    XCSoarDevEnv: 22 (guest) => 2222 (host) (adapter 1)
==> XCSoarDevEnv: Running 'pre-boot' VM customizations...
==> XCSoarDevEnv: Booting VM...
==> XCSoarDevEnv: Waiting for machine to boot. This may take a few minutes...
    XCSoarDevEnv: SSH address: 127.0.0.1:2222
    XCSoarDevEnv: SSH username: vagrant
    XCSoarDevEnv: SSH auth method: private key
==> XCSoarDevEnv: Machine booted and ready!
==> XCSoarDevEnv: Checking for guest additions in VM...
==> XCSoarDevEnv: Setting hostname...
==> XCSoarDevEnv: Rsyncing folder: /cygdrive/c/Users/scell/github/scls19fr/XCSoar/ => /xcsoar-host-src
==> XCSoarDevEnv:   - Exclude: [".vagrant/", "./build"]
==> XCSoarDevEnv: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> XCSoarDevEnv: flag to force provisioning. Provisioners marked to run always will still run.

it seems to be compiling now... (even with Windows as guest OS)

$ make TARGET=UNIX
  GET     output/download/boost_1_77_0.tar.bz2
https://boostorg.jfrog.io/artifactory/main/release/1.77.0/source/boost_1_77_0.tar.bz2 https://sourceforge.net/projects/boost/files/boost/1.77.0/boost_1_77_0.tar.bz2/download fc9f85fc030e233142908241af7a846e60630aa7388de9a5fafb1f3a26840854 ./output/download
download https://boostorg.jfrog.io/artifactory/main/release/1.77.0/source/boost_1_77_0.tar.bz2
  UNTAR   boost_1_77_0.tar.bz2
Applying patch /xcsoar-host-src/lib/boost/patches/no_locale.diff
Applying patch /xcsoar-host-src/lib/boost/patches/ublas_no_serialization.patch
Applying patch /xcsoar-host-src/lib/boost/patches/no_math_throw
Applying patch /xcsoar-host-src/lib/boost/patches/geometry_missing_include
Now at patch /xcsoar-host-src/lib/boost/patches/geometry_missing_include
  CXX     output/UNIX/dbg/src/io/MapFile.o
  CXX     output/UNIX/dbg/src/io/ConfiguredFile.o
  CXX     output/UNIX/dbg/src/io/DataFile.o
  CXX     output/UNIX/dbg/src/Airspace/ProtectedAirspaceWarningManager.o
  CXX     output/UNIX/dbg/src/Airspace/ActivePredicate.o
  CXX     output/UNIX/dbg/src/Task/Serialiser.o
  CXX     output/UNIX/dbg/src/Task/Deserialiser.o
  CXX     output/UNIX/dbg/src/Task/SaveFile.o
  CXX     output/UNIX/dbg/src/Task/LoadFile.o
  CXX     output/UNIX/dbg/src/Task/TaskFile.o
  CXX     output/UNIX/dbg/src/Task/TaskFileXCSoar.o
  CXX     output/UNIX/dbg/src/Task/TaskFileIGC.o
  CXX     output/UNIX/dbg/src/Task/TaskFileSeeYou.o
  CXX     output/UNIX/dbg/src/Task/DefaultTask.o
  CXX     output/UNIX/dbg/src/Task/MapTaskManager.o

Syncing folder can be done using:

vagrant rsync-auto

@scls19fr
Copy link
Contributor Author

Well compile doesn't work as expected

$ make TARGET=UNIX
  GET     output/download/boost_1_77_0.tar.bz2
https://boostorg.jfrog.io/artifactory/main/release/1.77.0/source/boost_1_77_0.tar.bz2 https://sourceforge.net/projects/boost/files/boost/1.77.0/boost_1_77_0.tar.bz2/download fc9f85fc030e233142908241af7a846e60630aa7388de9a5fafb1f3a26840854 ./output/download
download https://boostorg.jfrog.io/artifactory/main/release/1.77.0/source/boost_1_77_0.tar.bz2
  UNTAR   boost_1_77_0.tar.bz2
Applying patch /xcsoar-host-src/lib/boost/patches/no_locale.diff
Applying patch /xcsoar-host-src/lib/boost/patches/ublas_no_serialization.patch
Applying patch /xcsoar-host-src/lib/boost/patches/no_math_throw
Applying patch /xcsoar-host-src/lib/boost/patches/geometry_missing_include
Now at patch /xcsoar-host-src/lib/boost/patches/geometry_missing_include
  CXX     output/UNIX/dbg/src/io/MapFile.o
  CXX     output/UNIX/dbg/src/io/ConfiguredFile.o
  CXX     output/UNIX/dbg/src/io/DataFile.o
  CXX     output/UNIX/dbg/src/Airspace/ProtectedAirspaceWarningManager.o
  CXX     output/UNIX/dbg/src/Airspace/ActivePredicate.o
  CXX     output/UNIX/dbg/src/Task/Serialiser.o
  CXX     output/UNIX/dbg/src/Task/Deserialiser.o
  CXX     output/UNIX/dbg/src/Task/SaveFile.o
  CXX     output/UNIX/dbg/src/Task/LoadFile.o
  CXX     output/UNIX/dbg/src/Task/TaskFile.o
  CXX     output/UNIX/dbg/src/Task/TaskFileXCSoar.o
  CXX     output/UNIX/dbg/src/Task/TaskFileIGC.o
  CXX     output/UNIX/dbg/src/Task/TaskFileSeeYou.o
  CXX     output/UNIX/dbg/src/Task/DefaultTask.o
  CXX     output/UNIX/dbg/src/Task/MapTaskManager.o
  CXX     output/UNIX/dbg/src/Task/ProtectedTaskManager.o
  CXX     output/UNIX/dbg/src/Task/FileProtectedTaskManager.o
  CXX     output/UNIX/dbg/src/Task/RoutePlannerGlue.o
  CXX     output/UNIX/dbg/src/Task/ProtectedRoutePlanner.o
  CXX     output/UNIX/dbg/src/Task/TaskStore.o
  CXX     output/UNIX/dbg/src/Task/TypeStrings.o
  CXX     output/UNIX/dbg/src/Task/ValidationErrorStrings.o
  CXX     output/UNIX/dbg/src/RadioFrequency.o
  CXX     output/UNIX/dbg/src/Engine/Navigation/TraceHistory.o
  CXX     output/UNIX/dbg/src/Engine/Navigation/Aircraft.o
  CXX     output/UNIX/dbg/src/Engine/Trace/Point.o
  CXX     output/UNIX/dbg/src/Engine/Trace/Trace.o
  CXX     output/UNIX/dbg/src/Engine/Trace/Vector.o
  CXX     output/UNIX/dbg/src/Engine/Util/Gradient.o
  CXX     output/UNIX/dbg/src/Engine/ThermalBand/ThermalBand.o
  CXX     output/UNIX/dbg/src/Engine/ThermalBand/ThermalSlice.o
  CXX     output/UNIX/dbg/src/Engine/ThermalBand/ThermalEncounterBand.o
  CXX     output/UNIX/dbg/src/Engine/ThermalBand/ThermalEncounterCollection.o
  CXX     output/UNIX/dbg/src/HorizonWidget.o
  CXX     output/UNIX/dbg/src/Renderer/TextRowRenderer.o
  CXX     output/UNIX/dbg/src/Renderer/TwoTextRowsRenderer.o
  CXX     output/UNIX/dbg/src/Renderer/HorizonRenderer.o
  CXX     output/UNIX/dbg/src/Renderer/GradientRenderer.o
  CXX     output/UNIX/dbg/src/Renderer/GlassRenderer.o
In file included from ./src/ui/canvas/opengl/Scissor.hpp:28,
                 from src/Renderer/GlassRenderer.cpp:29:
./src/ui/canvas/opengl/Globals.hpp:37:10: fatal error: glm/fwd.hpp: No such file or directory
   37 | #include <glm/fwd.hpp>
      |          ^~~~~~~~~~~~~
compilation terminated.
make: *** [build/compile.mk:108: output/UNIX/dbg/src/Renderer/GlassRenderer.o] Error 1

Did I missed something?

@lordfolken
Copy link
Contributor

You are missing the submodules: (other git repos references from inside xcsoar's git) use the recursive option when cloning.

git clone --recursive git://github.com/XCSoar/XCSoar

@lordfolken
Copy link
Contributor

I think you want to exclude ./output

@scls19fr
Copy link
Contributor Author

scls19fr commented Oct 20, 2021

I think if I use rsync instead of VirtualBox shared folder I don't need to exclude any directory because source will be synced from host to guest at vagrant up (and only this way as it doesn't seems to be multidirectional sync according to https://www.vagrantup.com/docs/synced-folders/rsync )
So output directory can be created and have symlink on guest side without hassle because it's not really shared with the host.

So I don't really know currently what is the best solution... remove rsync__exclude or using rsync__exclude: ['./output']

My main issue being that creating this box is very long!

@scls19fr
Copy link
Contributor Author

scls19fr commented Oct 20, 2021

The bad side of this is that running a command line will be required to sync from host to guest

The rsync and rsync-auto commands can be used to force a resync and to automatically resync when changes occur in the filesystem. Without running these commands, Vagrant only syncs the folders on vagrant up or vagrant reload.

So this can be done either by doing

vagrant rsync-auto

or

vagrant reload

@scls19fr
Copy link
Contributor Author

During the end of provisioning we can see

    XCSoarDevEnv: Cloning into 'xcsoar-src'...
    XCSoarDevEnv: done.
    XCSoarDevEnv: Submodule 'android/UsbSerial' (git://github.com/XCSoar/UsbSerial) registered for path 'android/UsbSerial'
    XCSoarDevEnv: Submodule 'android/ioio' (git://github.com/xcsoar/ioio) registered for path 'android/ioio'
    XCSoarDevEnv: Submodule 'lib/glm' (git://github.com/g-truc/glm) registered for path 'lib/glm'
    XCSoarDevEnv: Cloning into '/home/vagrant/xcsoar-src/android/UsbSerial'...
    XCSoarDevEnv: Cloning into '/home/vagrant/xcsoar-src/android/ioio'...
    XCSoarDevEnv: Cloning into '/home/vagrant/xcsoar-src/lib/glm'...
    XCSoarDevEnv: Submodule path 'android/UsbSerial': checked out '461648295bb5937505d64d7fbead7a7e4632ba34'
    XCSoarDevEnv: Submodule path 'android/ioio': checked out 'ecb3a8522efe7ee73bac0eb3f4f5743a8be2c439'
    XCSoarDevEnv: Submodule path 'lib/glm': checked out '416fa93e42f8fe1d85a93888a113fecd79e01453'

I think I didn't forget submodules but I'm still getting this fatal error: glm/fwd.hpp when running

make TARGET=UNIX

in the /xcsoar-host-src folder

and this folder is different from ~/xcsoar-src

my problem was that I was probably confusing both...

and I didn't wrote

git submodule update --init --recursive

in my XCSoar git repository on the host

@lordfolken
Copy link
Contributor

What is your use-case exactly?

The vagrant container was mainly intended to be used as ssh box to edit and compile xcsoar in. If you install X or use the FrameBuffer you can also execute it inside.

@lordfolken
Copy link
Contributor

If you edit in your own ide and just need a compiler, the docker approach would fit better.

@scls19fr
Copy link
Contributor Author

scls19fr commented Oct 21, 2021

I would like to be able to edit source from host, compile on the guest (which works fine now) but also run GUI on the guest.
So that last part still need some work (installing X). In such a use case Vagrant is probably a better approach than Docker.

PS : to display GUI (with appropriate keymap) I did on the guest side:

sudo tasksel install xfce-desktop
sudo dpkg-reconfigure keyboard-configuration

PS2 : it can be interesting to have VirtualBox Guest Additions installed
it can be done by following this (on the guest side)
https://linuxize.com/post/how-to-install-virtualbox-guest-additions-on-debian-10/

or on the host side you can simply install a Vagrant plugin like so

vagrant plugin install vagrant-vbguest

@scls19fr
Copy link
Contributor Author

scls19fr commented Oct 21, 2021

Not sure rsync__exclude: ['./output'] is the correct value because I lost my output folder on the guest OS.

Steps to reproduce the problem

On guest:

cd /xcsoar-host-src
make TARGET=UNIX
CTRL + C quickly to avoid the long compilation process
ls -l output

shows download directory inside output directory

On host

cd ide/vagrant
vagrant rsync-auto

back to guest

vagrant@XCSoarDevEnv:/xcsoar-host-src$ ls -l output/
ls: cannot access 'output/': No such file or directory

output directory doesn't exist anymore (which is a bit annoying given time necessary to compile)

I also tried rsync__exclude: ['../../output'] but it doesn't help (I only did a vagrant reload maybe vagrant destroy and vagrant up are required to take care of this parameter change.

@scls19fr
Copy link
Contributor Author

I can confirm that rsync__exclude: ['../../output'] doesn't work also (even when destroying and rebuilding a new VM).

Maybe an other workaround to avoid output directory being lost when rsyncing could be to create an output directory inside guest VM and create a symbolic link:

On the guest

mkdir ~/output
cd /xcsoar-host-src
ln -s ~/output/ output

@scls19fr
Copy link
Contributor Author

scls19fr commented Oct 22, 2021

I didn't noticed previously in https://xcsoar.readthedocs.io/en/latest/build.html TARGET_OUTPUT_DIR parameter which could be a good workaround

make TARGET=UNIX TARGET_OUTPUT_DIR=~/output

with such a solution, maybe normal synced folder could be used (not necessarily rsync shared folder)

@scls19fr scls19fr changed the title Fix Vagrantfile (WIP) Fix Vagrantfile Oct 22, 2021
@scls19fr
Copy link
Contributor Author

Unfortunately the build system doesn't seems to take account of this parameter for download

vagrant@XCSoarDevEnv:/opt/xcsoar$ make TARGET=UNIX TARGET_OUTPUT_DIR=~/output
  GET     output/download/boost_1_77_0.tar.bz2
download https://boostorg.jfrog.io/artifactory/main/release/1.77.0/source/boost_1_77_0.tar.bz2
Traceback (most recent call last):
  File "/xcsoar-host-src/./build/download.py", line 15, in <module>
    download_and_verify(*sys.argv[1:])
  File "/xcsoar-host-src/build/python/build/download.py", line 40, in download_and_verify
    os.link(tmp.name, path)
PermissionError: [Errno 1] Operation not permitted: '/xcsoar-host-src/output/download/tmpwm2zudfn' -> './output/download/boost_1_77_0.tar.bz2'
make: *** [build/libboost.mk:14: output/download/boost_1_77_0.tar.bz2] Error 1

@scls19fr
Copy link
Contributor Author

scls19fr commented Oct 22, 2021

According to build/libboost.mk and build/dirs.mk

make TARGET=UNIX OUT=~/ouput

should work

@scls19fr
Copy link
Contributor Author

I'm facing an other problem using OUT parameter...

vagrant@XCSoarDevEnv:/opt/xcsoar$ make TARGET=UNIX OUT=~/output
  CXX     /home/vagrant/output/UNIX/dbg/src/ResourceLoader.o
In file included from src/ResourceLoader.cpp:46:
/home/vagrant/output/UNIX/include/resource_data.h:26:1: error: could not convert '{resource_IDR_FAIL}' from '<brace-enclosed initializer list>' to 'ConstBuffer<void>'
   26 | };
      | ^
      | |
      | <brace-enclosed initializer list>
/home/vagrant/output/UNIX/include/resource_data.h:26:1: error: could not convert '{resource_IDR_INSERT}' from '<brace-enclosed initializer list>' to 'ConstBuffer<void>'
/home/vagrant/output/UNIX/include/resource_data.h:26:1: error: could not convert '{resource_IDR_REMOVE}' from '<brace-enclosed initializer list>' to 'ConstBuffer<void>'
/home/vagrant/output/UNIX/include/resource_data.h:26:1: error: could not convert '{resource_IDR_WAV_BEEPBWEEP}' from '<brace-enclosed initializer list>' to 'ConstBuffer<void>'
/home/vagrant/output/UNIX/include/resource_data.h:26:1: error: could not convert '{resource_IDR_WAV_CLEAR}' from '<brace-enclosed initializer list>' to 'ConstBuffer<void>'
/home/vagrant/output/UNIX/include/resource_data.h:26:1: error: could not convert '{resource_IDR_WAV_DRIP}' from '<brace-enclosed initializer
list>' to 'ConstBuffer<void>'
/home/vagrant/output/UNIX/include/resource_data.h:26:1: error: could not convert '{resource_IDR_FAIL}' from '<brace-enclosed initializer list>' to 'ConstBuffer<void>'
/home/vagrant/output/UNIX/include/resource_data.h:26:1: error: could not convert '{resource_IDR_INSERT}' from '<brace-enclosed initializer list>' to 'ConstBuffer<void>'
/home/vagrant/output/UNIX/include/resource_data.h:26:1: error: could not convert '{resource_IDR_REMOVE}' from '<brace-enclosed initializer list>' to 'ConstBuffer<void>'
/home/vagrant/output/UNIX/include/resource_data.h:26:1: error: could not convert '{resource_IDR_WAV_BEEPBWEEP}' from '<brace-enclosed initializer list>' to 'ConstBuffer<void>'
/home/vagrant/output/UNIX/include/resource_data.h:26:1: error: could not convert '{resource_IDR_WAV_CLEAR}' from '<brace-enclosed initializer list>' to 'ConstBuffer<void>'
/home/vagrant/output/UNIX/include/resource_data.h:26:1: error: could not convert '{resource_IDR_WAV_DRIP}' from '<brace-enclosed initializer
list>' to 'ConstBuffer<void>'
make: *** [build/compile.mk:108: /home/vagrant/output/UNIX/dbg/src/ResourceLoader.o] Error 1

@scls19fr
Copy link
Contributor Author

After running

make -j 4 TARGET=UNIX OUT=~/output

I'm currently facing

make: *** No rule to make target 'output/data/COPYING.gz', needed by '/home/vagrant/output/data/COPYING.gz.c'.  Stop.
make: *** Waiting for unfinished jobs....

I have no idea why

@lordfolken
Copy link
Contributor

I have never used the OUT parameter. I would expect a lot of failures there...
The rsync_auto is exactly that. So the vm will synchronize any changes on the guest back to the host as they occur. However as far as i understand only guest->host not the other way round.

I would go with a share mount (smb) for example. That way you have 1 state and forget about the sync issues.

@scls19fr
Copy link
Contributor Author

The problem with sync (even rsync) is that it is on folder basis... and so even if xcsoar repository on host doesn't have an ouput folder, and this folder is create on guest side, it will be completly lost when halting and running again a vm.
That's the problem with the current setting with output in "standard" ouput folder... that's why using OUT parameter maybe a workaround

Unfortunately when changing OUT parameter value, I haven't been able to compile XCSoar.

So the workaround can be done by using symbolic link from /opt/xcsoar/output to for example ~/output.

When synchronization using rsync occurs (from host to guest), /opt/xcsoar/output symbolic link is broken but fortunately content of ~/output is not lost (and so compilation can be quicker!)

@MaxKellermann
Copy link
Contributor

I have no idea what this is about, but there hasn't been any progress in nearly two years.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants