WIP: Vagrant

[clbarnes]

Proof of concept of dev environment setup in a VM using vagrant. This eases development on different operating systems, and isolates the database and OS-level dependencies. The host and guest can both transparently edit the code; such changes are persisted in both.

Tools like VScode and PyCharm can connect to remotes over SSH, use the remote python environment etc.

To do

• Ideally, refactor to share at least some scripts with travis and/or docker
• Add optional dependencies (R, hdf5 etc.)
• Install pgAdmin and expose its port
  • Despite being a webapp, it depends on Qt and tries to connect to a display?!

Issues

• interaction with git should be done on the host, not the guest. The host likely has a more comprehensive global gitignore, and obviously their user name/ email set up. Ideally we'd uninstall git in the VM to prevent people from accidentally committing there, but IIRC CATMAID expects to have the git history.
• I've kept the virtual environment out of the source directory, but node_modules will end up in the source directory, possibly mismatching the host's own version.

Notes

• virtualenvwrapper wasn't getting sourced properly, so I went with venv, which is meant to be built-in, but isn't in the ubuntu package.

Queries

• As this is only for devs, should I setup the database and user as well? Or would that get in the way of loading other test data, like L1 backups?

[aschampion]

We should fix the 18.04 deps and switch to that. No point in creating up a bunch of dev envs with < 1 year of service life left.

[clbarnes]

interaction with git should be done on the host, not the guest.

I'm now writing pre-commit and pre-rebase hooks (do we need pre-merge as well?) which check that the user is not called vagrant and inform users that they can have a configuration local to the source directory, or commit from their host, or set up their git config in the VM (all should work). The hooks aren't written if those hooks already exist.

Turns out git 2.8+ has an option to prevent commits if the user and email isn't configured, much better.

[clbarnes]

The latest version installs R and sets up a directory for the userspace package library, but doesn't install the R packages, because they take a very long time and I believe CATMAID itself can take care of that bit. I should add the compile-time dependencies of those packages, though.

[clbarnes]

May be some space savings to be had by trying a different box (base image): there are a bunch for ubuntu.

• Hashicorp: makers of vagrant, as official as it gets. "Streamlined" box, which means it may be missing some tools.
• Bento: other than hashicorp, the only officially recommended box maker. More fully-featured. Only works with virtualbox.
• generic: what I'm using at the moment, because it has the most providers (VM backends). However, it seems to be broken for 18.04 (proxy settings don't allow access to certain PPAs)
• ubuntu: canonical's boxes. Only works with virtualbox. Hosted on a potato (~100k download speed).

[clbarnes]

The major remaining issue is a big one: I can't access the django dev server. The port forwarding is working (tested with python -m http.server), so I'm not sure what's going on. Maybe the dev server doesn't like being accessed from another machine? But there's no log output to suggest it's blocking a connection.

[clbarnes]

This is pretty much done besides the dev server connection issue, which arguably is quite important. It's in a state to be checked out, if anyone wanted to weigh in on that.

[clbarnes]

If the dev server is started with manage.py runserver [::]:8000, and the host machine is pointed at 0.0.0.0:8888 (8000->8888 is the forwarding scheme in the Vagrantfile), it's accessible from outside. However, initial load takes a long time, and produces some errors in the console:

Starting ASGI/Channels version 2.4.0 development server at http://[::]:8000/
Quit the server with CONTROL-C.
HTTP GET / 200 [23.29, ::ffff:10.0.2.2:60314]
Forbidden: /client/datastores/settings/
HTTP GET /client/datastores/settings/ 403 [0.04, ::ffff:10.0.2.2:60870]
Forbidden: /client/datastores/settings/

then a bunch more 403s for the same endpoint. Then

HTTP GET /dataviews/show/2 200 [0.03, ::ffff:10.0.2.2:60872]
Exception inside application: 'NoneType' object has no attribute 'group_add'
  File "/home/vagrant/catmaid-env/lib/python3.6/site-packages/channels/sessions.py", line 183, in __call__
    return await self.inner(receive, self.send)
  File "/home/vagrant/catmaid-env/lib/python3.6/site-packages/channels/middleware.py", line 41, in coroutine_call
    await inner_instance(receive, send)
  File "/home/vagrant/catmaid-env/lib/python3.6/site-packages/channels/consumer.py", line 62, in __call__
    await await_many_dispatch([receive], self.dispatch)
  File "/home/vagrant/catmaid-env/lib/python3.6/site-packages/channels/utils.py", line 51, in await_many_dispatch
    await dispatch(result)
  File "/home/vagrant/catmaid-env/lib/python3.6/site-packages/asgiref/sync.py", line 244, in __call__
    return await asyncio.wait_for(future, timeout=None)
  File "/usr/lib/python3.6/asyncio/tasks.py", line 339, in wait_for
    return (yield from fut)
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/vagrant/catmaid-env/lib/python3.6/site-packages/channels/db.py", line 14, in thread_handler
    return super().thread_handler(loop, *args, **kwargs)
  File "/home/vagrant/catmaid-env/lib/python3.6/site-packages/asgiref/sync.py", line 277, in thread_handler
    return func(*args, **kwargs)
  File "/home/vagrant/catmaid-env/lib/python3.6/site-packages/channels/consumer.py", line 105, in dispatch
    handler(message)
  File "/home/vagrant/catmaid-env/lib/python3.6/site-packages/channels/generic/websocket.py", line 39, in websocket_con
nect
    self.connect()
  File "/CATMAID/django/applications/catmaid/consumers.py", line 22, in connect
    async_to_sync(self.channel_layer.group_add)(get_user_group_name(user.id), self.channel_name)
  'NoneType' object has no attribute 'group_add'
WebSocket DISCONNECT /channels/updates/ [::ffff:10.0.2.2:60882]

EDIT:

Or starting the server with 0.0.0.0:8000 and navigating to 127.0.0.1 on the host, also works. Same errors though. Maybe I need to forward some more ports for the websockets? Although those ports seem to be the same ones used when picking up the static files: that works fine.

[clbarnes]

Ubuntu 18.04 branch is here - for now, it just strips imagemagick out (although the graphicksmagick stack and pgmagick are still in there taking 10 minutes to build...). Seems to be an issue with one of the npm dependencies now, though.

[clbarnes]

Rebased on dev, which removes *magick and therefore significantly speeds up builds.

18.04 is behaving very weirdly with npm dependencies. It seems inconsistent between VM re-provisioning, and the exact same node/npm versions work on 16.04 and fail on 18.04. Some of the dependency issues can be worked around by npm installing the failing 2nd order dependency, then the dependency we actually need, but doing this for a rotating set of dependencies for every install is obviously not working. The advice in various forum threads/ github issues is just to delete node_modules and the package-lock.json: that doesn't seem to work in this instance. My proposal is to get it working on a 16.04 image, which at least is consistent with docker and travis. When we update those, we can update docker as well: the presence or absence of these problems in different 18.04 containers may be diagnostically helpful.

Still to do:

• Improve docs with basic vagrant command usage
• Clarify unix directory scheme, as users may not necessarily be using unix themselves
• Document that a lot of stderr output is fine (echoing commands from set -v)
• Forward the port on which docs are served locally

[clbarnes]

Tom has been heroically debugging the npm issue into the night and it seems like there's some issue regarding node_modules living on the host, and virtualbox [1,2]

[1] laravel/homestead#1239
[2] https://www.virtualbox.org/ticket/18776

In the 1804 branch, Tom has solved this by using another overlayfs to ensure that node_modules only lives on the guest, which solves another of my initial problems. All hail Tom.

[tomka]

😁 Yeah it took a bit to figure out what the issue is. Apparently the problem is a bug in the VirtualBox Guest Additions in versions 6 and later.The reason this is no problem on the 16.04 VM is that it still uses version 5.2.32 and the 18.04 container uses version 6.1.2. Downgrading the Guest Additions on 18.04 seems rather tricky and I couldn't get it work reliably. The overlaysfs solutions seems to be working well though and also allows us to have separate node_modules folders inside the Vagrant VM and outside (which might be useful given that host and guest are typically different OSs).

This will bite us in 16.04 too should the VirtualBox Guest Additions be upgraded there to v6.x. (I am not sure, but I would guess this is rather unlikely.) If you still consider it useful, I am fine with merging the 16.04 vagrant branch first. Although I haven't tested the overlayfs stuff a lot, to me it seems like the 18.04 branch works fine, too, and I would be equally okay to merge this instead. If we merge the 16.04 branch we should maybe open a new PR with the 18.04 branch to not forget about how to work around the node_modules issue.

[clbarnes]

Let's push on with 18.04 if it's not more difficult that 16.04. I'm working on the postgres bit at the moment - how to access it from the outside. It seems like it's not even running at the moment.

[clbarnes]

Closing in favour of #1985