Docker compose setup for dev and integration tests

[mihaibivol]

Howto:

Get the latest Docker setup.

• If you are using Mac, please build a simple box with docker-engine above 1.10 and docker-compose above 1.6.0.

Make sure you can run docker without sudo.

• id $USER
  If you are not in the docker group, run the following command and then restart docker. If this doesn't work, just restart your machine :)
• sudo usermod -a -G docker $USER

Get docker-compose:

sudo pip install docker-compose

• Add DOCKER_DATA env variable in your .bashrc or .zshrc. In this directory you will have all the persistent data between Docker runs.

export DOCKER_DATA=~/inspirehep_docker_data/

By default the virtualenv and everything else will be kept on /tmp and they will be available only until the next reboot.

• Install a host persistent venv and build assets

docker-compose pull
docker-compose -f docker-compose.deps.yml run --rm pip
docker-compose -f docker-compose.deps.yml run --rm assets

• Run the service locally

docker-compose up

Go to localhost:5000

• Populate database

docker-compose run --rm web scripts/recreate_records

• Run tests in an isolated environment:

docker-compose -f docker-compose.test.yml run --rm test
docker-compose -f docker-compose.test.yml kill
docker-compose -f docker-compose.test.yml rm

The kill command is mandatory, otherwise you will use the test DB instead of the correct one in dev.

I don't know how to kill all the other services just after the run command exited.

Extra useful tips

• Run a random shell

After docker-compose up just run

docker-compose run --rm web inspirehep shell

• Reload code in a worker

With docker-compose up just run:

docker-compose restart worker

• Quick and safe reindex

With docker-compose up just run:

docker-compose restart worker && docker-compose run --rm web scripts/recreate_records

TODOs:

• Fix indexing
• Add test compose file
• Make data persistent for dev

Future Stuff:

• Use build arg to actually create a staging or prod image.

[mihaibivol]

@inspirehep/inspire-dev Feel free to comment on how these commands can be further simplified. e.g. I'm not sure how to drop the lengthy docker-compose -f docker-compose.build.yml

[coveralls]

Coverage increased (+0.009%) to 53.899% when pulling a76c414 on mihaibivol:dockerize into f0d6b44 on inspirehep:invenio3.

[kaplun]

Is there some preparatory steps where one can install icu and reindexing plugin?

[jacquerie]

I almost got this working. One problem under Mac OS is that ports aren't forwarded, so one has to type ${UGLY_URL}:5000 to access the web app, or set up port forwarding in Virtualbox, or forward all ports using a local nginx (http://stackoverflow.com/questions/25327012/access-docker-from-external-machine-in-network).

[jacquerie]

Actually, one has to enable one of these solutions, otherwise the application will return a 404 (I have no idea why...).

Other than that, I measured that this solution runs tests 5X slower than before, but achieves better isolation and reproducibility, a tradeoff which I think we can afford (it's 12 seconds vs 1 minute on my machine). Moreover, disk usage is just 300MB, probably thanks to the --rm option.

So it's a 👍 for me. @kaplun, if we decide to go forward on this (it's your call anyway!) I think we should have a Monday meeting to explain the new developer workflow that @mihaibivol is building.

[kaplun]

Eheh, Indeed I'd like to understand it as well... What do you think to do when we introduce beard and the thousands of compiled dependencies?

[coveralls]

Coverage increased (+0.009%) to 53.899% when pulling 0c545d9 on mihaibivol:dockerize into f0d6b44 on inspirehep:invenio3.

[mihaibivol]

Currently everything that we "build" is cached in /virtualenv in the base service. This means that for dev at least you only have to install the requirements once and update them exactly as you would do when using a local venv. I am still not sure what's the best take for running this setup on Travis and caching at a level at which accidentaly removing deps from install_requires actually makes the build fail.

[jacquerie]

What's the reason we don't use the linked post's solution for fixing docker-compose version?

[mihaibivol]

This has the same version and makes .travis.yml less verbose. Also it's the same as I did locally.

[jacquerie]

Yeah, but isn't his approach more robust? For example, let's say that the version on pip updates to 2.0.0, which is incompatible with the Docker version we require.

[jalavik]

Awesome stuff @mihaibivol. I will give this a closer look and test.

[coveralls]

Coverage increased (+0.05%) to 53.902% when pulling 0c4b02a on mihaibivol:dockerize into aeed027 on inspirehep:invenio3.

[coveralls]

Coverage increased (+0.05%) to 53.902% when pulling 4b06a40 on mihaibivol:dockerize into aeed027 on inspirehep:invenio3.

[jacquerie]

If Mac OS X developers want to test the Docker way here's a Vagrantfile to help them do so: jacquerie@7ddb99c.

Unfortunately I wasn't able to have docker-machine running + ES (known issue of mismatched UIDs), so for now Vagrant is the way.

[jacquerie]

...you're going to hate me a bit, but here's another commit you should cherry-pick and squash: jacquerie@071ce85

This one adds the license headers to the files you introduced.

[jacquerie]

travis_retry to avoid trivial build errors?

[jacquerie]

And maybe remove the comment and put it before everything, where it was. Or does that break stuff?

[jacquerie]

LGTM. Cherry-pick those two commits, squash, incorporate those nits and

[coveralls]

Coverage increased (+0.008%) to 54.576% when pulling 76610f1 on mihaibivol:dockerize into 19921ac on inspirehep:invenio3.

[kaplun]

Guys you have my full trust on this branch :-) Don't know if @jalavik wanted to further comment...

[jacquerie]

🚢 🚢 🚢

[jalavik]

Great stuff guys. Note that it is still optional if one wants to develop using docker. This branch does not enforce it.

[jacquerie]

BTW we kept the Vagrantfile out of this PR because we are waiting to see if Docker for Mac could be an alternative: https://blog.docker.com/2016/03/docker-for-mac-windows-beta/.