Split Dockerfile into dev+prod version

[bhearsum]

Along with https://bugzilla.mozilla.org/show_bug.cgi?id=1304101, I've come to realize just how inefficient the dev workflow is right now if you have a slow connection. There are numerous places where we don't use caches properly, and we rebuild the UI far more than necessary. There's lots of going in this patch, so I'll make some inline comments to try to explain it all.

[bhearsum]

I think this file is pretty straightforward - I'm just stripping out all the test-only stuff. It's a PITA to test it now though, because there's no way to fire up prod-style containers without tweaking docker-compose.yml. Perhaps I should add a companion docker-compose.yml that uses this docker file? In the meantime, you can adjust the "dockerfile" option, remove volume mounts, and the entrypoint from admin to make it work.

[JohanLorenzo]

Tweaking docker-compose.yml worked fine from my side

[bhearsum]

Yeah, it works fine, but it's a PITA to do if you want to regularly test production-like containers.

[bhearsum]

The idea of this and the previous block is that we want to do all the network stuff first. Copying in just the absolutely necessary files means we don't re-run pip nor npm install unless something that affects their outcome changes.

[bhearsum]

Similarly, we copy in the UI on its own, and then build it before copying in anything else. This means that you'll have this command cached if you're only hacking on the backend.

[JohanLorenzo]

Nice!

[bhearsum]

This file used to be our pre-tox one, but I've repurposed it and moved the requirements that we used to specify in tox.ini into here, and pinned them. This should help with reproducability.

[bhearsum]

Using a volume mount here means that ".tox" ends up on the host, which means we don't re-download and re-install all of the test dependencies with every run. This should be a big speed up for those on slower connections.

[bhearsum]

This was moved to Dockerfile.dev.

[bhearsum]

Something that his explicitly doesn't fix is the permissions problem on files created inside of the containers (eg, ui/dist). I'm not sure this is fixable right now after reading through moby/moby#2259.

@nthomas-mozilla, @nurav, @JohanLorenzo - I need to take a second pass at this with fresh eyes next week, but if any of you have a chance to give it a try I'd appreciate it.

[JohanLorenzo]

The changes make sense to me. FWIW, I had to recreate the images locally, otherwise balrogadmin complains about mysql not being installed. In order to test locally, I wiped every images I had, git clean -fdx my repo, applied your changes and built'em. I've had no issue so far, and the dev built the UI twice, like expected.

The prod instance works from my side as well. I just had to comment out both volumes and entrypoints, just like you said.

Regarding the failure in CI, I haven't experienced it, run-tests.sh works fine here.. It seems related to volumes mounting after building the dev instance. Maybe -v pwd:/app doesn't work while already being in a dockerized environment?

[JohanLorenzo]

Nice!

[JohanLorenzo]

Good idea!

[JohanLorenzo]

Is update still necessary here?

[bhearsum]

I suppose not!

[JohanLorenzo]

Tweaking docker-compose.yml worked fine from my side

[bhearsum]

The latest commits do a few more things:

• Stop doing any UI stuff in Dockerfile.dev, because the node_modules and built UI don't get used in dev because of the volume mount. This happens as part of startup now.
• Get rid of test-entrypoint.sh in favour of merging it in with run.sh. This seemed like an obvious improvement after building the UI at startup.

Permissions are still an issue, and I don't think I can do anything about it. The workaround discussed in the aforementioned docker issue is to run with --user, and then create a user with that UID at startup. Unfortunately, I can't find a way to pass along the correct UID in docker-compose.yml. $USER is typically available in the environment, but $UID is not. Unless I can find a fix, everything created at startup (.tox, .cache, ui/{dist,generated,node_modules}) will be root-owned. We can mitigate this a bit by updating docs to have people build their UI on the host before running docker-compose, but it's not a perfect solution.

I still need to figure out the Docker run issue in CI, too.

[bhearsum]

I asked the Taskcluster folks about the Docker error, and it looks like volume mounts are not supported:

16:39 < bhearsum> unrelated to the above, is it expected that bind mounts with dind don't work? i get: https://pastebin.mozilla.org/8913587
16:42 < dustin> I suspect so
16:42 < dustin> since the source mount would be from the parent
16:42 < bhearsum> ah, darn
16:43 < bhearsum> does that mean that when i run a container in a dind-enabled image, that it ends up running that container on its hosts docker daemon?
16:43 < dustin> yes

Given that, building the UI in the Dockerfile may be unavoidable.

[JohanLorenzo]

it looks like volume mounts are not supported

Oooooh 😞 . We can also mitigate that by either updating the documentation (downloading the dependencies is only done once during the image building, after node_modules is fetched on the host machine), or by creating a Dockerfile.test. I believe the first one is better, as it doesn't create another new environment.

[bhearsum]

Can you flesh out exactly what the first option is, I'm not sure I fully understand what you're suggesting. I do agree that adding a Dockerfile.test is undesireable though - I don't want to be in a situation where local tests are different than CI tests.

[bhearsum]

@JohanLorenzo and I brainstormed and hacked on this a bit this morning. We realized that one of the biggest pain points for local development is that "npm install" and "npm build" are sometimes run in the Docker container, and other times run on the host. Because of the aforementioned permissions issue with volume mounts, this can be very difficult. There can also be some problems if the host node/npm is incompatible with the one used in Docker.

We came up with a clever idea to fix this (mostly @JohanLorenzo), which is to run the UI with "lineman" run in its own container. Doing this means that all npm install and build commands run within the container, and lineman will automatically rebuild the UI whenever files change - eliminating the need for rebuilds on the host. This has the nice bonus of forcing local development to use non-minified sources, which makes debugging in the browser a LOT easier.

Johan is also looking at relocating the "generated" and "node_modules" directory into .cache, to avoid having root-owned files in ui/.

The other issue we still had to deal with was the inability to volume mount in Taskcluster. This means that we cannot rely on volume mounts to run tests. To workaround this, we've started copying the entire repository into the image in Dockerfile.dev. This ends up overridden by the volume mount when running docker-compose, but it eliminates the need for a volume mount in run-tests.sh.

This needs

[bhearsum]

I've tested this personally on Linux, Mac and Windows. It worked 100% out of box on Linux and Mac, but I'm running into issues on Windows, probably related to the volume mount, or possibly related to me doing something dumb. @F3real, would you mind giving this a try? I've already merged #134 into this branch, but if you're able to make it work I'd love to know what, if any, changes you have to make.

I'd also appreciate some testing from other folks on Linux and Mac, there may be some different configurations that I'm not covering in my own testing. @rail, @nthomas-mozilla, @nurav, @njirap, @xunga - if any of you have time, I'm curious how "docker-compose up" goes for you.

Regardless of platform, I strongly advise removing all existing Balrog Docker images before you try to run docker-compose.

[F3real]

Well I'll make another PR so that in Windows we see entrypoints in Dockerfiles. But this still won't fix issue completely (it will just help docker-up compose not break at start). Main problem is that by default all files miss execute permission and then they get reported as missing.
I was trying to fix it using
RUN chmod +x <some file>
but that just fixes one file and it still breaks.
Now things I still haven't tested are:

1. Simply adding execute to all files ( not sure how to do this for all balrog_db and pub since they get just built from .yml file, or maybe I still just haven't found dockerfile for them).
2. Also I heard that you can specify user who owns files in dockerfile so they aren't owned by docker, but I didn't really look into it yet.

[bhearsum]

Thanks @F3real, I appreciate you testing this.

[rail]

The first time I ran docker-compose build because apt-get install failed to download a file due to some network issues (remote closed connection). We can improve this by using apt-get ... -o Acquire::Retires=5 install ... instead. Not a blocker though.

[bhearsum]

Rail ran into an interesting issue when testing this out. It looks to be the same or related to docker/compose#2172, and appears to be rooted in a race condition when docker-compose updates /etc/hosts for linked containers. My suspicion is that it only happens with older versions of docker-compose, and possibly only with certain versions of Docker. Rail was able unable to reproduce it consistently, so I'm not going to block on it -- everything worked for him otherwise.

[JohanLorenzo]

Yay! 🙌