Docker dev environment

[jgmize]

This is still WIP but opening now for early testing and feedback. See https://bugzilla.mozilla.org/show_bug.cgi?id=1276208#c2 for more details

TODO:

• docs
• unit test suite passing
• functional test suite passing
• headless test suite passing
• containerized firefox + selenium hub integration tests passing
• determine if collectstatic should be run during docker image creation

[alexgibson]

@jgmize - I'd be happy to try this out, but I'm a little lost on where to start. I do have the Docker Beta for Mac installed, but not really sure where to go from there?

[jgmize]

@jgmize - I'd be happy to try this out, but I'm a little lost on where to start. I do have the Docker Beta for Mac installed, but not really sure where to go from there?

Thanks @alexgibson, first just try checking out the branch locally and run 'make help' to give you a list of commands with brief explanations of what you can do. try out 'make gulp' first and let me know if it "just works" or not. On my machine 'make gulp' eats a lot of CPU, so you might also try just running 'make devserver' to run bedrock itself in the docker container and running 'gulp media:watch' on your local machine as a workaround.

[jgmize]

Looks like I broke 'make help' in my last commit; will fix that, but in the meantime you can just view the docs/make-commands.rst file directly

[jgmize]

another issue with 'make gulp' is that currently control-c doesn't actually stop it-- you can switch to another terminal window and run 'make stop' from the bedrock dir to stop all your docker containers, or you can quit and relaunch docker from the gui.

[alexgibson]

I tried make gulp and got this error:

make gulp
docker run --env-file .env -v /Users/agibson/sites/bedrock\:/app  -w /app -p "8000:8000" mozorg/bedrock_dev\:14bde6250ff607294052e40000f79fa1c4128f9a gulp
Unable to find image 'mozorg/bedrock_dev:14bde6250ff607294052e40000f79fa1c4128f9a' locally
Pulling repository docker.io/mozorg/bedrock_dev
docker: Tag 14bde6250ff607294052e40000f79fa1c4128f9a not found in repository docker.io/mozorg/bedrock_dev.
See 'docker run --help'.
make: *** [gulp] Error 125

[jgmize]

The most recent build pushed mozorg/bedrock_dev:cb1cac83530c1a6491d7b04df1c3e9bfff3321e2 which is based on the latest commit (cb1cac8) in this PR. The make commands detect your local git hash and default to using it if the commit isn't tagged. You can also override the VERSION environment variable or pass it in as an arg, (for example VERSION=latest) or whatever the parent commit of your local hash. Based on your error message, you're on 14bde6250ff607294052e40000f79fa1c4128f9a which is not currently part of this branch. If you pull down the latest commit of this branch and then run make gulp it should hopefully work.

I considered having the make commands fall back automatically to latest if you're on a local commit that hasn't been built or pushed, similar to how I set it up in the viewsourconf Makefile but I was concerned that the additional complexity and "magic" could cause confusion. I'd be very interested in feedback on this.

[alexgibson]

I considered having the make commands fall back automatically to latest if you're on a local commit that hasn't been built or pushed, similar to how I set it up in the viewsourconf Makefile but I was concerned that the additional complexity and "magic" could cause confusion. I'd be very interested in feedback on this.

My first thought is how would this work with pulling down other people's branches, PR's that need code review for example. It would be nice to try and make this as easy as possible, given that it's an almost daily occurrence?

[jgmize]

My first thought is how would this work with pulling down other people's branches, PR's that need code review for example. It would be nice to try and make this as easy as possible, given that it's an almost daily occurrence?

One way would be to just type VERSION=latest into your shell first and then run your make commands, which will then use the latest tag instead of your local git hash. However, if you're testing something that requires a new docker image (for example due to a new python package added to requirements) that has not been built by our ci you can run make build-dev locally, and it will tag the local docker image with that git sha automatically and any subsequent commands will use that locally-built image.

[alexgibson]

One way would be to just type VERSION=latest into your shell first and then run your make commands, which will then use the latest tag instead of your local git hash. However, if you're testing something that requires a new docker image (for example due to a new python package added to requirements) that has not been built by our ci you can run make build-dev locally, and it will tag the local docker image with that git sha automatically and any subsequent commands will use that locally-built image.

Makes sense, thanks (this is still a whole new world to me). I ran VERSION=latest make gulp and things "just worked" 👍

[alexgibson]

This seems to work for me pretty nicely locally, although I had to avoid using "make gulp" as it uses up way too much CPU. Running "make devserver" in one terminal window, and "gulp media:watch" in another seems to work pretty well performance wise.

I'll give this some more testing, but I look forward to trying it out with some real day to day dev-work. Nice job 🚀

[jgmize]

As mentioned in IRC, I've added a gulp docker command which runs gulp on the host instead of a container to work around the high CPU utilization of watching for filesystem changes in a shared volume within the container, but it spawns make devserver to run the django dev server in a docker container. The environment variables pass through, so if for example you export VERSION=latest then run gulp docker it will use the latest tag for the docker container instead of your local git sha.

[pmac]

I've been thinking about the latest and magic thing... I'm not sure off hand how to do this, but I think it should be possible to check to see if the git sha tag exists in docker, and if not fall back to latest rather than failing, perhaps printing a warning about the switch... I just fear that support requests and consternation will increase a lot if it fails to Just Work™️

[jgmize]

I've been thinking about the latest and magic thing... I'm not sure off hand how to do this, but I think it should be possible to check to see if the git sha tag exists in docker, and if not fall back to latest rather than failing, perhaps printing a warning about the switch... I just fear that support requests and consternation will increase a lot if it fails to Just Work™️

Yes, as I mentioned in #4165 (comment) that's what I did with the viewsourceconf Makefile, and we can take a similar approach here if the tradeoffs are worth it.

[jgmize]

one simple option would be to change the gulp docker command to spawn make devserver VERSION=latest as the backend process instead of make devserver

[jgmize]

I noticed that even running locally the gulp watch process is fairly CPU intensive, eating 50-60% on my machine. Since the minimum that gulp is doing now is just copying from media to static automatically I added a little make fswatch-media command that replicates that functionality with a fraction of the CPU via fswatch (installable via brew install fswatch).

[jgmize]

continuing on that theme, I was thinking we could use fswatch to trigger linting and other single run gulp tasks within a container, eliminating the need to install gulp locally-- I'll experiment with this as well

[jgmize]

continuing on that theme, I was thinking we could use fswatch to trigger linting and other single run gulp tasks within a container, eliminating the need to install gulp locally-- I'll experiment with this as well

Added a52aec5 to do this. make fswatch-media will now copy media files to static and if the changed file is js will run gulp js:lint in a container via the new make js-lint command. This is much more efficient than running gulp watch natively and eliminates the need to have node, npm, gulp, etc. installed locally.

[jgmize]

Just pushed some changes that reduced the bedrock dev image from 393 MB to 249 MB: https://hub.docker.com/r/mozorg/bedrock_dev/tags/

[pmac]

I'll just say that it's far more likely for a webdev to already have node, npm, gulp, etc. installed than fswatch. Can we also keep the gulp way?

[alexgibson]

I think @pmac does have a good point here. I'm all for a super efficient fswatch method for those that want it, but I think the majority of devs (read contributors) might be content enough with a simple npm install?

[jgmize]

Yes, I was planning on supporting all three approaches and documenting the tradeoffs.

[alexgibson]

I'd like to vote to remove commands like make gulp, as it uses up way too much CPU and the site runs slowly. Running gulp docker seems fine, keeps CPU low and the site loads quickly.

[alexgibson]

I'd like to vote to remove commands like make gulp, as it uses up way too much CPU and the site runs slowly. Running gulp docker seems fine, keeps CPU low and the site loads quickly.

After discussion keeping make gulp around might be fine for Linux users. We're going to try and document the recommended install paths for both OSX and Linux users individually.

[jgmize]

This needs to be reworked significantly, closing in favor of new PR.