Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker dev environment #4165

Closed
wants to merge 46 commits into from

Conversation

@jgmize
Copy link
Member

commented May 27, 2016

This is still WIP but opening now for early testing and feedback. See https://bugzilla.mozilla.org/show_bug.cgi?id=1276208#c2 for more details

TODO:

  • docs
  • unit test suite passing
  • functional test suite passing
  • headless test suite passing
  • containerized firefox + selenium hub integration tests passing
  • determine if collectstatic should be run during docker image creation
@alexgibson

This comment has been minimized.

Copy link
Member

commented Jun 2, 2016

@jgmize - I'd be happy to try this out, but I'm a little lost on where to start. I do have the Docker Beta for Mac installed, but not really sure where to go from there?

@jgmize

This comment has been minimized.

Copy link
Member Author

commented Jun 2, 2016

@jgmize - I'd be happy to try this out, but I'm a little lost on where to start. I do have the Docker Beta for Mac installed, but not really sure where to go from there?

Thanks @alexgibson, first just try checking out the branch locally and run 'make help' to give you a list of commands with brief explanations of what you can do. try out 'make gulp' first and let me know if it "just works" or not. On my machine 'make gulp' eats a lot of CPU, so you might also try just running 'make devserver' to run bedrock itself in the docker container and running 'gulp media:watch' on your local machine as a workaround.

@jgmize

This comment has been minimized.

Copy link
Member Author

commented Jun 2, 2016

Looks like I broke 'make help' in my last commit; will fix that, but in the meantime you can just view the docs/make-commands.rst file directly

@jgmize

This comment has been minimized.

Copy link
Member Author

commented Jun 2, 2016

another issue with 'make gulp' is that currently control-c doesn't actually stop it-- you can switch to another terminal window and run 'make stop' from the bedrock dir to stop all your docker containers, or you can quit and relaunch docker from the gui.

@alexgibson

This comment has been minimized.

Copy link
Member

commented Jun 3, 2016

I tried make gulp and got this error:

make gulp
docker run --env-file .env -v /Users/agibson/sites/bedrock\:/app  -w /app -p "8000:8000" mozorg/bedrock_dev\:14bde6250ff607294052e40000f79fa1c4128f9a gulp
Unable to find image 'mozorg/bedrock_dev:14bde6250ff607294052e40000f79fa1c4128f9a' locally
Pulling repository docker.io/mozorg/bedrock_dev
docker: Tag 14bde6250ff607294052e40000f79fa1c4128f9a not found in repository docker.io/mozorg/bedrock_dev.
See 'docker run --help'.
make: *** [gulp] Error 125
@jgmize

This comment has been minimized.

Copy link
Member Author

commented Jun 3, 2016

The most recent build pushed mozorg/bedrock_dev:cb1cac83530c1a6491d7b04df1c3e9bfff3321e2 which is based on the latest commit (cb1cac8) in this PR. The make commands detect your local git hash and default to using it if the commit isn't tagged. You can also override the VERSION environment variable or pass it in as an arg, (for example VERSION=latest) or whatever the parent commit of your local hash. Based on your error message, you're on 14bde6250ff607294052e40000f79fa1c4128f9a which is not currently part of this branch. If you pull down the latest commit of this branch and then run make gulp it should hopefully work.

I considered having the make commands fall back automatically to latest if you're on a local commit that hasn't been built or pushed, similar to how I set it up in the viewsourconf Makefile but I was concerned that the additional complexity and "magic" could cause confusion. I'd be very interested in feedback on this.

@alexgibson

This comment has been minimized.

Copy link
Member

commented Jun 3, 2016

I considered having the make commands fall back automatically to latest if you're on a local commit that hasn't been built or pushed, similar to how I set it up in the viewsourconf Makefile but I was concerned that the additional complexity and "magic" could cause confusion. I'd be very interested in feedback on this.

My first thought is how would this work with pulling down other people's branches, PR's that need code review for example. It would be nice to try and make this as easy as possible, given that it's an almost daily occurrence?

@jgmize

This comment has been minimized.

Copy link
Member Author

commented Jun 3, 2016

My first thought is how would this work with pulling down other people's branches, PR's that need code review for example. It would be nice to try and make this as easy as possible, given that it's an almost daily occurrence?

One way would be to just type VERSION=latest into your shell first and then run your make commands, which will then use the latest tag instead of your local git hash. However, if you're testing something that requires a new docker image (for example due to a new python package added to requirements) that has not been built by our ci you can run make build-dev locally, and it will tag the local docker image with that git sha automatically and any subsequent commands will use that locally-built image.

@alexgibson

This comment has been minimized.

Copy link
Member

commented Jun 3, 2016

One way would be to just type VERSION=latest into your shell first and then run your make commands, which will then use the latest tag instead of your local git hash. However, if you're testing something that requires a new docker image (for example due to a new python package added to requirements) that has not been built by our ci you can run make build-dev locally, and it will tag the local docker image with that git sha automatically and any subsequent commands will use that locally-built image.

Makes sense, thanks (this is still a whole new world to me). I ran VERSION=latest make gulp and things "just worked" 👍

@alexgibson

This comment has been minimized.

Copy link
Member

commented Jun 3, 2016

This seems to work for me pretty nicely locally, although I had to avoid using "make gulp" as it uses up way too much CPU. Running "make devserver" in one terminal window, and "gulp media:watch" in another seems to work pretty well performance wise.

I'll give this some more testing, but I look forward to trying it out with some real day to day dev-work. Nice job 🚀

@jgmize

This comment has been minimized.

Copy link
Member Author

commented Jun 3, 2016

As mentioned in IRC, I've added a gulp docker command which runs gulp on the host instead of a container to work around the high CPU utilization of watching for filesystem changes in a shared volume within the container, but it spawns make devserver to run the django dev server in a docker container. The environment variables pass through, so if for example you export VERSION=latest then run gulp docker it will use the latest tag for the docker container instead of your local git sha.

@pmac

This comment has been minimized.

Copy link
Member

commented Jun 3, 2016

I've been thinking about the latest and magic thing... I'm not sure off hand how to do this, but I think it should be possible to check to see if the git sha tag exists in docker, and if not fall back to latest rather than failing, perhaps printing a warning about the switch... I just fear that support requests and consternation will increase a lot if it fails to Just Work™️

@jgmize

This comment has been minimized.

Copy link
Member Author

commented Jun 3, 2016

I've been thinking about the latest and magic thing... I'm not sure off hand how to do this, but I think it should be possible to check to see if the git sha tag exists in docker, and if not fall back to latest rather than failing, perhaps printing a warning about the switch... I just fear that support requests and consternation will increase a lot if it fails to Just Work™️

Yes, as I mentioned in #4165 (comment) that's what I did with the viewsourceconf Makefile, and we can take a similar approach here if the tradeoffs are worth it.

@jgmize

This comment has been minimized.

Copy link
Member Author

commented Jun 3, 2016

one simple option would be to change the gulp docker command to spawn make devserver VERSION=latest as the backend process instead of make devserver

@jgmize

This comment has been minimized.

Copy link
Member Author

commented Jun 3, 2016

I noticed that even running locally the gulp watch process is fairly CPU intensive, eating 50-60% on my machine. Since the minimum that gulp is doing now is just copying from media to static automatically I added a little make fswatch-media command that replicates that functionality with a fraction of the CPU via fswatch (installable via brew install fswatch).

@jgmize

This comment has been minimized.

Copy link
Member Author

commented Jun 3, 2016

continuing on that theme, I was thinking we could use fswatch to trigger linting and other single run gulp tasks within a container, eliminating the need to install gulp locally-- I'll experiment with this as well

@jgmize

This comment has been minimized.

Copy link
Member Author

commented Jun 4, 2016

continuing on that theme, I was thinking we could use fswatch to trigger linting and other single run gulp tasks within a container, eliminating the need to install gulp locally-- I'll experiment with this as well

Added a52aec5 to do this. make fswatch-media will now copy media files to static and if the changed file is js will run gulp js:lint in a container via the new make js-lint command. This is much more efficient than running gulp watch natively and eliminates the need to have node, npm, gulp, etc. installed locally.

@jgmize jgmize force-pushed the jgmize:docker-dev branch from 069bb58 to 6349c36 Jun 4, 2016
@jgmize

This comment has been minimized.

Copy link
Member Author

commented Jun 4, 2016

Just pushed some changes that reduced the bedrock dev image from 393 MB to 249 MB: https://hub.docker.com/r/mozorg/bedrock_dev/tags/

@pmac

This comment has been minimized.

Copy link
Member

commented Jun 4, 2016

I'll just say that it's far more likely for a webdev to already have node, npm, gulp, etc. installed than fswatch. Can we also keep the gulp way?

@alexgibson

This comment has been minimized.

Copy link
Member

commented Jun 4, 2016

I think @pmac does have a good point here. I'm all for a super efficient fswatch method for those that want it, but I think the majority of devs (read contributors) might be content enough with a simple npm install?

@jgmize

This comment has been minimized.

Copy link
Member Author

commented Jun 4, 2016

Yes, I was planning on supporting all three approaches and documenting the tradeoffs.

@alexgibson

This comment has been minimized.

Copy link
Member

commented Jun 15, 2016

I'd like to vote to remove commands like make gulp, as it uses up way too much CPU and the site runs slowly. Running gulp docker seems fine, keeps CPU low and the site loads quickly.

@alexgibson

This comment has been minimized.

Copy link
Member

commented Jun 15, 2016

I'd like to vote to remove commands like make gulp, as it uses up way too much CPU and the site runs slowly. Running gulp docker seems fine, keeps CPU low and the site loads quickly.

After discussion keeping make gulp around might be fine for Linux users. We're going to try and document the recommended install paths for both OSX and Linux users individually.

@jgmize jgmize force-pushed the jgmize:docker-dev branch 6 times, most recently from d4f1a72 to 324797b Aug 1, 2016
@jgmize jgmize force-pushed the jgmize:docker-dev branch from 3d72868 to 6dfae23 Aug 23, 2016
@jgmize

This comment has been minimized.

Copy link
Member Author

commented Oct 28, 2016

This needs to be reworked significantly, closing in favor of new PR.

@jgmize jgmize closed this Oct 28, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.