New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker dev environment #4165

Closed
wants to merge 46 commits into
base: master
from

Conversation

Projects
None yet
3 participants
@jgmize
Member

jgmize commented May 27, 2016

This is still WIP but opening now for early testing and feedback. See https://bugzilla.mozilla.org/show_bug.cgi?id=1276208#c2 for more details

TODO:

  • docs
  • unit test suite passing
  • functional test suite passing
  • headless test suite passing
  • containerized firefox + selenium hub integration tests passing
  • determine if collectstatic should be run during docker image creation

@jgmize jgmize added the WIP label May 27, 2016

@alexgibson

This comment has been minimized.

Show comment
Hide comment
@alexgibson

alexgibson Jun 2, 2016

Member

@jgmize - I'd be happy to try this out, but I'm a little lost on where to start. I do have the Docker Beta for Mac installed, but not really sure where to go from there?

Member

alexgibson commented Jun 2, 2016

@jgmize - I'd be happy to try this out, but I'm a little lost on where to start. I do have the Docker Beta for Mac installed, but not really sure where to go from there?

@jgmize

This comment has been minimized.

Show comment
Hide comment
@jgmize

jgmize Jun 2, 2016

Member

@jgmize - I'd be happy to try this out, but I'm a little lost on where to start. I do have the Docker Beta for Mac installed, but not really sure where to go from there?

Thanks @alexgibson, first just try checking out the branch locally and run 'make help' to give you a list of commands with brief explanations of what you can do. try out 'make gulp' first and let me know if it "just works" or not. On my machine 'make gulp' eats a lot of CPU, so you might also try just running 'make devserver' to run bedrock itself in the docker container and running 'gulp media:watch' on your local machine as a workaround.

Member

jgmize commented Jun 2, 2016

@jgmize - I'd be happy to try this out, but I'm a little lost on where to start. I do have the Docker Beta for Mac installed, but not really sure where to go from there?

Thanks @alexgibson, first just try checking out the branch locally and run 'make help' to give you a list of commands with brief explanations of what you can do. try out 'make gulp' first and let me know if it "just works" or not. On my machine 'make gulp' eats a lot of CPU, so you might also try just running 'make devserver' to run bedrock itself in the docker container and running 'gulp media:watch' on your local machine as a workaround.

@jgmize

This comment has been minimized.

Show comment
Hide comment
@jgmize

jgmize Jun 2, 2016

Member

Looks like I broke 'make help' in my last commit; will fix that, but in the meantime you can just view the docs/make-commands.rst file directly

Member

jgmize commented Jun 2, 2016

Looks like I broke 'make help' in my last commit; will fix that, but in the meantime you can just view the docs/make-commands.rst file directly

@jgmize

This comment has been minimized.

Show comment
Hide comment
@jgmize

jgmize Jun 2, 2016

Member

another issue with 'make gulp' is that currently control-c doesn't actually stop it-- you can switch to another terminal window and run 'make stop' from the bedrock dir to stop all your docker containers, or you can quit and relaunch docker from the gui.

Member

jgmize commented Jun 2, 2016

another issue with 'make gulp' is that currently control-c doesn't actually stop it-- you can switch to another terminal window and run 'make stop' from the bedrock dir to stop all your docker containers, or you can quit and relaunch docker from the gui.

@alexgibson

This comment has been minimized.

Show comment
Hide comment
@alexgibson

alexgibson Jun 3, 2016

Member

I tried make gulp and got this error:

make gulp
docker run --env-file .env -v /Users/agibson/sites/bedrock\:/app  -w /app -p "8000:8000" mozorg/bedrock_dev\:14bde6250ff607294052e40000f79fa1c4128f9a gulp
Unable to find image 'mozorg/bedrock_dev:14bde6250ff607294052e40000f79fa1c4128f9a' locally
Pulling repository docker.io/mozorg/bedrock_dev
docker: Tag 14bde6250ff607294052e40000f79fa1c4128f9a not found in repository docker.io/mozorg/bedrock_dev.
See 'docker run --help'.
make: *** [gulp] Error 125
Member

alexgibson commented Jun 3, 2016

I tried make gulp and got this error:

make gulp
docker run --env-file .env -v /Users/agibson/sites/bedrock\:/app  -w /app -p "8000:8000" mozorg/bedrock_dev\:14bde6250ff607294052e40000f79fa1c4128f9a gulp
Unable to find image 'mozorg/bedrock_dev:14bde6250ff607294052e40000f79fa1c4128f9a' locally
Pulling repository docker.io/mozorg/bedrock_dev
docker: Tag 14bde6250ff607294052e40000f79fa1c4128f9a not found in repository docker.io/mozorg/bedrock_dev.
See 'docker run --help'.
make: *** [gulp] Error 125
@jgmize

This comment has been minimized.

Show comment
Hide comment
@jgmize

jgmize Jun 3, 2016

Member

The most recent build pushed mozorg/bedrock_dev:cb1cac83530c1a6491d7b04df1c3e9bfff3321e2 which is based on the latest commit (cb1cac8) in this PR. The make commands detect your local git hash and default to using it if the commit isn't tagged. You can also override the VERSION environment variable or pass it in as an arg, (for example VERSION=latest) or whatever the parent commit of your local hash. Based on your error message, you're on 14bde6250ff607294052e40000f79fa1c4128f9a which is not currently part of this branch. If you pull down the latest commit of this branch and then run make gulp it should hopefully work.

I considered having the make commands fall back automatically to latest if you're on a local commit that hasn't been built or pushed, similar to how I set it up in the viewsourconf Makefile but I was concerned that the additional complexity and "magic" could cause confusion. I'd be very interested in feedback on this.

Member

jgmize commented Jun 3, 2016

The most recent build pushed mozorg/bedrock_dev:cb1cac83530c1a6491d7b04df1c3e9bfff3321e2 which is based on the latest commit (cb1cac8) in this PR. The make commands detect your local git hash and default to using it if the commit isn't tagged. You can also override the VERSION environment variable or pass it in as an arg, (for example VERSION=latest) or whatever the parent commit of your local hash. Based on your error message, you're on 14bde6250ff607294052e40000f79fa1c4128f9a which is not currently part of this branch. If you pull down the latest commit of this branch and then run make gulp it should hopefully work.

I considered having the make commands fall back automatically to latest if you're on a local commit that hasn't been built or pushed, similar to how I set it up in the viewsourconf Makefile but I was concerned that the additional complexity and "magic" could cause confusion. I'd be very interested in feedback on this.

@alexgibson

This comment has been minimized.

Show comment
Hide comment
@alexgibson

alexgibson Jun 3, 2016

Member

I considered having the make commands fall back automatically to latest if you're on a local commit that hasn't been built or pushed, similar to how I set it up in the viewsourconf Makefile but I was concerned that the additional complexity and "magic" could cause confusion. I'd be very interested in feedback on this.

My first thought is how would this work with pulling down other people's branches, PR's that need code review for example. It would be nice to try and make this as easy as possible, given that it's an almost daily occurrence?

Member

alexgibson commented Jun 3, 2016

I considered having the make commands fall back automatically to latest if you're on a local commit that hasn't been built or pushed, similar to how I set it up in the viewsourconf Makefile but I was concerned that the additional complexity and "magic" could cause confusion. I'd be very interested in feedback on this.

My first thought is how would this work with pulling down other people's branches, PR's that need code review for example. It would be nice to try and make this as easy as possible, given that it's an almost daily occurrence?

@jgmize

This comment has been minimized.

Show comment
Hide comment
@jgmize

jgmize Jun 3, 2016

Member

My first thought is how would this work with pulling down other people's branches, PR's that need code review for example. It would be nice to try and make this as easy as possible, given that it's an almost daily occurrence?

One way would be to just type VERSION=latest into your shell first and then run your make commands, which will then use the latest tag instead of your local git hash. However, if you're testing something that requires a new docker image (for example due to a new python package added to requirements) that has not been built by our ci you can run make build-dev locally, and it will tag the local docker image with that git sha automatically and any subsequent commands will use that locally-built image.

Member

jgmize commented Jun 3, 2016

My first thought is how would this work with pulling down other people's branches, PR's that need code review for example. It would be nice to try and make this as easy as possible, given that it's an almost daily occurrence?

One way would be to just type VERSION=latest into your shell first and then run your make commands, which will then use the latest tag instead of your local git hash. However, if you're testing something that requires a new docker image (for example due to a new python package added to requirements) that has not been built by our ci you can run make build-dev locally, and it will tag the local docker image with that git sha automatically and any subsequent commands will use that locally-built image.

@alexgibson

This comment has been minimized.

Show comment
Hide comment
@alexgibson

alexgibson Jun 3, 2016

Member

One way would be to just type VERSION=latest into your shell first and then run your make commands, which will then use the latest tag instead of your local git hash. However, if you're testing something that requires a new docker image (for example due to a new python package added to requirements) that has not been built by our ci you can run make build-dev locally, and it will tag the local docker image with that git sha automatically and any subsequent commands will use that locally-built image.

Makes sense, thanks (this is still a whole new world to me). I ran VERSION=latest make gulp and things "just worked" 👍

Member

alexgibson commented Jun 3, 2016

One way would be to just type VERSION=latest into your shell first and then run your make commands, which will then use the latest tag instead of your local git hash. However, if you're testing something that requires a new docker image (for example due to a new python package added to requirements) that has not been built by our ci you can run make build-dev locally, and it will tag the local docker image with that git sha automatically and any subsequent commands will use that locally-built image.

Makes sense, thanks (this is still a whole new world to me). I ran VERSION=latest make gulp and things "just worked" 👍

@alexgibson

This comment has been minimized.

Show comment
Hide comment
@alexgibson

alexgibson Jun 3, 2016

Member

This seems to work for me pretty nicely locally, although I had to avoid using "make gulp" as it uses up way too much CPU. Running "make devserver" in one terminal window, and "gulp media:watch" in another seems to work pretty well performance wise.

I'll give this some more testing, but I look forward to trying it out with some real day to day dev-work. Nice job 🚀

Member

alexgibson commented Jun 3, 2016

This seems to work for me pretty nicely locally, although I had to avoid using "make gulp" as it uses up way too much CPU. Running "make devserver" in one terminal window, and "gulp media:watch" in another seems to work pretty well performance wise.

I'll give this some more testing, but I look forward to trying it out with some real day to day dev-work. Nice job 🚀

@jgmize

This comment has been minimized.

Show comment
Hide comment
@jgmize

jgmize Jun 3, 2016

Member

As mentioned in IRC, I've added a gulp docker command which runs gulp on the host instead of a container to work around the high CPU utilization of watching for filesystem changes in a shared volume within the container, but it spawns make devserver to run the django dev server in a docker container. The environment variables pass through, so if for example you export VERSION=latest then run gulp docker it will use the latest tag for the docker container instead of your local git sha.

Member

jgmize commented Jun 3, 2016

As mentioned in IRC, I've added a gulp docker command which runs gulp on the host instead of a container to work around the high CPU utilization of watching for filesystem changes in a shared volume within the container, but it spawns make devserver to run the django dev server in a docker container. The environment variables pass through, so if for example you export VERSION=latest then run gulp docker it will use the latest tag for the docker container instead of your local git sha.

@pmac

This comment has been minimized.

Show comment
Hide comment
@pmac

pmac Jun 3, 2016

Member

I've been thinking about the latest and magic thing... I'm not sure off hand how to do this, but I think it should be possible to check to see if the git sha tag exists in docker, and if not fall back to latest rather than failing, perhaps printing a warning about the switch... I just fear that support requests and consternation will increase a lot if it fails to Just Work™️

Member

pmac commented Jun 3, 2016

I've been thinking about the latest and magic thing... I'm not sure off hand how to do this, but I think it should be possible to check to see if the git sha tag exists in docker, and if not fall back to latest rather than failing, perhaps printing a warning about the switch... I just fear that support requests and consternation will increase a lot if it fails to Just Work™️

@jgmize

This comment has been minimized.

Show comment
Hide comment
@jgmize

jgmize Jun 3, 2016

Member

I've been thinking about the latest and magic thing... I'm not sure off hand how to do this, but I think it should be possible to check to see if the git sha tag exists in docker, and if not fall back to latest rather than failing, perhaps printing a warning about the switch... I just fear that support requests and consternation will increase a lot if it fails to Just Work™️

Yes, as I mentioned in #4165 (comment) that's what I did with the viewsourceconf Makefile, and we can take a similar approach here if the tradeoffs are worth it.

Member

jgmize commented Jun 3, 2016

I've been thinking about the latest and magic thing... I'm not sure off hand how to do this, but I think it should be possible to check to see if the git sha tag exists in docker, and if not fall back to latest rather than failing, perhaps printing a warning about the switch... I just fear that support requests and consternation will increase a lot if it fails to Just Work™️

Yes, as I mentioned in #4165 (comment) that's what I did with the viewsourceconf Makefile, and we can take a similar approach here if the tradeoffs are worth it.

@jgmize

This comment has been minimized.

Show comment
Hide comment
@jgmize

jgmize Jun 3, 2016

Member

one simple option would be to change the gulp docker command to spawn make devserver VERSION=latest as the backend process instead of make devserver

Member

jgmize commented Jun 3, 2016

one simple option would be to change the gulp docker command to spawn make devserver VERSION=latest as the backend process instead of make devserver

@jgmize

This comment has been minimized.

Show comment
Hide comment
@jgmize

jgmize Jun 3, 2016

Member

I noticed that even running locally the gulp watch process is fairly CPU intensive, eating 50-60% on my machine. Since the minimum that gulp is doing now is just copying from media to static automatically I added a little make fswatch-media command that replicates that functionality with a fraction of the CPU via fswatch (installable via brew install fswatch).

Member

jgmize commented Jun 3, 2016

I noticed that even running locally the gulp watch process is fairly CPU intensive, eating 50-60% on my machine. Since the minimum that gulp is doing now is just copying from media to static automatically I added a little make fswatch-media command that replicates that functionality with a fraction of the CPU via fswatch (installable via brew install fswatch).

@jgmize

This comment has been minimized.

Show comment
Hide comment
@jgmize

jgmize Jun 3, 2016

Member

continuing on that theme, I was thinking we could use fswatch to trigger linting and other single run gulp tasks within a container, eliminating the need to install gulp locally-- I'll experiment with this as well

Member

jgmize commented Jun 3, 2016

continuing on that theme, I was thinking we could use fswatch to trigger linting and other single run gulp tasks within a container, eliminating the need to install gulp locally-- I'll experiment with this as well

@jgmize

This comment has been minimized.

Show comment
Hide comment
@jgmize

jgmize Jun 4, 2016

Member

continuing on that theme, I was thinking we could use fswatch to trigger linting and other single run gulp tasks within a container, eliminating the need to install gulp locally-- I'll experiment with this as well

Added a52aec5 to do this. make fswatch-media will now copy media files to static and if the changed file is js will run gulp js:lint in a container via the new make js-lint command. This is much more efficient than running gulp watch natively and eliminates the need to have node, npm, gulp, etc. installed locally.

Member

jgmize commented Jun 4, 2016

continuing on that theme, I was thinking we could use fswatch to trigger linting and other single run gulp tasks within a container, eliminating the need to install gulp locally-- I'll experiment with this as well

Added a52aec5 to do this. make fswatch-media will now copy media files to static and if the changed file is js will run gulp js:lint in a container via the new make js-lint command. This is much more efficient than running gulp watch natively and eliminates the need to have node, npm, gulp, etc. installed locally.

@jgmize

This comment has been minimized.

Show comment
Hide comment
@jgmize

jgmize Jun 4, 2016

Member

Just pushed some changes that reduced the bedrock dev image from 393 MB to 249 MB: https://hub.docker.com/r/mozorg/bedrock_dev/tags/

Member

jgmize commented Jun 4, 2016

Just pushed some changes that reduced the bedrock dev image from 393 MB to 249 MB: https://hub.docker.com/r/mozorg/bedrock_dev/tags/

@pmac

This comment has been minimized.

Show comment
Hide comment
@pmac

pmac Jun 4, 2016

Member

I'll just say that it's far more likely for a webdev to already have node, npm, gulp, etc. installed than fswatch. Can we also keep the gulp way?

Member

pmac commented Jun 4, 2016

I'll just say that it's far more likely for a webdev to already have node, npm, gulp, etc. installed than fswatch. Can we also keep the gulp way?

@alexgibson

This comment has been minimized.

Show comment
Hide comment
@alexgibson

alexgibson Jun 4, 2016

Member

I think @pmac does have a good point here. I'm all for a super efficient fswatch method for those that want it, but I think the majority of devs (read contributors) might be content enough with a simple npm install?

Member

alexgibson commented Jun 4, 2016

I think @pmac does have a good point here. I'm all for a super efficient fswatch method for those that want it, but I think the majority of devs (read contributors) might be content enough with a simple npm install?

@jgmize

This comment has been minimized.

Show comment
Hide comment
@jgmize

jgmize Jun 4, 2016

Member

Yes, I was planning on supporting all three approaches and documenting the tradeoffs.

Member

jgmize commented Jun 4, 2016

Yes, I was planning on supporting all three approaches and documenting the tradeoffs.

@alexgibson

This comment has been minimized.

Show comment
Hide comment
@alexgibson

alexgibson Jun 15, 2016

Member

I'd like to vote to remove commands like make gulp, as it uses up way too much CPU and the site runs slowly. Running gulp docker seems fine, keeps CPU low and the site loads quickly.

Member

alexgibson commented Jun 15, 2016

I'd like to vote to remove commands like make gulp, as it uses up way too much CPU and the site runs slowly. Running gulp docker seems fine, keeps CPU low and the site loads quickly.

@alexgibson

This comment has been minimized.

Show comment
Hide comment
@alexgibson

alexgibson Jun 15, 2016

Member

I'd like to vote to remove commands like make gulp, as it uses up way too much CPU and the site runs slowly. Running gulp docker seems fine, keeps CPU low and the site loads quickly.

After discussion keeping make gulp around might be fine for Linux users. We're going to try and document the recommended install paths for both OSX and Linux users individually.

Member

alexgibson commented Jun 15, 2016

I'd like to vote to remove commands like make gulp, as it uses up way too much CPU and the site runs slowly. Running gulp docker seems fine, keeps CPU low and the site loads quickly.

After discussion keeping make gulp around might be fine for Linux users. We're going to try and document the recommended install paths for both OSX and Linux users individually.

pmac and others added some commits Aug 12, 2016

Bug 1276208: Upgrade nodejs in dockerfiles
- Change the base image to quay.io/mozmar/base
- Upgrade prod node dependencies
- Switch to yuglify for js and css minification

(cherry picked from commit 0ab71c8)
Remove hashin and argparse from dev requirements
work around hashin 0.5 broken in pip 8.1.1-2ubuntu0.1 and later
remove argparse as it is an optional dependency only used in python 2.6

jgmize added some commits Aug 1, 2016

Remove redundant migrations that delay startup
DB migrations should be run once, not on every container
Fix push-usw, push-euw, and .env
Strip DEV= from .env
Remove local registry prefix from deis pull command
@jgmize

This comment has been minimized.

Show comment
Hide comment
@jgmize

jgmize Oct 28, 2016

Member

This needs to be reworked significantly, closing in favor of new PR.

Member

jgmize commented Oct 28, 2016

This needs to be reworked significantly, closing in favor of new PR.

@jgmize jgmize closed this Oct 28, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment