Create production containers with all code copied in instead of mounted #54
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is currently a copy of the dev Dockerfile to set up as a baseline before we start changing it for production builds
This uses the same http-server image as the development webapp, but copies all non-plot files into the container. This ensures that the container can be scanned for vulnerabilities. The plots are stored in a mounted volume because they are generated by the `viz_scripts` container and stored in the shared, mounted volume. ``` => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 42B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/danjellz/http-server:1.4 0.0s => [internal] load build context 0.0s => => transferring context: 1.76kB 0.0s => [1/4] FROM docker.io/danjellz/http-server:1.4 0.0s => CACHED [2/4] RUN mkdir -p /public 0.0s => CACHED [3/4] COPY ../client /public/client 0.0s => CACHED [4/4] COPY ../*.html /public 0.0s => exporting to image 0.0s => => exporting layers 0.0s => => writing image sha256:ff0154b07bbb0d6dc2e78343bf4f842fe2eee85c980ae04d4e01788356b6eab6 0.0s => => naming to docker.io/em-pub-dash-prod/frontend 0.0s ``` ``` dashboard_1 | Starting up http-server, serving ./ dashboard_1 | dashboard_1 | http-server version: 14.1.1 dashboard_1 | dashboard_1 | http-server settings: dashboard_1 | CORS: disabled dashboard_1 | Cache: 3600 seconds dashboard_1 | Connection Timeout: 120 seconds dashboard_1 | Directory Listings: visible dashboard_1 | AutoIndex: visible dashboard_1 | Serve GZIP Files: false dashboard_1 | Serve Brotli Files: false dashboard_1 | Default File Extension: none dashboard_1 | dashboard_1 | Available on: dashboard_1 | http://127.0.0.1:8080 dashboard_1 | http://172.20.0.4:8080 dashboard_1 | Hit CTRL-C to stop the server dashboard_1 | dashboard_1 | [2022-09-07T20:57:57.039Z] "GET /" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0" dashboard_1 | (node:1) [DEP0066] DeprecationWarning: OutgoingMessage.prototype._headers is deprecated dashboard_1 | (Use `node --trace-deprecation ...` to show where the warning was created) dashboard_1 | [2022-09-07T20:57:57.127Z] "GET /client/css/nrel.application.min.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0" dashboard_1 | [2022-09-07T20:57:57.128Z] "GET /client/css/bootstrap.min.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0" dashboard_1 | [2022-09-07T20:57:57.129Z] "GET /client/css/dashboard.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0" ``` The resulting web page is displayed correctly
|
Testing done for webapp: BuildsRunsRenders
|
…ronments
In both:
- use the `dev-only` image and clone from Andrew's repo
- this is the workaround for
e-mission#53 until
e-mission/e-mission-server#875 is merged
In the dev files:
- force a version for the http server
- build the viz_scripts image instead of using a pinned image
- This subsumes e-mission#53
- Since the context is now `viz_scripts` and not `viz_scripts/docker` to
maintain consistency with the prod server, change the paths to the various files
In the prod files:
- Remove the mounted code directories
- Change the dockerfiles to copy files into the image
Testing done: This builds, but DOES NOT WORK
Checking this in temporarily so that we can highlight the fix when we do get it to work
```
Building notebook-server
[+] Building 865.9s (20/20) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 973B 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/emission/e-mission-server.dev.server-only:4.0.0 0.0s
=> CACHED [1/8] FROM docker.io/emission/e-mission-server.dev.server-only:4.0.0 0.0s
=> [internal] load build context 0.1s
=> => transferring context: 112.46kB 0.0s
=> CACHED [ 2/15] ADD docker/environment36.dashboard.additions.yml / 0.0s
=> CACHED [ 3/15] RUN bash -c "/clone_server.sh" 0.0s
=> CACHED [ 4/15] RUN /bin/bash -c "cd e-mission-server && source setup/activate.sh && conda env update --name emission --file setup/environment36.notebook.additions.yml" 0.0s
=> CACHED [ 5/15] RUN /bin/bash -c "cd e-mission-server && source setup/activate.sh && conda env update --name emission --file /environment36.dashboard.additions.yml" 0.0s
=> [ 6/15] RUN mkdir -p /usr/src/app/saved-notebooks 0.4s
=> [ 7/15] WORKDIR /usr/src/app/saved-notebooks 0.0s
=> [ 8/15] COPY auxiliary_files . 0.0s
=> [ 9/15] COPY bin . 0.1s
=> [10/15] COPY conf . 0.1s
=> [11/15] COPY *.ipynb . 0.1s
=> [12/15] COPY *.py . 0.1s
=> [13/15] ADD docker/start_notebook.sh /usr/src/app/start_notebook.sh 0.0s
=> [14/15] RUN chmod u+x /usr/src/app/start_notebook.sh 0.5s
=> [15/15] ADD docker/crontab /usr/src/app/crontab 0.1s
=> exporting to image 35.9s
=> => exporting layers 35.9s
=> => writing image sha256:61830a6f2563e657350d435c6c512f32f31c0c6e1cc13a9ad5a9bc3b57d66360 0.0s
=> => naming to docker.io/em-pub-dash-prod/viz-scripts 0.0s
Building dashboard
[+] Building 2.9s (10/10) FINISHED
```
```
notebook-server_1 | /usr/src/app/e-mission-server /usr/src/app/saved-notebooks
notebook-server_1 | DB host = db
notebook-server_1 | cp: cannot stat 'e-mission-server/conf/storage/db.conf': No such file or directory
notebook-server_1 | Web host = 0.0.0.0
notebook-server_1 | /usr/src/app/saved-notebooks
notebook-server_1 | cat: saved-notebooks/conf/storage/db.conf: No such file or directory
notebook-server_1 | /usr/src/app/start_notebook.sh: line 27: pushd: e-mission-server: No such file or directory
notebook-server_1 | /usr/src/app/start_notebook.sh: line 29: setup/setup.sh: No such file or directory
```
|
Testing for the notebooks: Builds but does not run |
I'm not sure which of these actually fixed the issue, but I think it may be the WORKDIR because I `docker exec`ed into the container and I did see an e-mission-server directory even before the change. ``` $ docker exec -it em-public-dashboard_notebook-server_1 /bin/bash root@eae107992c4a:/usr/src/app# ls crontab e-mission-server saved-notebooks start_notebook.sh root@eae107992c4a:/usr/src/app# cd e-mission-server/ root@eae107992c4a:/usr/src/app/e-mission-server# ls LICENSE conf emission runAllTests.sh OpenSourceLicenses.md docs figs runIndividualTests.sh README.md e-mission-ipy.bash front runSingleTestDebug.sh ThirdPartyContributors.md e-mission-jupyter.bash miniconda.sh setup Timeseries_Sample.ipynb e-mission-locust.bash modeify.md zephyr bin e-mission-py.bash runAllTests.bat ``` Testing done: - Images build properly on dev ``` db uses an image, skipping dashboard uses an image, skipping Building notebook-server [+] Building 0.4s (14/14) FINISHED => [internal] load build definition from Dockerfile.dev 0.0s => => transferring dockerfile: 47B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/emission/e-mission-server.dev.server-only:4.0.0 0.0s => [1/9] FROM docker.io/emission/e-mission-server.dev.server-only:4.0.0 0.0s => [internal] load build context 0.0s => => transferring context: 1.59kB 0.0s => CACHED [2/9] ADD docker/environment36.dashboard.additions.yml / 0.0s => CACHED [3/9] RUN /bin/bash -c "/clone_server.sh" 0.0s => CACHED [4/9] WORKDIR /usr/src/app 0.0s => CACHED [5/9] RUN /bin/bash -c "cd e-mission-server && source setup/activate.sh && conda env update --name emission --file setup/environment36.notebook.additions.yml" 0.0s => CACHED [6/9] RUN /bin/bash -c "cd e-mission-server && source setup/activate.sh && conda env update --name emission --file /environment36.dashboard.additions.yml" 0.0s => [7/9] ADD docker/start_notebook.sh /usr/src/app/start_notebook.sh 0.0s => [8/9] RUN chmod u+x /usr/src/app/start_notebook.sh 0.2s => [9/9] ADD docker/crontab /usr/src/app/crontab 0.0s => exporting to image 0.0s => => exporting layers 0.0s => => writing image sha256:7370fc9abb3c1afa3bb288b4c8dc50871f423a6aaaa7207a043a559524005286 0.0s => => naming to docker.io/em-pub-dash-dev/viz-scripts 0.0s ``` - Images build properly on prod ``` [+] Building 0.2s (9/9) FINISHED => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 42B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/danjellz/http-server:1.4 0.0s => [1/4] FROM docker.io/danjellz/http-server:1.4 0.0s => [internal] load build context 0.0s => => transferring context: 1.76kB 0.0s => CACHED [2/4] RUN mkdir -p /public 0.0s => CACHED [3/4] COPY ../client /public/client 0.0s => CACHED [4/4] COPY ../*.html /public 0.0s => exporting to image 0.0s => => exporting layers 0.0s => => writing image sha256:ff0154b07bbb0d6dc2e78343bf4f842fe2eee85c980ae04d4e01788356b6eab6 0.0s => => naming to docker.io/em-pub-dash-prod/frontend 0.0s ``` Starts properly ``` notebook-server_1 | Running notebook in docker, change host:port to localhost:47962 in the URL below notebook-server_1 | [I 23:09:23.113 NotebookApp] Serving notebooks from local directory: /usr/src/app/saved-notebooks notebook-server_1 | [I 23:09:23.113 NotebookApp] Jupyter Notebook 6.4.12 is running at: notebook-server_1 | [I 23:09:23.113 NotebookApp] http://b7e5a7ce688f:8888/?token=e245f4f98b2c3f7e39d33132c9ec85a68b9434f4f41f8a44 notebook-server_1 | [I 23:09:23.113 NotebookApp] or http://127.0.0.1:8888/?token=e245f4f98b2c3f7e39d33132c9ec85a68b9434f4f41f8a44 notebook-server_1 | [I 23:09:23.113 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). notebook-server_1 | [C 23:09:23.116 NotebookApp] notebook-server_1 | notebook-server_1 | To access the notebook, open this file in a browser: notebook-server_1 | file:///root/.local/share/jupyter/runtime/nbserver-25-open.html notebook-server_1 | Or copy and paste one of these URLs: notebook-server_1 | http://b7e5a7ce688f:8888/?token=e245f4f98b2c3f7e39d33132c9ec85a68b9434f4f41f8a44 notebook-server_1 | or http://127.0.0.1:8888/?token=e245f4f98b2c3f7e39d33132c9ec85a68b9434f4f41f8a44 notebook-server_1 | [I 23:09:42.201 NotebookApp] 302 GET /?token=e245f4f98b2c3f7e39d33132c9ec85a68b9434f4f41f8a44 (172.24.0.1) 0.620000ms ``` And we can see the notebooks in the browser.
|
The notebook containers also work properly now on devon prodon startin browser
|
|
@aGuttman, if you can review this, I can merge and unblock cloud services |
Closed
This is the shared volume where the viz scripts save the plots and the frontend reads them. The `docker-compose` files map an existing host directory to the specified volume. This is as per request from NREL IT > you need to define that in Dockerfile, not in docker compose, we are not using docker compose for deploy on AWS. > yeah, simply add VOLUME plots in your dockerfile Testing done: - Rebuilt the images ``` $ docker-compose -f docker-compose.yml build db uses an image, skipping Building notebook-server [+] Building 0.2s (22/22) FINISHED => => naming to docker.io/em-pub-dash-prod/viz-scripts 0.0s Building dashboard [+] Building 0.1s (9/9) FINISHED => => naming to docker.io/em-pub-dash-prod/frontend 0.0s ``` Restarted the containers ``` $ docker-compose -f docker-compose.yml up Starting em-public-dashboard_db_1 ... done Recreating em-public-dashboard_notebook-server_1 ... done Recreating em-public-dashboard_dashboard_1 ... done ``` logged in to the container and checked ``` $ docker exec -it em-public-dashboard_notebook-server_1 /bin/bash root@dfd9c5b45eb0:/usr/src/app# ls /plots average_miles_mode_confirm.png average_miles_mode_confirm2_2021_10_vail.png average_miles_mode_confirm_.png ... ```
+ Ensure that the production docker-compose runs the scripts periodically (CRON=TRUE) instead of starting an interactive juypter terminal + Change the set of scripts that are run in the crontab to account for the refactoring into generic and mode-specific notebooks. Split occured in 9e1745e @zackAemmer, note that notebook changes also have to be reflected in the crontab for them to be run properly as part of the nightly update. Happy to walk you through how the public dashboard works now to add to your repertoire of system designs. Testing done: The cronjobs are launched but fail with the error ``` python: can't open file 'bin/update_mappings.py': [Errno 2] No such file or directory python: can't open file 'bin/generate_plots.py': [Errno 2] No such file or directory python: can't open file 'bin/generate_plots.py': [Errno 2] No such file or directory python: can't open file 'bin/update_mappings.py': [Errno 2] No such file or directory python: can't open file 'bin/update_mappings.py': [Errno 2] No such file or directory python: can't open file 'bin/generate_plots.py': [Errno 2] No such file or directory ... ```
Without this change, the top level directories were not preserved while copying files into the docker container. Unlike *nix, COPY a . when a is a directory copies all the files _in_ a into the current directory. ``` $ ls saved-notebooks/ analysis generic_timeseries.ipynb plots.py clients log purpose_labels.csv conf mapping_dictionaries.ipynb scaffolding.py energy_calculations.ipynb mode_labels.csv storage energy_intensity.csv mode_specific_metrics.ipynb update_mappings.py generate_plots.py mode_specific_timeseries.ipynb variation_across_individuals.ipynb generic_metrics.ipynb net ``` Fixed by specifying a target directory, as in `COPY a ./a` After this change, the directories are retained ``` $ ls saved-notebooks/ auxiliary_files generic_metrics.ipynb mode_specific_timeseries.ipynb bin generic_timeseries.ipynb plots.py conf mapping_dictionaries.ipynb scaffolding.py energy_calculations.ipynb mode_specific_metrics.ipynb variation_across_individuals.ipynb ``` And the notebooks now run without errors ``` (emission) /usr/src/app/saved-notebooks# python bin/generate_plots.py generic_metrics.ipynb prepilot --date 2022 04 Successfully downloaded config with version 1 for Staging environment for testing programs only and data collection URL https://openpath-stage.nrel.gov/api/ Running at 2022-09-08T06:23:03.828275+00:00 with args Namespace(date=[2022, 4], plot_notebook='generic_metrics.ipynb', program='prepilot') ``` And generate plots ``` ls /plots/*2022* /plots/average_miles_mode_confirm_2022_04_prepilot.png /plots/average_miles_mode_confirm_2022_04_prepilot.txt /plots/miles_mode_confirm_2022_04_prepilot.png /plots/miles_mode_confirm_2022_04_prepilot.txt /plots/ntrips_commute_mode_confirm_2022_04_prepilot.png /plots/ntrips_commute_mode_confirm_2022_04_prepilot.txt /plots/ntrips_mode_confirm_2022_04_prepilot.png /plots/ntrips_mode_confirm_2022_04_prepilot.txt /plots/ntrips_per_day_2022_04_prepilot.png /plots/ntrips_per_day_2022_04_prepilot.txt /plots/ntrips_per_weekday_2022_04_prepilot.png /plots/ntrips_per_weekday_2022_04_prepilot.txt /plots/ntrips_purpose_2022_04_prepilot.png /plots/ntrips_purpose_2022_04_prepilot.txt /plots/ntrips_under10miles_mode_confirm_2022_04_prepilot.png /plots/ntrips_under10miles_mode_confirm_2022_04_prepilot.txt ``` which are displayed correct in the UI
|
Current testing:
|
- Define a new environment variable called `STUDY_CONFIG` that can be modified for each program - In `generate_plots.py` read the study config from the environment, falling back to `stage-program` Bonus fix: Improve the debuggability of the script by printing out the URL before downloading from it, and exiting if the download fails @zackAemmer FYI in case you want to learn how to pass in variables to a docker container
Javascript browsers cannot, for security reasons, read environment variables. There are workarounds using config files that are even automatically generated using webpack, but we are not using webpack now and are unlikely to use it in the future given that this is a temporary solution. So we use the same approach as we do on the phone to infer the study name from the URL. However, we cannot test this functionality easily on dev, because the hostname is always "localhost". Looking at search params only in that case to finish testing. Testing done: - Accessed frontend using http://localhost:3274/, saw the staging config - Accessed it using http://localhost:3274/?study_config=durham, saw the durham config
|
Testing screenshots: Accessed frontend using http://localhost:3274/, saw the staging config
Accessed it using http://localhost:3274/?study_config=durham, saw the durham config
|
|
@aGuttman @zackAemmer if either of you can review this in the next hour or so that would be good. Each commit lists the testing done for its changes, and I have also added testing snapshots throughout the PR. |
… into create_prod_docker_compose Resolved the merge conflict in `crontab` by retaining the new set of notebooks but passing in `default` per e-mission@86160ba
|
Resolved the merge conflict in |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.