Docker-compose dev environment #1133

hornc · 2018-09-17T03:07:00Z

Closes #1063
makes some changes related to ;) #1064

and hopefully #1065

Description:

Clarifies the split between olbase and oldev
- olbase is a single application standard image that will run OL as the openlibrary user, needs to be deployed sensibly to be useful
- oldev has whatever else we need to operate our dev environment, i.e. pre-populated dev db, other services and tools. oldev may, and should, be reduced over time as we split out services.
Volume mounts local dev directory and creates docker compose volumes to retain the js + css that was created at build time, and the vendor dir that is used to fetch git submodules, which is also populated on build.
Uses a new openlibray-docker.yml config file to not interfere with vagrant, and avoids using sed to fix things. I'd suggested we rename this back to openlibrary.yml when we fully remove all traces of vagrant.
Adds the gunicorn --reload option in dev so code changes are automatically detected and reloaded 🎉

Usage

With the volume mount in place and the gunicorn --reload option a developer can edit code locally and the webserver will automatically reload the code on save.

The instructions in docker/README.md are 👌

mekarpeles · 2018-09-17T03:33:19Z

conf/openlibrary-docker.yml

+    editions_view: http://127.0.0.1:5984/editions/_fti/_design/seeds/by_seed
+
+admin:
+    olsystem_root: /home/noufal/projects/OL/olsystem/


I think this can be pruned?

It is used in code here:

openlibrary/openlibrary/plugins/admin/code.py

Line 538 in 75452cf

f = open("%s/olsystem.yml"%config.admin.olsystem_root)

but I can't get the page /admin/services to display. Looks like it is not registered in the setup() further down in that file. Shall I prune the entire service_status class?

mekarpeles · 2018-09-17T03:34:02Z

conf/openlibrary-docker.yml

+
+libraries_admin_email: admin@dev-instance
+
+graphite_base_url: http://ol-admin.us.archive.org:7080


I don't think we want developer instances recording to graphite (analytics)

hornc · 2018-09-17T06:33:19Z

conf/openlibrary-docker.yml

+libraries_admin_email: admin@dev-instance
+
+graphite_base_url: http://ol-admin.us.archive.org:7080
+geoip_database: var/cache/GeoLiteCity.dat


@mekarpeles what is this for, I don't see it in olsystem?

Used in https://github.com/internetarchive/openlibrary/blob/75452cf6f0e3c23e1030395c93e232686ebaa417/openlibrary/core/geo_ip.py , but there is no GeoLiteCity.dat at any of the locations

conf/openlibrary-docker.yml

mekarpeles · 2018-09-19T07:33:05Z

lgmt, requesting one more pair of eyeballs; there's a lot here. Config especially looks much better

hornc · 2018-09-19T20:25:57Z

Thanks @mekarpeles ! @cdrini, could you please do a quick review and let me know if you see anything that is alarming, or could do with improvement? I'd like to get these change in soon as I have another (import API, not docker) PR that is enabled by these changes :)

cdrini · 2018-09-19T20:45:06Z

👍 I can take a look/test it out now! These changes look great, btw :)

cdrini

Looks great! A few small changes. I wouldn't close #1064 yet since I want to discuss the matter a little more (:P), but I'll remove it as a subtask of removing vagrant (assuming no resistance). I do think the split is much clearer as a result of this PR. :)

cdrini · 2018-09-19T21:07:41Z

docker/README.md

@@ -33,8 +33,7 @@ docker-compose stop
 docker-compose rm -v
 ```

-Note: You must build `olbase` first before `oldev`. Currently (August 2018) the division is a bit arbitrary. More should be moved into `olbase` once we clarify
-the production deployment requirements. Currently these docker images are only intented for development environments.
+Note: You must build `olbase` first before `oldev`. `olbase` is intended to be the core Open Library image, acting as a base for production and development. `oldev` adds a test databse and any other tools that are helpful for local development. Currently (Sep 2018) these docker images are only intented for development environments.


Typo: databse

docker/ol-docker-start.sh

cdrini · 2018-09-19T21:24:19Z

docker/ol-docker-start.sh

  --state-file solr-update.offset \
  --ol-url http://web/" &

 # ol server, running in the foreground to avoid exiting container
-su openlibrary -c "authbind --deep scripts/openlibrary-server conf/openlibrary.yml --gunicorn -w4 -t180 -b:80"
+su openlibrary -c "authbind --deep scripts/openlibrary-server $CONFIG --gunicorn --reload -w4 -t180 -b:80"


Can you place the parameters on separate lines, like the preceding command? It would make it a little easier to read. Also replace those short parameter names with the full-length ones.

docker/docker-compose.yml

conf/openlibrary-docker.yml

docker/README.md

cdrini · 2018-09-19T23:06:28Z

I'm not sure if it's because of this PR, but #1051 also no longer appears to be an issue \o/

hornc · 2018-09-20T00:06:20Z

@cdrini if this fixes #1051, that is great! Keep an eye on it, because even though I think I see how this might have fixed it (some of those dirs will no longer be accessing the windows fs, but using the docker volume mounts) I think some of the symlinks will still be mounted from Windows. Test that runningmake all without your workaround doesn't error too. I think there a still some symlinks that will be taken from your local windows dir. At least if tghey are not all fixed, this suggests an approach to move them off the Windows fs.

hornc · 2018-09-21T01:23:53Z

@cdrini I think I have addressed all of your comments! Last one was the comments explaining the volume and bind mounts 👍 🙇‍♂️ Let me know if its good to go!

mekarpeles · 2018-09-21T01:40:48Z

w/ latest changes, this lgtm

cdrini

Haha, found one more typo :P everything else looks good though!

cdrini · 2018-09-21T13:15:34Z

conf/openlibrary.yml

@@ -1,55 +1,32 @@
-# Open Library fastcgi configuration
+# Open Library Docker (dev) configuration


Assuming this should say vagrant? :P

mekarpeles reviewed Sep 17, 2018

View reviewed changes

hornc commented Sep 17, 2018

View reviewed changes

hornc mentioned this pull request Sep 17, 2018

fix account header errors on dev #1135

Merged

hornc added 6 commits September 18, 2018 11:04

better split for olbase and oldev, relates to internetarchive#1064

c31dc9a

volume mount local dev by default in docker compose

270eeb3

use custom docker config

c71b1c1

docker-start to do its own solr-reindex as Makefile has hardcoded config

fa24937

trim unneeded config

6c8357d

allow gunicorn to autoreload changed code, speeds up dev!

5bb7105

hornc force-pushed the docker branch from 8788076 to 5bb7105 Compare September 18, 2018 01:11

hornc commented Sep 18, 2018

View reviewed changes

conf/openlibrary-docker.yml Outdated Show resolved Hide resolved

hornc added 3 commits September 19, 2018 12:16

remove old n/a dev support addresses

e9c748c

remove deprecated keys

9de78b5

remove duplicate+deprecated availability url

e861d5b

hornc changed the title ~~WIP Docker-compose dev environment~~ Docker-compose dev environment Sep 19, 2018

remove TODO now done

2b5cad2

hornc requested a review from cdrini September 19, 2018 20:23

cdrini requested changes Sep 19, 2018

View reviewed changes

hornc added 4 commits September 20, 2018 12:29

tidy up original openlibrary.yml

06cb71f

add Code Updates section

992fe2a

full param names for gunicorn

455dcd3

dd explanation for dev volume and bind mounts

9a49632

hornc force-pushed the docker branch from e03db5a to 9a49632 Compare September 21, 2018 01:18

mekarpeles merged commit c74c400 into internetarchive:master Sep 21, 2018

cdrini reviewed Sep 21, 2018

View reviewed changes

cdrini mentioned this pull request Sep 21, 2018

Symlinks not working when using docker from Windows #1051

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker-compose dev environment #1133

Docker-compose dev environment #1133

hornc commented Sep 17, 2018 •

edited

Loading

mekarpeles Sep 17, 2018

hornc Sep 17, 2018

mekarpeles Sep 17, 2018

hornc Sep 17, 2018

hornc Sep 17, 2018 •

edited

Loading

mekarpeles commented Sep 19, 2018

hornc commented Sep 19, 2018 •

edited

Loading

cdrini commented Sep 19, 2018 •

edited

Loading

cdrini left a comment

cdrini Sep 19, 2018

cdrini Sep 19, 2018

cdrini commented Sep 19, 2018

hornc commented Sep 20, 2018

hornc commented Sep 21, 2018

mekarpeles commented Sep 21, 2018

cdrini left a comment

cdrini Sep 21, 2018


		libraries_admin_email: admin@dev-instance

		graphite_base_url: http://ol-admin.us.archive.org:7080

		@@ -1,55 +1,32 @@
		# Open Library fastcgi configuration
		# Open Library Docker (dev) configuration

Docker-compose dev environment #1133

Docker-compose dev environment #1133

Conversation

hornc commented Sep 17, 2018 • edited Loading

Description:

Usage

mekarpeles Sep 17, 2018

Choose a reason for hiding this comment

hornc Sep 17, 2018

Choose a reason for hiding this comment

mekarpeles Sep 17, 2018

Choose a reason for hiding this comment

hornc Sep 17, 2018

Choose a reason for hiding this comment

hornc Sep 17, 2018 • edited Loading

Choose a reason for hiding this comment

mekarpeles commented Sep 19, 2018

hornc commented Sep 19, 2018 • edited Loading

cdrini commented Sep 19, 2018 • edited Loading

cdrini left a comment

Choose a reason for hiding this comment

cdrini Sep 19, 2018

Choose a reason for hiding this comment

cdrini Sep 19, 2018

Choose a reason for hiding this comment

cdrini commented Sep 19, 2018

hornc commented Sep 20, 2018

hornc commented Sep 21, 2018

mekarpeles commented Sep 21, 2018

cdrini left a comment

Choose a reason for hiding this comment

cdrini Sep 21, 2018

Choose a reason for hiding this comment

hornc commented Sep 17, 2018 •

edited

Loading

hornc Sep 17, 2018 •

edited

Loading

hornc commented Sep 19, 2018 •

edited

Loading

cdrini commented Sep 19, 2018 •

edited

Loading