Multiple devstack releases with one machine

[ztraboo]

This is the start of changing devstack to accommodate multiple releases (master, open-release-name) as mentioned here by @kdmccormick
https://discuss.openedx.org/t/docker-devstack-multiple-releases-one-machine/1902/4

Replace all the hard-coded container references like:
docker [exec|run|...] edx.devstack.<service>

with calls to docker-compose:
docker-compose [exec|run|...] <service>

[openedx-webhooks]

Thanks for the pull request, @ztraboo! I've created OSPR-4468 to keep track of it in JIRA. JIRA is a place for product owners to prioritize feature reviews by the engineering development teams.

Feel free to add as much of the following information to the ticket:

• supporting documentation
• edx-code email threads
• timeline information ("this must be merged by XX date", and why that is)
• partner information ("this is a course on edx.org")
• any other information that can help Product understand the context for the PR

All technical communication about the code itself will still be done via the GitHub pull request interface. As a reminder, our process documentation is here.

[ztraboo]

@kdmccormick I recent made this change to accommodate automatically naming the Docker compose project based on release.
https://github.com/edx/devstack/pull/532/files#diff-86df7081aaade96499ca968414b07e48

[kdmccormick]

@ztraboo Thanks for the PR! I'll take a look when I can.

Apologies in advance if the review is slow-moving. Devstack isn't my primary work, so I've been doing the best I can to fit in fixes and reviews around other projects.

[natabene]

@ztraboo Thank you for your contribution. @kdmccormick This is ready for your review.

[kdmccormick]

Wow, this must have been a lot of finding-and-replacing. Nice!

Only a couple of comments from me. I still need to test this on my own devstack -- Hopefully I'll be able to do that on Monday. When I do, I'll let you know how it works for me.

[ztraboo]

Wow, this must have been a lot of finding-and-replacing. Nice!

Only a couple of comments from me. I still need to test this on my own devstack -- Hopefully I'll be able to do that on Monday. When I do, I'll let you know how it works for me.

@kdmccormick A lot of find and fix was more like it. Yeah there were several files touched so hopefully that doesn't complicate this update.

[kdmccormick]

@ztraboo Commenting to let you know I haven't forgotten about this. I upgraded to Ubuntu 20.04 a couple days ago and I'm still working getting my development environment set back up.

[arbrandes]

@ztraboo, @kdmccormick, just a quick note that I'm about to test this on an Ubuntu 19.10 box, where there's already a master devstack installed, for the express purpose of testing juniper.rc2. Fingers crossed! If this works, it'll save me many hours of devstack rebuilding.

[arbrandes]

...aaand it worked beautifully. +1 from me.

[arbrandes]

How about this instead:

container_name: "${COMPOSE_PROJECT_NAME:-edx}.devstack.lms"

It would make container names more easily predictable by external services. I'm thinking of blockstore devstacks, for instance, which would otherwise need complicated changes due to this PR.

Tagging @bradenmacdonald and @kdmccormick for thoughts.

[arbrandes]

Or maybe:

container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.lms"

For backwards compatibility with a minimal diff. You'd then be able to ditch the network alias below, and still have predictable names.

[bradenmacdonald]

I'd prefer to focus on just making blockstore (and blockstore-test) an (optional?) part of the regular edX devstack distribution, vs. adapting its idiosyncratic devstack setup.

CC @ormsbee - I'm not clear on the path to make that happen.

[ztraboo]

@arbrandes My changes are based on what @kdmccormick mentioned here https://discuss.openedx.org/t/docker-devstack-multiple-releases-one-machine/1902/4 since docker compose environment variable COMPOSE_PROJECT_NAME supports this out of the box with prefixing the container names with this project name.

COMPOSE_PROJECT_NAME
Sets the project name. This value is prepended along with the service name to the container on start up. For example, if your project name is myapp and it includes two services db and web, then Compose starts containers named myapp_db_1 and myapp_web_1 respectively.
https://docs.docker.com/compose/reference/envvars/#compose_project_name#compose_project_name

I'm using docker compose Aliases to still allow containers to reach each other on the network.
https://docs.docker.com/compose/compose-file/compose-file-v2/#aliases

See example of how Aliases are use here with this PR.
https://github.com/edx/devstack/pull/532/files#diff-d9f40066d6470c3b707e9bbb0ad21f74

[kdmccormick]

@bradenmacdonald I don't know what the specific complexities of adding Blockstore to devstack are, but if you added it like any other service (Registrar is a good example of how to do that), it would by default be an "optional service".

Only the services listed in DEFAULT_SERIVCES in options.mk are pulled, run, and provisioned by default - the others are ignored unless:

• they are specifically referenced, such as in make dev.{up|pull|provision|etc}.blockstore, or
• DEFAULT_SERVICES is overridden in options.local.mk to include more services.

[ztraboo]

@kdmccormick Any update on testing this PR on your end?

[regisb]

@ztraboo I think there is some confusion here. When you change the COMPOSE_PROJECT_NAME, it does not affect how containers address one another inside the network. Thus, setting the COMPOSE_PROJECT_NAME does not imply that you should also define aliases.

[kdmccormick]

Thanks for your patience @ztraboo . I'm doing comprehensive testing right now.

[kdmccormick]

Data separation: looks good!

I did the following:

• Provisioned one devstack and smoke-tested it.
• Provisioned devstack-other and smoke-tested it.
• Made a DB modification in devstack-other and confirmed it doesn't affect devstack.
• Destroyed devstack-other and confirmed it doesn't affect devstack.

Nice 🥇

Backwards-incompatability with container names:

The old container names (edx.devstack.lms, etc.) don't work. I figure that:

...
    networks:
      default:
        aliases:
          - edx.devstack.lms

doesn't actually make a container name alias, it makes a container network hostname alias. So edx.devstack.lms is available as a hostname to containers on the network, but would not work in Docker commands like docker exec edx.devstack.lms and such.

This isn't a dealbreaker, but it would be a breaking change for people who made custom scripts to integrate against devstack. I see two routes to address this:

• Set all the container names to edx.${COMPOSE_PROJECT_NAME}.<service>, thus keeping the default container names the same.
• Make it a breaking change, but notify discuss.openedx.org of the potential breakage a few days before merging it. I could distribute the notification inside edX if you decide to go this route.

Cannot run multiple devstack simultaneously:

You probably realized this, but the hard-coded port numbers we have prevent multiple devstacks from being up at the same time. This isn't a problem, but should probably be made clear in the docs.

[ztraboo]

Data separation: looks good!

• Provisioning one devstack and smoke-testing it.
• Provisioning devstack-other and smoke-testing it.
• Making a DB modification in devstack-other and confirming it doesn't affect devstack.
• Destroying devstack-other and confirming it doesn't affect devstack.

Nice 🥇

Backwards-incompatability with container names:

The old container names (edx.devstack.lms, etc.) don't work. I figure that:

...
    networks:
      default:
        aliases:
          - edx.devstack.lms

doesn't actually make a container name alias, it makes a container network hostname alias. So edx.devstack.lms is available as a hostname to containers on the network, but would not work in Docker commands like docker exec edx.devstack.lms and such.

This isn't a dealbreaker, but it would be a breaking change for people who made custom scripts to integrate against devstack. I see two routes to address this:

• Set all the container names to edx.${COMPOSE_PROJECT_NAME}.<service>, thus keeping the default container names the same.
• Make it a breaking change, but notify discuss.openedx.org of the potential breakage a few days before merging it. I could distribute the notification inside edX if you decide to go this route.

Cannot run multiple devstack simultaneously:

You probably realized this, but the hard-coded port numbers we have prevent multiple devstacks from being up at the same time. This isn't a problem, but should probably be made clear in the docs.

@kdmccormick I'm on board for whatever you recommend for a change here. Let me know if you'd like for me to add back in the container names as you describe above or keep the change and notify discuss.openedx.org accordingly.

See this mentioned above
https://github.com/edx/devstack/pull/532#discussion_r425227932

If we add back in the container names should we remove the network alias? Just let me know what you prefer. I'm ok with either option that you mentioned above. Maybe someone else from the community can chime in here on option they prefer. I'm leaning more toward keeping a fixed container name to avoid issues mentioned here https://github.com/edx/devstack/pull/532#discussion_r425225387

Based on what I read about aliases here it should work but I see your point in that docker command may not work correctly. Docker-compose would though since we're seeing it work with all those changes I made.

https://docs.docker.com/compose/compose-file/compose-file-v2/#aliases

Aliases (alternative hostnames) for this service on the network. Other containers on the same network can use either the service name or this alias to connect to one of the service’s containers. Since aliases is network-scoped, the same service can have different aliases on different networks.

cc: @arbrandes @bradenmacdonald @pomegranited

[kdmccormick]

@ztraboo

@kdmccormick I'm on board for whatever you recommend for a change here. Let me know if you'd like for me to add back in the container names as you describe above or keep the change and notify discuss.openedx.org accordingly.

Of the two options, I recommend adding the container names, mainly because it will make this change less controversial (and therefore more likely to merge soon :).

If we add back in the container names should we remove the network alias? Just let me know what you prefer.

I do recommend removing the network aliases, as they are redundant with the <service>.devstack.edx network names, which are isolated in the compose project network.

Based on what I read about aliases here it should work but I see your point in that docker command may not work correctly. Docker-compose would though since we're seeing it work with all those changes I made.

The docker -> docker-compose changes you made definitely prevents the container naming scheme from breaking anything in the edx/devstack repository, which is awesome and enables the multiple devstacks on one machine. However, I know of developers who write their own custom docker-compose files and scripts to integrate their service with devstack, many of which use hard-coded docker commands and container names. Changing the default container names would break those.

[ztraboo]

@kdmccormick I committed this change edx@626b46c, however, the test passed in Travis CI but when I try to re-provision my devstack for master branch it looks like MongoDB shuts down after launching and I get an infinite loop when here. Any advice?
https://github.com/CUCWD/devstack/blob/ztraboo/multiple-releases-one-machine/provision.sh#L144-L149

devstack > mongo > edx.devstack.mongo

2020-05-18T22:51:48.414+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=mongo.devstack.edx
2020-05-18T22:51:48.418+0000 I CONTROL  [initandlisten] db version v3.6.17
2020-05-18T22:51:48.418+0000 I CONTROL  [initandlisten] git version: 3d6953c361213c5bfab23e51ab274ce592edafe6
2020-05-18T22:51:48.418+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2020-05-18T22:51:48.418+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2020-05-18T22:51:48.418+0000 I CONTROL  [initandlisten] modules: none
2020-05-18T22:51:48.418+0000 I CONTROL  [initandlisten] build environment:
2020-05-18T22:51:48.418+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
2020-05-18T22:51:48.418+0000 I CONTROL  [initandlisten]     distarch: x86_64
2020-05-18T22:51:48.418+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2020-05-18T22:51:48.418+0000 I CONTROL  [initandlisten] options: { net: { bindIpAll: true }, storage: { engine: "wiredTiger", journal: { enabled: false }, mmapv1: { smallFiles: true } } }
2020-05-18T22:51:48.420+0000 I STORAGE  [initandlisten] 
2020-05-18T22:51:48.420+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-05-18T22:51:48.420+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2020-05-18T22:51:48.420+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3471M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),,log=(enabled=false),
2020-05-18T22:51:48.973+0000 I STORAGE  [initandlisten] WiredTiger message [1589842308:973099][1:0x7ff4f7a52a40], txn-recover: Set global recovery timestamp: 0
2020-05-18T22:51:48.991+0000 W STORAGE  [initandlisten] Detected configuration for non-active storage engine mmapv1 when current storage engine is wiredTiger
2020-05-18T22:51:48.991+0000 I CONTROL  [initandlisten] 
2020-05-18T22:51:48.991+0000 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2020-05-18T22:51:48.991+0000 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2020-05-18T22:51:48.991+0000 I CONTROL  [initandlisten] 
2020-05-18T22:51:48.995+0000 F CONTROL  [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details.
2020-05-18T22:51:48.995+0000 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2020-05-18T22:51:48.995+0000 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2020-05-18T22:51:48.996+0000 I REPL     [initandlisten] shutdown: removing all drop-pending collections...
2020-05-18T22:51:48.996+0000 I REPL     [initandlisten] shutdown: removing checkpointTimestamp collection...
2020-05-18T22:51:48.996+0000 I STORAGE  [initandlisten] WiredTigerKVEngine shutting down
2020-05-18T22:51:49.020+0000 I STORAGE  [initandlisten] WiredTiger message [1589842309:20263][1:0x7ff4f7a52a40], txn-recover: Set global recovery timestamp: 0
2020-05-18T22:51:49.056+0000 I STORAGE  [initandlisten] shutdown: removing fs lock...
2020-05-18T22:51:49.056+0000 I CONTROL  [initandlisten] now exiting
2020-05-18T22:51:49.056+0000 I CONTROL  [initandlisten] shutting down with code:62