-
-
Notifications
You must be signed in to change notification settings - Fork 28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add docker-compose setup #28
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome! Thanks for sharing!
Yes, more 👀 and manual testing for this PR would be appreciated.
If everything goes well with the functionality & working state of this one, docker-compose
will be moved to st2-docker
project and eventually replace the https://github.com/stackstorm/st2-docker repo contents.
This way we could give a second life for the https://github.com/stackstorm/st2-docker/ which had no contributor for a long time with unsupported Ubuntu Trusty.
As #16 merged, this deployment is based on new |
Sorry, had to post this comment again as I used the wrong account before. This one was originally posted as direct reply to the issue and before armab's first reply. "I really like the idea and would like to contribute to it. I was just working on an extension of st2-docker that adds prometheus and grafana with preprovisioned dashboards, etc. so that the user has a full st2 environment incl. monitoring & visualization which can be helpful during the evaluation. Can we summarize here what it takes to get this merged? I'll test the PR in my lab on the weekend and am happy to contribute and push the docker-compose setup. :)" |
I spent some time this evening to play around with the setup and it works just great. No flaws or issues so far. Things I tried:
I'll spend some more time with this setup in the next days and keep you posted. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added a few notes while most of them are basically about the wording. The most important one is the one regarding the exposed st2web HTTP port already being bound to 0.0.0.0.
I found an issue with
ExampleRunning containers(If you want to try it on your own, the command to get the output as md table is
docker-compose restartWorks as expected and ends up in:
This is with a delay of 32 minutes but it shows that all containers were restarted at the same time:
Now running containers
So the IPs changed. The errorst2web access log if the upstreams aren't resolved to the new IP:
Verification that the DNS resolution itself works
SolutionInitially I assumed an issue with the order of the containers being restarted or something like that which takes more time but I actually now the solution already because the same issue hit us at the company like 6 months ago. So it was pretty helpful to write everything down. :) I'll keep the content above in the comment as this might be helpful for others when troubleshooting something. So, the issue is that nginx resolves all the upstream hosts on startup and caches these IP addresses forever with the default configuration. The fix
So the new
For some reason all 3 changes (setting the resolver and defining + using the variable) are required to enforce that nginx resolves the upstream hosts at runtime. So it's not enough to set just the resolver for example. Helpful links on this topic: I noticed that this issue is not just an issue with the docker-compose branch and will create a PR for st2-dockerfiles, too. |
volumes: | ||
- stackstorm-mongodb:/data/db | ||
dns_search: . | ||
rabbitmq: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can see that rabbitmq starts up delayed and makes st2api fail to start. We need another healtcheck here:
2020-05-14 09:43:40,020 DEBUG [-] Using Python: 2.7.12 (/opt/stackstorm/st2/bin/python)
2020-05-14 09:43:40,020 DEBUG [-] Using config files: /etc/st2/st2.conf,/etc/st2/st2.docker.conf,/etc/st2/st2.user.conf
2020-05-14 09:43:40,021 DEBUG [-] Using logging config: /etc/st2/logging.api.gunicorn.conf
2020-05-14 09:43:40,063 INFO [-] Connecting to database "st2" @ "mongo:27017" as user "None".
2020-05-14 09:43:40,115 INFO [-] Successfully connected to database "st2" @ "mongo:27017" as user "None".
2020-05-14 09:44:06,143 ERROR [-] Connections to RabbitMQ cannot be re-established: [Errno 111] ECONNREFUSED
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixing my comment, we don't need a healthcheck. We need the "wait-for-it.sh" here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Or, adding restart: on-failure
on st2api (and others) solves the problem the as well. But I'd prefer wait-for-it for a cleaner startup.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I haven't seen outright startup failure in my testing. Most of the containers depend on rabbitmq and most will throw ECONNREFUSED during cold (re)start, but once RabbitMQ initializes they all picked right up. Before I setup the "depends on", order the logs were spammed with ECONNREFUSED.
Are you seeing st2api error out and then you have to restart it by hand? We could be seeing different results based on speed and cpu of the host.
stackstorm-ha containers behave in a manner similiar to the suggested restart: on-failure
option. When I do a deploy, most of the containers may end up restarting 2-8 times while waiting for rabbitmq to get its act together (and if it doesn't, they go into CrashLoopBackOff).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had an eye on this as well. My oberservations basically confirm what @ytjohn said.
- The rabbitmq container restarts faster than the st2api service. Therefore we see ECONNREFUSED during the shutdown of st2api.
- if the st2api container can not connect to the rabbitmq host for any reason (I prodoced ECONNREFUSED errors as well as EHOSTUNREACH errors) st2api recovers on it's own as soon as the connection works again
No manual intervention was used on any of the test cases described above.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@winem @ytjohn I don't know why I'm facing another scenario unlike yours. I'm feeling this is an improvement done in st2 3.3-dev.
Let me write my environmental details:
- ST2_VERSION=3.2.0 (I think thats different from yours)
- MBP 2017
- OSX 10.14.6 Mojave
- Docker for Mac 2.3.2.0 with Engine 19.03.8 and Compose 1.25.5
The interesting part is that, I had to lower the Docker for mac VM settings to reproduce the issue. To clarify, I made 1 CPU, 2 GB Memory and 1 GB of Swap and st2api failed on boot.
After st2api fails on boot, it didn't retry connecting unlike yours, because the container itself is dead, as you can see in the below shell output:
docker-compose ps
Name Command State Ports
------------------------------------------------------------------------------------------------------------------------------------
stackstorm-compose_mongo_1 docker-entrypoint.sh mongod Up 27017/tcp
stackstorm-compose_rabbitmq_1 docker-entrypoint.sh rabbi ... Up 25672/tcp, 4369/tcp, 5671/tcp, 5672/tcp
stackstorm-compose_redis_1 docker-entrypoint.sh redis ... Up 6379/tcp
stackstorm-compose_st2actionrunner_1 /opt/stackstorm/st2/bin/st ... Up
stackstorm-compose_st2api_1 /opt/stackstorm/st2/bin/st ... Exit 1
stackstorm-compose_st2auth_1 /opt/stackstorm/st2/bin/st ... Up 9100/tcp
stackstorm-compose_st2client_1 /bin/bash Exit 0
stackstorm-compose_st2garbagecollector_1 /opt/stackstorm/st2/bin/st ... Up
stackstorm-compose_st2notifier_1 /opt/stackstorm/st2/bin/st ... Up
stackstorm-compose_st2resultstracker_1 /opt/stackstorm/st2/bin/st ... Up
stackstorm-compose_st2rulesengine_1 /opt/stackstorm/st2/bin/st ... Up
stackstorm-compose_st2scheduler_1 /opt/stackstorm/st2/bin/st ... Up
stackstorm-compose_st2sensorcontainer_1 /opt/stackstorm/st2/bin/st ... Up
stackstorm-compose_st2stream_1 /opt/stackstorm/st2/bin/st ... Up 9102/tcp
stackstorm-compose_st2timersengine_1 /opt/stackstorm/st2/bin/st ... Up
stackstorm-compose_st2web_1 /bin/bash -c if [ ${ST2WEB ... Up (unhealthy) 443/tcp, 127.0.0.1:8000->80/tcp
stackstorm-compose_st2workflowengine_1 /opt/stackstorm/st2/bin/st ... Up
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've tried some more, including shutting down mongo. With mongo down, st2scheduler will eventually fail after so many retries.
I set all of the containers to restart: on-failure
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@TunaYagci, I am wondering if this may be something specific to Mac or your docker version. But this is as unqualified as it gets.
I'll do a bunch of tests with the latest changes from @ytjohn later tonight. The most recent ones on Friday and Saturday evening went fine.
@TunaYagci Pushed a bunch of changes, can you please review and resolve conversations if they're fixed? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ytjohn thanks for everything!
I think I don't have the permission to resolve conversations, but all looks good to me! |
@ytjohn I wanted to let you know that current version doesn't support "Storing Secrets", you're getting "MESSAGE: Crypto key not found" error while storing secrets with I've looked up the stackstorm-ha build but couldn't find a solution there as well. There are 2 open issues about it, here and there. editing: The single all-in-one stackstorm image actually thought about this: # Setup symmetric crypto key for datastore
RUN mkdir -p /etc/st2/keys \
&& st2-generate-symmetric-crypto-key --key-path /etc/st2/keys/datastore_key.json \
&& usermod -a -G st2 st2 && chgrp st2 /etc/st2/keys && chmod o-r /etc/st2/keys \
&& chgrp st2 /etc/st2/keys/datastore_key.json && chmod o-r /etc/st2/keys/datastore_key.json \
&& crudini --set /etc/st2/st2.conf keyvalue encryption_key_path /etc/st2/keys/datastore_key.json \
&& crudini --set /etc/st2/st2.conf auth enable True \
&& crudini --set /etc/st2/st2.conf content packs_base_paths /opt/stackstorm/packs.dev
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is awesome! 👍
I just tried it and amazed how nicely it integrates with the existing Docker images, follows the best practices and avoids conflicts with the K8s. That was our ideal goal of having one set of images for both orchestrators.
Thank you for putting it all together for docker-compose
, @ytjohn!
I just left a few minor comments to improve below.
This is exciting to see! QQ: Is this intended to replace https://github.com/StackStorm/st2-docker? |
Yes, it is. :) |
I wasn't able to follow up on this for a few days, but this morning I pushed a few improvements:
|
…ompose environments
printf " It's recommended to use \033[1mst2client\033[0m container to work with StackStorm cluster.\n" | ||
# Is K8s environment | ||
if [ -n "$KUBERNETES_PORT" ]; then | ||
printf " \033[1mWarning!\033[0m Do not edit configs, packs or any content inplace as they will be overridden. Modify Helm values.yaml instead!\n" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added adjustments for the welcome messages so it works properly in both K8s and docker-compose environments.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work! 👍
Thanks a lot @ytjohn for dedication to bring docker-compose and @winem @TunaYagci @trstruth for review, feedback and testing. Awesome team effort here!
FYI we're including docker-compose deployment in this repo as a temporary home.
Later it will be moved to the https://github.com/stackstorm/st2-docker and replace entire repository.
Docker-compose deployment based on one-service-per-container Docker images (https://github.com/StackStorm/st2-dockerfiles) with Ubuntu 18.04 LTS as a base OS and Python 3.