Repository navigation
Add helper script and related docs #7218
Description
See parent issue for more info/context.
There have been some issues where docker-compose has failed and it's unclear how to fix it.
We should publish a bash script that can be run to check the health of a docker-compose up call. Specifically it should:
- make sure the two containers are up:
haproxy,medic-os - make sure all services in
medic-osare running: API, CouchDB, nginx - offer helpful tips on how to troubleshoot any of the services that may have gotten stuck
- be aware that there may be more than one instance of medic-os running both in running
execcalls the specify container name and that there may be port conflicts - Be sure the admin password to CHT is easy to find. It can some times be set to something unknown (alternately - just set forcefully to the same thing every time)
- publish documentation on docs site on how to use this helper script and it's output.
It'd be great to have the script give a fixed output that users can get used to and easily find the information they're looking for. Just brainstorming here, but this might look something like this:
-------------------------------------------------------------
CHT Instance: Foo Bar Smang
CHT Version: 3.9.1
URL: https://192-168-1-1.my.local-ip.co
Fauxton: http://172.18.0.2:5984/_utils/
Admin Login: Medic
Admin Password: Medic321
+ Medic OS
- horticulturalist - RUNNING ('horticulturalist/horticulturalist' started successfully)
- medic-api - RUNNING ('medic-api/medic-api' started successfully)
- couchdb - RUNNING ('medic-core/couchdb' started successfully)
- nginx - RUNNING ('medic-core/nginx' started successfully)
- openssh - RUNNING ('medic-core/openssh' started successfully)
- medic-couch2pg - RUNNING ('medic-couch2pg/medic-couch2pg' started successfully)
- postgresql - RUNNING ('medic-rdbms/postgresql' started successfully)
- medic-sentinel - RUNNING ('medic-sentinel/medic-sentinel' started successfully)
- cron - RUNNING ('system-services/cron' started successfully)
- syslog - RUNNING ('system-services/syslog' started successfully)
+ HA Proxy
- haproxy - RUNNING (May 11 21:32:59 0df0af7b7c90 haproxy[1]: Proxy couch-backend2 started.)
-------------------------------------------------------------
This is likely TMI and could skip a lot of the granular statuses. However the top part I've had to go looking for a lot and would massively helpful to have easily accessible.
Metadata
Metadata
Assignees
Labels
No labels
Activity
mrjones-plip commentedon May 11, 2021
@binokaryg - thoughts/comments on my requirements above?
binokaryg commentedon May 12, 2021
This looks super helpful, thanks @mrjones-plip.
Can we hide the password unless specified when running the script? We might want to share the output with others and might forget to hide the password, especially if the same password is used elsewhere.
mrjones-plip commentedon May 12, 2021
@binokaryg - Thanks for the feedback! Glad this is looking helpful.
Yup yup, this should be easy enough!
mrjones-plip commentedon May 12, 2021
I've intentionally done a lot of testing of
docker-compose upcalls, and so far this is the error mode I reliably see (haproxy exited with code 1). When checking withcurlit reported:And then:
Details
click to see full docker-compose logs for this
mrjones-plip commentedon Jun 2, 2021
After hitting the above error today, I did a
ctrl + cand then for kicks I did another (press Ctrl+C again to force). When re-runningdocker-compose upI got a new error!medic-osfailed to start:Indeed, nobody is home as expected:
And then:
Details
click to see full docker-compose logs for this
mrjones-plip commentedon Jun 4, 2021
OK! I'm happy to report I believe I have a way to reliably reproduce a startup bug. On Ubuntu 18 with docker 20.10.6, build 370c289 and docker-compose 1.25.4, build 8d51620a I can reliably get an error:
-dso you're attachedctrl + cdocker system prune&&docker volume prunedocker-compose upthis reliably results in:
the fix being to
ctrl + cagain andupagain.Hareet commentedon Jun 6, 2021
@mrjones-plip
A couple things here, all related to entrypoints and networking. Your haproxy container is unable to talk to your medic-os container. As you highlighted, we spoke about how restarts would be the best fix. You can also play around with the restart declaration in the compose template to handle some of the scenarios you will discover.
Entrypoints
The issue above is resolved by entrypoint scripts, and essentially running something like
wait-for-it.shfor your other container's service to come up. We have resolved this in architecture v3 for all the separate containers and services. We still need to thoroughly test that. You should be able to use our test haproxy image that contains the wait for it script for port 5984, but it may break in other untested scenarios. Furthermore, I'd recommend just restarting compose for this particular scenario like you recommended at the bottom and performing further testing for startup issues on our new images, compose architecture.Networking
Say haproxy never resolves medic-os but couchdb is up inside? There is some networking changes across Operating Systems. Your haproxy container isn't able to find the medic-os container in this setup. This is resolved by: https://github.com/medic/medic-infrastructure/blob/master/images/haproxy/haproxy.cfg#L48 and setting that
COUCHDB_HOSTvariable aslocalhostin your docker-compose file. You will also need to declare host networking in your compose file. I'd recommend this approach and just using a range of uncommon ports. This would work on most versions of operations systems. Obviously, we don't recommend this setup for production.The proper way for a "one solution" fix would be to discover networking configuration before docker-compose launches to configure necessary networking variables in the compose file. Something we may think about resolving later.
Also, there's a lot of scripts and systemctl/supervisor status inside medic-os. Those do not correctly reflect the status of the services overtime due to issues with running a large VM inside a container. A lot of what you are solving, is essentially that, and is solved out of the box in arch v3.
mrjones-plip commentedon Jun 7, 2021
@Hareet - thanks for the thoughts on how this might be improved! I very much defer to SRE for what's the best solution to public updates to
docker-compose.ymla la changing networking tohost. I think this would be better than private App Services only internal publishing of a newdocker-compose.yml.My working assumption right now is that SRE is working through the queue of tickets for the larger App Builder effort and trying to see where they fit in schedule wise. Please let me know if this is not the case!
mrjones-plip commentedon Jul 27, 2021
Adding some notes now that I'm getting started on this:
https://YOU-IP-HERE-OK.my.local-ip.co. We'll just enhance the script to match the LAN IP with the router IP!ipis a common enough util? 🤞