[WIP] fix(controller/supervisord): Supervisord to start deis-controller

[developerinlondon]

Currently the Deis Controller uses a bash file for startup. This doesn't deal with
issues when certain parts of the bash file breaks down. Instead this PR will aim to
work on building a supervisord alternative that can monitor the processes (maybe
using superlance)

This is WIP and requires a lot of feedback. Trying to put together a supervisord.conf
file at first.

Closes #1306

[developerinlondon]

The etcd update can be put into supervisord as a separate process. might be better off as a cronjob possibly.. ?
we have a choice between keeping it an internal cron or an external instance monitor that just regularly checks the health of the different containers and reports them to etcd.

So far I can see the following that needs to be moved over:

• configure etcd
• wait for etcd to be available
• wait until etcd has discarded potentially stale values
• set etcd values
• wait for confd to run once and install initial templates
• wait for confd to populate all values
• run database migration
• start celery workers
• spawn gunicorn
• smart shutdown
• spawn confd
• publish to etcd the injected PORT

I am not entirely sure of the best way to achieve the initialization..

opinions?

[bacongobbler]

I have a feeling that all of this can just get refactored into a much better pipeline. Using supervisord is just a hacky workaround over a hacky implementation, IMO. Docker containers should be running one process and one process only. To respond to your list of things in this light, I'd say:

1. Most of those things on your list (configure etcd, set etcd values) can be placed inside the systemd unit file as either PreExecStart or PostExecStart commands with a common TTL set if the unit goes down.

2. The confd hacks should be removed as well as confd itself. The only place we use confd is inside templates/confd_settings.py, which we can just use the etcd client library already installed to retrieve those values and the system would be much more efficient because of it (i.e. it'd be sucking much less RAM with confd gone). We lose "dynamic configuration" inside the container, but then again we can always try SIGHUP'ing the container and experiment with that.

3. database migrations can occur in another systemd oneshot job or inline as a PostExecStart.

4. celery workers should be separated out into another systemd job using the same container. One container, one process.

5. "smart shutdown" is another hack to get around the fact that we're not following "The Docker Way" ™️. This can be removed.

6. publishing the port to etcd, like the others, can be run as a systemd PostExecStart command.

If you're up for it, I'd much prefer this path over opting for supervisord, but the choice is ultimately our BDFL's, @gabrtv.

[developerinlondon]

All of the above makes sense @bacongobbler . systemd hooks would make it a lot lighter. I wouldn't mind restructuring it like that, but I am at present very tied up with my work with some deadlines, but can look at it in my spare time in the near future if nobody picks it up it.

[bacongobbler]

If I find myself with some spare cycles during v0.11.0 or I'm bored one of these weekends I'll whip up a PR :)

[carmstrong]

@developerinlondon We haven't forgotten about this - we're working on a new init behind the scenes. We're tracking this in #1561. Please feel free to follow and contribute there. Thanks!! :)