ref(controller): query fleet state

[bacongobbler]

Before, we relied on django-fsm to give us a general idea of what state
the container object in the model is at. This did not give users an
accurate idea of the state of their containers. Introducing a new
.state() method to the scheduler as well as changing Container.state to
call it's own scheduler's .state() method allows users to directly
understand what state their containers are in.

closes #2766
closes #2987

This was thoroughly tested by hand since the controller does not wrap unit tests around fleet.py, but this should cover it :)

Points: PTAL at the systemdActiveStateMap in fleet.py and let me know if the mappings between the dbus API's ActiveState and what used to be DjangoFSM's JobState is correct.

[bacongobbler]

Something that @gabrtv brought up was that we should have a benchmark test for the old deis ps and the new deis ps. This one will definitely take longer since it's a direct http call... Might be better to cache the request somewhere, somehow. Open to ideas!

[bacongobbler]

I had to add a new close_db_connections decorator to make up for the post_transition decorator that was replaced. This is necessary in multi-threaded function calls which modify a model's state. Source code courtesy of https://code.djangoproject.com/ticket/22420#comment:17

[aledbf]

Might be better to cache the request somewhere, somehow. Open to ideas!

@bacongobbler why the cache? I know that it's going to be "slower" than a sql query but I need the real state not something from 60 seconds ago

[gabrtv]

@aledbf when i discussed with @bacongobbler I was referring to an in-memory cache per request that avoided N Fleet UnitState queries for N containers (as currently implemented). The result would be "real state", just a more efficient query.

[bacongobbler]

We could optimize by querying the global unit state instead of querying for each individual job state, then filtering for the specific job. That would allow us to retrieve the state of all containers in one request rather than N requests, as @gabrtv mentions :)

[bacongobbler]

After looking through fleet's API with @gabrtv, it seems like the global unit state request is paginated. That may be more trouble than it's worth; it could potentially hurt our common use case (between 1-5 containers). I'll have to do some benchmarks, but as long as it's not terribly slow when dealing with 20+ containers then we should be okay.

[bacongobbler]

while fixing this up, I'm finding that deis destroy takes forever, but that's because we destroy each container in series rather than parallel. deis ps is actually quite fast!

Shall I fix that up, similar to how we scale containers?

[bacongobbler]

As an example:

><> deis destroy

 !    WARNING: Potentially Destructive Action
 !    This command will destroy the application: foo
 !    To proceed, type "foo" or re-run this command with --confirm=foo

> foo
Destroying foo...
done in 99s
Git remote deis removed

[bacongobbler]

@gabrtv I was testing deis ps with 20 containers scaled up. The response time was < 1 second. I've also made App.destroy() delete its containers in parallel so we should be good now for manual testing :)

EDIT: never mind... looks like I'll have to come up with a better mapping between systemd's active/sub states and Deis' container states.

[mboersma]

Heh, ActiveState.

[mboersma]

This should fix container state disagreement that we have seen from time to time. Code LGTM. Nice work @bacongobbler.

[carmstrong]

Code LGTM. We can also remove the last note from http://docs.deis.io/en/latest/managing_deis/backing_up_data/ but in the sake of getting this PR merged, that can be done in a follow-up PR.