Remove some of the scheduling abstraction in favour of more tightly coupled model <-> k8s integration

[helgi]

Right now all the model code has to interact with the scheduler via a thin and very constraint scheduler.k8s and scheduler.abstract API which includes very simple constructs such as start, create, destroy, deploy, scale and so on.

This is a hangover from when we had many schedulers and we were catering to the lowest common denominator and doing things in a very simplistic manner to reduce the surface area of models <-> scheduler interaction.

An example of that is when deis apps:create is ran then the only way k8s:service is created is when a scale event is triggered and internal to that scheduling code it is discovered that ReplicationController and Service is missing for the given app, the scheduling code creates those and then continues to scale from 0 to 1.

The above makes it very hard for us to appropriately hook into the k8s life cycle to drop annotations (such as domain information) to an existing Service without teaching Domain model how to create Services in addition to update. Keeping with the Domain example I have had to teach Domain minimal k8s-fu to store information.

This was all much simpler with etcd due to a far more adhoc nature. Drop a key, confd picks it up and does stuff with it somewhere. Full k8s integration is forcing us to think about models differently

By bringing k8s up one layer and giving models more direct access then we can keep the k8s-client clean and potentially separated out from workflow. This would involve bringing back #48, fixing it up and extending it so models can use it easily.

• Is this something we want to invest in?
• Is it bad to bring more awareness of k8s up into the models?
• How will we do the tests (we never really integrated etcd into the tests as an example)?

[krancour]

This makes sense to me, and this write up makes the issue more clear to me than our convo yesterday. Things such as "service," for example are unique to k8s. Keeping a generic interface that was suitable for multiple schedulers before, meant not hooking into app creation, for instance, to define a service. It had to be done elsewhere where it was a tad trickier. I get it.

It no longer needs to be this way. To be fair, there was probably a clever way to have done this better in the past too, but hindsight is 20/20. I'm no django expert, but here's what I have in mind...

Separate generic, highly reusable k8s functions such as crud operations on services, rcs, etc. into their own module for eventual extraction into a package. (I know you already intend that.) Then create a layer between that and the model with very generic fuction names on its interface-- things like postAppCreate() or postDomainAdd(). The idea would be that in any given implementation of this layer, the hooks can be implemented against the underlying scheduler (k8s in this case) without function names like start() or scale() polluting the interface. This layer's functions can be observers of all the events on the model. Theoretical support for another scheduler in the future is thus only a matter of providing an alternate implementation of a listener/observer layer that's already hooked into the model.

This would allow, for instance, k8s service creation to be triggered by app creation, even though services aren't a thing in other schedulers. Persisting a new app object triggers postAppCreate(), which in this case creates a service, but in the theoretical case of another scheduler might do something else entirely or may even be a no-op.

[helgi]

I'm going to close this as we have done a lot to separate things out and the high level functions left in the scheduler are pretty slim, beside run, which is up for a rewrite with Jobs