Getting the CA to play well with a custom scheduler

[consideRatio]

I opened this issue: kubernetes/kubernetes#71070

I understand it as that the Cluster Autoscaler is utilizing hardcoded logic from the default scheduler. Perhaps it would it be possible in the future to avoid hardcoding this but instead cooperating with the scheduler for the pod?

[MaciekPytel]

There were many discussions of this in the past. The problem is CA doesn't actually know anything about scheduling. It's built around importing scheduler code and using it as a sort of black box oracle. All CA decisions are based on simulations, performed by creating in memory node objects and feeding them to scheduler to see if currently pending pods would be able to schedule if a new node was added.

There were some discussions about exposing a 'dry run' API in scheduler. The problem is we need to run a ton of those scenarios, using some imaginary nodes, pretending some nodes don't exist or pretending some pods are running on different pods than they really are. Scheduler doesn't support any of that and even if it did we believe the performance impact of serializing all those objects and sending them to scheduler is prohibitive (we would need to make thousands of such requests per loop).

To sum up full support for custom scheduling would require massive changes in both scheduler and CA and we don't think it would work anyway. I think it's safe to say it's not likely to happen for general case.

Now, some forms of custom scheduling are fine - if you only touch priority functions, not predicates, than there is no problem, as CA ignores those anyway. If all you do is change scheduler config it should be possible to expose analogous config options in CA that would be forwarded to imported scheduler code. Finally if you add a new predicate function, you should be able to import it into CA, build your own image and use that.

[consideRatio]

@MaciekPytel this was an excellent write up! Thank you!!