[NOT FOR MERGING] Simulator for inference autoscaling

[jan-elastic]

Legend

Request count / inference time:

• orange: real value
• blue: estimated value by running average

Wait time:

• blue: max wait time (bucketed by minute)
• orange: average wait time (bucketed by minute)

Note: all times are in seconds

[jan-elastic]

This is what you get with virtual cores / hyperthreading (notice the increase in latency/queue size when you start hitting hyperthreaded CPUs):

[tveasey]

This looks good. I think the main consideration is do we make the estimators more effective. For example, I think one natural choice would be to try and estimate rate dynamics (say derivative). Also I think we should consider rejigging the estimate of inference time to allow for change as a function of allocation count. My expectation is this might yield significantly lower latency spikes.

[jan-elastic]

Example output:

[tveasey]

Mainly I think it is worthwhile commenting in a few places if other people need to follow this code later. Also I have one comment regarding initialisation. Anyway, results LGTM.

[tveasey]

I would add a comment here for the record...

Suggested change

	if dynamics_changed or abs(value - self.value)**2 / (self.variance + variance) > 100:
	# If we know we likely had a change in the quantity we're estimating or the prediction is 10 std off
	# we inject extra noise in the dynamics for this step.
	if dynamics_changed or abs(value - self.value)**2 / (self.variance + variance) > 100:

[tveasey]

One last thought. The estimator will behave badly with outliers: we'll immediately update the state to something in the vicinity of the outlier because of the 10 sigma rule.

For rate this is probably not so likely, but for inference time estimation there could be environmental factors which cause occasional very slow inferences. I would try simulating with occasional 10-20 x inference time values. As a minimum I would probably drop the 10 sigma rule for the inference time Q estimate. The approach which estimates the constant of proportionality between average inference time and allocation count as an extra state variable will also be robust to outliers.

[tveasey]

I think it is useful to discuss the basic ideas behind this for anyone else coming to this code

Suggested change

	class Estimator:
	class Estimator:
	"""
	This implements a 1 d Kalman filter with manoeuvre detection. Rather than a derived dynamics model
	we simply fix how much we want to smooth in the steady state.
	"""

[tveasey]

As above I would give some motivation for this:

Suggested change

	class Simulator:
	class Simulator:
	"""
	We simulate a Poisson process for inference arrivals with time varying rate parameter. This models a
	variety of rate dynamics: smooth ramp, smooth periodic, shock and steady.
	Inference is modelled as mean duration + uniform noise. The mean duration captures the behaviour of
	vCPUs which is that throughput is largely constant after all physical cores on a node are occupied.
	We assume perfect load balancing, i.e. that every allocation of inference is never waiting whilst there are
	inferences to be done. Inferences can only be picked off once they have arrived and allocations can only
	pick an inference "inference duration" after they picked their last inference. We assume they are selected
	FIFO.
	The key user parameters are the number of waiting inferences and the average and maximum delay to
	receive each inference which can be calculated from the difference between when inference calls arrived
	and when they are available.
	"""

[tveasey]

This saturates if rate is > 1 / 0.001. I guess it doesn't matter for the parameters you use for simulation, but it would be a little more robust if you generate next time instead as t += numpy.random.exponential(scale=1/self.get_request_rate(time)).

[tveasey]

Reading on I see this will have an impact on implementation of run stat estimation. This could all be resolved, but let's not bother to do that now. I think a comment is warranted that the simulator should only use rate << 1 / TIME_STEP.

[tveasey]

It's important to tie the initial variance to the measurement variance.

Alternatively, my inclination would probably be to initialise variance as None and fix the gain for the first measurement, i.e. gain = 0.99 if variance is None else (self.variance + process_variance) / (self.variance + process_variance + variance).

[tveasey]

Not important here, but in the actual system if we suddenly stop receiving inference requests having received them at a high rate we will never trigger down scaling if this is event driven. You will instead need some sort of heartbeat to check scaling.