plugin: Replace readClusterState with existing watch events

[sharnoff]

... and a few other commits tossed in for fun.

Probably best to review commit-by-commit.

The thing I'm most worried about is that this potentially may change buffer handling.

Might be a good idea to go for #840 first, but I suspect that might also have its own issues.

This is a precursor to #517, to minimize the number of places we need to change resource handling.

[Omrigan]

Do you mean these watch events?

autoscaling/pkg/plugin/plugin.go

Lines 150 to 160 in 76ceab2

	vwc := vmWatchCallbacks{
	submitConfigUpdated: func(logger *zap.Logger, pod util.NamespacedName, newCfg api.VmConfig) {
	pushToQueue(logger, pod.Name, func() { p.handleVMConfigUpdated(hlogger, pod, newCfg) })
	},
	submitBoundsChanged: func(logger *zap.Logger, vm *api.VmInfo, podName string) {
	pushToQueue(logger, vm.Name, func() { p.handleUpdatedScalingBounds(hlogger, vm, podName) })
	},
	submitNonAutoscalingVmUsageChanged: func(logger *zap.Logger, vm *api.VmInfo, podName string) {
	pushToQueue(logger, vm.Name, func() { p.handleNonAutoscalingUsageChange(hlogger, vm, podName) })
	},
	}

I am trying to understand how this relates to #873. For #873 my understanding was to handle those callbacks from pod watcher, and get rid of VM watcher entirely.

[sharnoff]

Do you mean these watch events?

Broadly, watch events like those - but it only relies on the Pod watch events, I think. (and specifically, this PR waits for only the initial Pod "add" events to be completely processed)

I am trying to understand how this relates to #873.

Right, yeah fair. This PR should be ok - we only care about the Pod watch events. (unless I missed something - please feel free to double-check me no that)

[Omrigan]

Have some questions

[Omrigan]

Can you use semaphore here instead?

• Acquire(nil, n) when events are added
• Release(1) when event is processed
• Acquire(ctxWithTimeout, n) when waiting for events processing

[sharnoff]

Seems simpler. Implemented it in caba29f but tbh I'm not sure if it's better. WDYT?

[Omrigan]

If the incEventCount is left as a semaphore wrapper anyway, maybe instead revert to your old implementation, but wrap all in a struct synchronization primitive?

The reason I suggested semaphore is that I am wary of using raw channels and atomics in the code, where it feels like it should be a synchronization primitive.

[sharnoff]

Made it a separate type in 521597e - LMK what you think.

[Omrigan]

I imagined the implementation with initEventsCount hidden inside the new helper.

Also, you don't need incEventCount now

[sharnoff]

I imagined the implementation with initEventsCount hidden inside the new helper.

Hm, it's hard to make that kind of abstraction safe to use (because the counter wouldn't be monotonic) -- maybe it'd be fine here, but I'd rather keep the logic inlined than make an unsafe abstraction for just this purpose.

So in its current state, incEventCount or equivalent is still needed (we can just directly add to the atomic; don't know whether that's better)

[sharnoff]

Realized that the necessity of debe3b2 (from #936) is partially due to 25bbe43 here — it adds Buffer always, but there's a handful of circumstances where we don't actually need it (like when a new VM is added but doesn't go through our scheduler)

[sharnoff]

Adapted debe3b2 here as c8e250e to fix that issue. Impact should have been close to zero in practice, but wanted to fix it just to be safe. @Omrigan wdyt?

[Omrigan]

Why do we even have SchedulerName as a part of the spec, why do we ever want it to be different?

[sharnoff]

Result of discussing this: it's part of the VM spec because neonvm is a logically separate dependency of "autoscaling", so the ability to run with/without the rest of autoscaling requires the ability to set the pod's scheduler name

[Omrigan]

Are you going to do rebase merge? If so, tmp commit needs to be squashed somewhere, maybe into middle commit.