Implement separate long build queues

[carlocab]

This implements a separate queue for long-running builds that are
identified by a suffix -long in the runner name.

The values for slots in src/queue_types.rb is probably wrong, and
will need adjusting.

This should achieve having an entirely separate queue for long build
jobs. What isn't clear to me is whether this leads to unused capacity
when there are no long build jobs to be done but there are queued short
build jobs waiting.

[carlocab]

⚠️ Warning ⚠️ completely untested

[carlocab]

There's probably a more elegant way of encoding the fact that we want a long variant of an existing queue type...

[Bo98]

This is good and is also how I would have done it. One problem here though is we now have 12 slots for short queue and a separate 12 slots for the long queue.

We don't have 24 slots, so what we actually want is 12 slots shared between the two of them and somehow make it dynamic so that long builds get limited to 6 or so but short builds can extend up to the full 12 if free.

My initial thought a while back was a priority queue but that probably doesn't work well if we want to limit long builds to a different quantity.

[carlocab]

We don't have 24 slots, so what we actually want is 12 slots shared between the two of them and somehow make it dynamic so that long builds get limited to 6 or so but short builds can extend up to the full 12 if free.

Yes, this was my thinking too. Just not quite sure how to make it work without invasive changes to existing code. Will give it more of a think.

[Bo98]

One approach could be to replace the start processors' Queue.new with something else, such as a double-headed linked list (with pointers to next and next short) with popping that's dynamic based on resource usage (tracked by a semaphore or whatever). That would also allow it to remain one queue in terms of processing.

That may not necessarily be the best structure. concurrent-ruby may or may not have some inspiration (LockFreeLinkedSet etc).

In the end, every queue will have short and long items so some sort of in-queue handling would make sense here. Ideally we would generalise it well, so that say a -massbottle as an extra tag later on or something can be very easily inserted into the same code.

[MikeMcQuaid]

Makes sense to me so far! Good to merge when @Bo98 is happy.

[carlocab]

This needs a bit of cleaning up, but I think it's in a state where we're able to discuss the approach.

[MikeMcQuaid]

Looks good! Only comments I would have are those already present. Thanks @carlocab!

[carlocab]

This check may not necessarily be what we want (because it allows us to queue many long build jobs if the default queue is empty), but I haven't worked out how to avoid the situation where we try to call @queue[:default].pop on an empty @queue[:default].

[Bo98]

Yeah this is an issue particularly since the mutex will not be released when waiting on pop.

A couple solutions may be one or both of:

• Blocking a condition variable when both are empty that is reset on <<.
• Not using Queue at all and instead use an array (given we won't be using the blocking feature anymore).

[Bo98]

Looks good, thanks!

Only issue I found was SharedState.instance in the JobQueue constructor caused a recursive loop because the SharedState constructor itself created the job queue.

https://github.com/Homebrew/ci-orchestrator/compare/f0fe50550a134351612979b45b790cbc180950b0..513797afe30e243e213fcb8c0425dbde0c7d3d9e

Will deploy shortly.

[Bo98]

Given the limit in homebrew-core is 2 PRs, with a PR containing 3 OS versions, don't we actually want this to be 6 instead?

[carlocab]

Maybe? Is that the count we want per queue_type? If so, then yes, I think that's correct.

[carlocab]

So these are our queue types:

ci-orchestrator/src/queue_types.rb

Lines 7 to 9 in 8b19a5f

	MACOS_X86_64_LEGACY = 0
	MACOS_ARM64 = 1
	MACOS_X86_64 = 2

So, yes, 6 seems correct. Will open a new PR.

[carlocab]

#16

[Bo98]

Yeah. All arm64 is all one queue type so 6 sounds correct to me.

x86_64 is technically two types but that's not going to be the case come macOS 15.