cloudtest: Parallelize upgrade test

[def-]

Seen failing in https://buildkite.com/materialize/nightly/builds/11783#01961ce8-bbdc-41da-b266-10397ff883e9

  Warning  FailedScheduling  82s (x18 over 9m49s)  default-scheduler  0/7 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 5 Insufficient memory. preemption: 0/7 nodes are available: 2 Preemption is not helpful for scheduling, 5 No preemption victims found for incoming pod.

Checklist

• This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
• This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
• If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
• If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
• If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

[def-]

@doy-materialize I think this is related to license keys, this looks suspiciously like it's failing after 24 GB:

Apr 10 07:27:00 ip-10-61-114-81.ec2.internal kernel: Memory cgroup out of memory: Killed process 211652 (clusterd) total-vm:24047444kB, anon-rss:4062840kB, file-rss:102776kB, shmem-rss:0kB, UID:999 pgtables:32032kB oom_score_adj:979

https://buildkite.com/materialize/nightly/builds/11792
Is it possible that the license key passing in in misc/python/materialize/cloudtest/k8s/environmentd.py is broken?
I see no output from environmentd about license keys though.

[doy-materialize]

hmmm, i'm not 100% sure here - we do set memory limits based on the cluster size that is provided, but the only thing the license key work changed was whether we prevent clusters from being launched if their declared memory usage for the cluster size would be higher than 24G. that said, it's also expected that environmentd won't complain about a missing license key if the process orchestrator is being used, so it is possible that something isn't being passed through correctly. i'd check the cluster size definitions first though?

[def-]

Green run: https://buildkite.com/materialize/nightly/builds/11819#_