parallel-workload test

[def-]

Reproduces panics:

bin/parallel-workload-test --threads=100 --seed=173227

thread 'coordinator' panicked at 'cannot fail to peek: CollectionMissing(User(23531))', /home/deen/git/materialize3/src/adapter/src/util.rs:280:23
thread 'coordinator' panicked at 'Dataflow oneshot-select-t1257 requested as_of (Antichain { elements: [1688594212000] }) not >= since (Antichain { elements: [1688594213108] })', src/adapter/src/coord/dataflows.rs:260:13

Before #20267: https://buildkite.com/materialize/nightlies/builds/2738 (failure time 1 min in CI)
After: https://buildkite.com/materialize/nightlies/builds/2739

Still a randomized test, but good enough to make sure this doesn't regress.

Fixes: #20274

Checklist

• This PR has adequate test coverage / QA involvement has been duly considered.
• This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
• If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
• If this PR will require changes to cloud orchestration, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
• This PR includes the following user-facing behavior changes:

[maddyblue]

Woo this is great! I hope to also make a deterministic repro for this specific failure sometime soon. Thanks for covering the random one.

[philip-stoev]

That is very impressive! Thank you. I think if we have the following random events:

• CREATE / DROP VIEW
• SET ISOLATION_LEVEL - (with 1% probability)
• [BEGIN] / COMMIT / ROLLBACK (with 10% probability)

We can say that the "concurrent DDL testing" has been taken care of for the foreseeable future.

[maddyblue]

Do you have thoughts about merging this so it's a nightly, or in some form where we can hit a button in buildkite and it runs on a specific PR?

[def-]

Do you have thoughts about merging this so it's a nightly, or in some form where we can hit a button in buildkite and it runs on a specific PR?

Yes, I now enabled the green subset of this in nightly as part of this change. Should be good to go in and hopefully find regressions. Edit: Green run: https://buildkite.com/materialize/nightlies/builds/2807 & https://buildkite.com/materialize/nightlies/builds/2809

[philip-stoev]

Thank you for the very quick turnaround on this, with immediate installation into Nightly, so exemplary work!