environmentd: Have environmentd panic on child crashes during testing

[philip-stoev]

By default, the process orchestrator will restart any computed and storageds that it manages. During testing, this will unfortunately mask panics that should have caused the test to fail instead.

• Add a new environmentd-command line option that will cause environmentd to panic if any of its children exit with an exit code that implies a crash or a panic happened on the child.
• Make sure the new behavior is in effect for sqllogictests, cargo unit tests and mzcompose-based tests
• do not panic environmentd in mzcompose workflows that explicitly use mz_panic() to panic a computed
• Remove the KillReplica Action from Zippy, that was using mz_panic() internally. The original idea of this Action was to panic a single replica, but in practice mz_panic() is evaluated by all replicas and they all panic together.

Motivation

• This PR fixes a previously unreported bug.

The fact that the process orchestrator is restarting panicked children masks bugs that should have been exposed during testing.

[philip-stoev]

With this PR #16318 is visible directly in the main CI, not even Nightly is required:

https://buildkite.com/materialize/nightlies/builds/1422#0184c2d2-25f0-4a4e-983c-9335a512c612

[benesch]

This looks great, except that I think the ReusePort is hiding a deeper bug! I think it's time for the process orchestrator to learn to use Unix domain sockets instead of TCP connections.

[philip-stoev]

@benesch I would like to move this forward as soon as possible, it is now or never while the builds are green even with panic-on-crash enabled.

[benesch]

Yes, apologies, I just can't get behind setting SO_REUSEPORT! It has me spooked. I hear you on getting this in ASAP, though. I just pushed up a big rewrite of the process orchestrator in #15810 that will hopefully avoid all these problems. 🤞🏽

[benesch]

#15810 just merged. I've rebased this on top of #15810 (with a few modifications to naming/comments). 🤞🏽 that it passes!

[benesch]

@philip-stoev I'm so sorry but there appears to be at least one real bug here. The legacy upgrade tests are crashing with:

kafka-u32: some element of the Sink as_of frontier is too far advanced for our output-gating timestamp: as_of Antichain { elements: [1670181633879] }, gate_ts: 1670181537332', src/storage/src/sink/kafka.rs:1063:13

I'm not sure what's going wrong in cargo-test. It's of course absolutely impossible to see what's going on there since the logs are disabled. Sigh.

[philip-stoev]

#15810 just merged. I've rebased this on top of #15810 (with a few modifications to naming/comments). 🤞🏽 that it passes!

@benesch can you push your rebase to GitHub?

[benesch]

I already did?

[philip-stoev]

Oh, my local branch must have messed up then . Let me reset it.

[benesch]

Aha! It's test_github_12546 that's to blame. Which makes sense! That test calls mz_panic. I'm working on a fix now.

[benesch]

Just pushed up a fix to disable propagate_crashes in test_github_12546. I'm feeling good about this one!