Worker Group Probing and Restarts via BackWaveMonitor and BackWaveOperator

[DCarlson12]

After submitting the transient fault issue for the Oracle store I’ve been thinking about some other difficulties in how probing readiness of the worker groups works currently. Instead of trying to classify a bunch of different fault codes or risk analysis paralysis supplying those codes as a configurable item, is there a better way this can be done?

Right now, once a worker group halts it stays halted until the process restarts, and there isn't a good way to see that it happened across a cluster. BackWaveHealthCheck only reports on the node that answers the request, so behind a load balancer /health tells you about whichever node you happened to hit. The dashboard doesn't show worker health at all besides throughput. The invariant metric isn't emitted for halts caused by unclassified faults, and it has no group or node tag anyway, so in practice you find out from a Critical log line. A single halt doesn't lose work, since the leases lapse and other nodes pick it up. But database-side events like a failover or sessions being killed hit every node at the same time, so it's realistic for the whole cluster to halt together. The only recovery then is recycling every process running the halted pumps. Unless you have the rights to access your production infrastructure, you’re stuck waiting for a health check monitor or rolling interval for process restarts. The escape hatch at the moment would require a hypervisor or a person to check the health check and perform process restarts—not something available in all scenarios and certainly not an extra step everyone wants to go through just to make up for the very real scenarios that can occur with a large scale halt.

What I'd like to propose is pumps reporting their health and a heartbeat to the configured store, sort of akin to how Hangfire tracks its servers. BackWaveMonitor would expose a probing method that returns every pump across the cluster (whatever runs in the schema), optionally filtered, including any whose heartbeat has gone stale. BackWaveOperator would get an audited restart method that takes the worker id of a halted pump; the node that owns it picks up the request on its next heartbeat and restarts just that pump. I'd want this to keep fail-stop as the default, work with polling only (no wake-up hints or extra database privileges), and need at most a small additive schema change so rolling upgrades still work.

There’s plenty of questions that arise from this being

• How often should heartbeats run, and is that configurable per worker group?
• When does the each pump heartbeat actually start? Probably after the schema is touched, so at earliest after the first worker group poll, and it must survive anything except an application shutdown signal so it can always attempt to heartbeat and consume any signals for restarting a worker group it owns.
• How would BackWaveMonitor interpret the heartbeat results when it goes to probe? A halted node/cluster reports no new heartbeats until the store connection is restored, and there is an indeterminate amount of time between conditions that halt the worker group and conditions that allow the heartbeat to successfully write again (where it only then can report the group as halted, if it is).
• For process-stopped cases, what system goes in and cleans up orphaned heartbeat entries and pending restart signals?

On the flip side of this, I guess I’m not sure what the halting mechanism that is implemented is really accomplishing? Why can’t a pump always retry unless being shut down? Keeping halting state seems to breed a lot of extra workarounds which might not actually be necessary.

[pdevito3]

Going to start looking into this a bit this weekend @DCarlson12!

[DCarlson12]

Great! Thanks for the consideration. I’m around if you have any questions.