fix(runtime): fail fast when proxy listeners terminate - #16
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related
Problem
The Zero application starts proxy orchestration in a detached Tokio task, then the root task waits only for Ctrl-C. If proxy/listener orchestration terminates unexpectedly, inbound sockets are dropped but IPC/control services and the process remain alive. This produces a false-healthy state: systemd sees a running Zero process while protocol ports have disappeared.
This matches a field report from one VPS where Zero remained responsive through its control plane after the configured listener ports stopped listening.
Changes
Supervise the data plane from the root application lifecycle
proxy.run_until(wait_for_shutdown_signal())directly under the application root instead ofproxy.spawn()+ an unsupervised join handle.engine.stoppedwithruntime_errorfor unexpected data-plane termination andsignalfor normal shutdown.Distinguish QUIC endpoint failure from per-client failure
The QUIC stream listener previously awaited both the client handshake and first bidirectional stream inline. A failed or abandoned client could therefore terminate the listener and, once root supervision was added, force the whole process to restart.
Now QUIC listener lifecycle and connection lifecycle are separated:
EngineErrorand triggers the application supervisor;Ok(())path.Add structured listener/runtime diagnostics
The field incident cannot be reproduced reliably, so this PR also makes the next occurrence actionable instead of leaving only a generic task-exit error.
Listener lifecycle logs now carry stable fields such as:
inbound_tag;protocol;transportwhere known;reasonvalues includingshutdown_signal,shutdown_channel_closed,listener_endpoint_closed,connection_accept_error, connection-task panic and listener-task error;The listener task wrapper logs the configured inbound identity both when a task returns cleanly and when it fails. This covers protocols whose inner listener loop does not expose all configured address metadata itself.
The orchestration supervisor logs unexpected listener and URLTest termination with:
core_instance_id;config_revision;TCP
accept()errors retain the listener instead of terminating it, but now include inbound tag, transport and local listen address so host-specific resource failures can be diagnosed from the journal. System-stack closure and shutdown-channel loss are also logged explicitly.URLTest runtime tasks log their group ID and task result so a URLTest task cannot collapse into an untraceable
UrlTestTaskExitedif it becomes the orchestration failure source.Scope
This PR intentionally does not add an in-process listener restart loop. The safe first recovery boundary is the Zero process: an unexpected data-plane failure must become visible and cause a non-zero exit rather than leave a partially alive process.
The exact host-specific trigger on the affected VPS still requires runtime evidence. This change makes that evidence observable and prevents the trigger from being hidden behind a false-healthy process.
Validation
Added focused regressions for application-level error supervision, expected versus unexpected listener/URLTest exits, QUIC handshake failure isolation, QUIC client close before first stream, and runtime ownership boundaries.
Repository formatting, workspace checks/tests, all-target/all-feature clippy, optional surfaces, musl compatibility, and GitHub CI pass. Follow-up host-side diagnostics in Zboard are tracked separately in zerodenet/zboard#42.