Skip to content

selfhost: three hardening gaps that turn a transient event into a durable wrong state #9544

Description

@JSONbored

Parent: #9487

Summary

Three small self-host correctness gaps found during live inspection of edge-nl-01 on 2026-07-27. Each one turns a transient event into a durable wrong state, and each is a few lines.

1. An empty secret file silently degrades to "unset"

src/selfhost/load-file-secrets.ts:26-45. A missing or unreadable <NAME>_FILE correctly throws (#6284), but a zero-byte or whitespace-only file sets env[NAME] = "" — which every downstream nonBlank() treats as unconfigured, and preflight.ts:173 deliberately skips absent values.

So a truncated GITHUB_WEBHOOK_SECRET file boots an instance that rejects every webhook, healthily, indefinitely. This is directly adjacent to the known secret-rotation footgun on this host, where the file is rewritten in place: the window in which it is momentarily empty is exactly when a container restart reads it.

  • Treat an empty post-trim secret file as the same fatal error as a missing one.
  • Keep the two failures distinct — a bad path and a truncated write are different operator problems and must not collapse into one message or one log event.

2. The crash handler can stall exit behind a telemetry flush

src/selfhost/process-lifecycle.ts:81-92 awaits flush() with no deadline before exit(1), wired to flushPostHog (src/server.ts:334). A wedged PostHog egress delays — or entirely prevents — the very restart the module exists to guarantee, leaving a process that has already logged a fatal alive and serving from a state it declared unsound.

  • Promise.race([flush(), sleep(N)]). Losing a few telemetry events is unambiguously the cheaper failure.
  • Bound BOTH fatal events identically; neither may hang the restart.

3. Half-written blobs are served forever

src/selfhost/blob-store.ts:48-52 writes with writeFile directly. Keys are input-hash-addressed (loopover/shots/<hash>.png), so a truncated PNG from a mid-write kill is never retried or overwritten — the next lookup for that same input finds a file and serves it as a permanently "valid" cache hit. The same codebase already does tmp+rename correctly at src/selfhost/private-config.ts:466-469.

  • Write to ${target}.tmp-<uuid> then rename. Unique temp names, so two concurrent puts of one key cannot publish each other's partial file.
  • Clean up the temp file on the failure path so a full disk cannot accrete orphans beside the real objects.

Tests

  • Zero-byte and whitespace-only secret files both throw, and the target is left unset — never "", the value that reads as unconfigured downstream.
  • The empty and unreadable failures stay distinguishable.
  • The process exits even when flush() never settles; a flush that settles first is still awaited.
  • No temp leftovers after either a successful or a failed put; concurrent puts of one key yield an intact object.
  • The tmp+rename mechanism itself is pinned — the crash window it protects cannot be reached by an in-process test, so nothing else would notice a revert.

Expected outcome

None of the three can leave a durable wrong state behind: no silently-unconfigured secret, no un-restarted process, no permanently-cached truncated object.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.orbGittensory Orb related - maintainer self-hosting analytics.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions