fix: postgres pitr hardening — HA restore guard, pure JSON stdout, testable watch timeout - #1058
Merged
Merged
Conversation
…estable watch timeout
- `pitr backup restore` now refuses HA clusters: an in-place restore only
replaces the root volume, leaving replicas diverged; the dashboard
reseeds replicas as part of the same operation.
- `pitr enable --json` on an already-enabled service kept stdout pure
JSON (the human note moves to stderr).
- Debug-build-only RAILWAY_POSTGRES_{WATCH_TIMEOUT,POLL_INTERVAL}_SECS
env overrides so the e2e harness can exercise the progress --watch
timeout branch in seconds (same pattern as RAILWAY_BACKBOARD_URL).
- schema.json regenerated from the Public subgraph post-mono exposure of
volumeInstanceBackupRestore's replicaServiceIds/wipeServiceIds; the
only structural change is those two args (rest is generated-file
reflow).
…it the staged swap, deploy The server-side createVolumeFromSnapshot workflow copies the backup into a fresh volume and then only STAGES the volume swap; the dashboard has the user apply staged changes as a second step. The CLI previously reported "running in the background" and returned, leaving the service on the old volume indefinitely (caught by the new in-place-restore e2e step: the post-backup marker never disappeared). Now: warn about pre-existing staged changes up front, wait for the copy (30-min budget via wait_for_workflow_up_to -- the copy scales with volume size), then commit + deploy, reporting "Restored ... and deployed".
Collaborator
Author
|
Added a third fix the new e2e step caught live: |
An auth refusal on workflowStatus is deterministic (e.g. a workflow type that doesn't register the queries the resolver authorizes against) -- retrying can't heal it. Previously the poll silently retried its whole budget (30 min for backup restore). Surface it immediately with a note that the workflow keeps running server-side. Companion to railwayapp/mono#34897, which registers the auth query handlers on createVolumeFromSnapshot itself.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-ups from the adversarial review + e2e coverage audit of the
railway postgresstack (#1052–#1055).What
pitr backup restorerefuses HA clusters. An in-place restore replaces only the root's volume; replicas would rejoin with newer, diverged timelines. The CLI cannot reseed replicas (that path is dashboard-only for now), so it bails with a pointer to the dashboard.pitr enable --jsonkeeps stdout pure JSON when the service is already enabled — the human-readable note goes to stderr, so--json | jqworks on every path.RAILWAY_POSTGRES_WATCH_TIMEOUT_SECS/RAILWAY_POSTGRES_POLL_INTERVAL_SECS(compiled out of release builds, same pattern asRAILWAY_BACKBOARD_URL) so the e2e harness can exercise theprogress --watchtimeout branch in seconds instead of 10 minutes.schema.jsonregenerated from the Public subgraph after railwayapp/mono#34881 exposedreplicaServiceIds/wipeServiceIdsonvolumeInstanceBackupRestore. Semantic diff verified: those two args are the only structural change (the rest of the 36k-line diff is generated-file reflow).Tests
cargo fmt --checkclean.