Skip to content

fix(rivetkit): retry get_or_create as get on allocated datacenter conflict - #5627

Merged
abcxff merged 1 commit into
mainfrom
stack/fix-rivetkit-retry-get_or_create-as-get-on-allocated-datacenter-conflict-uosmqvxt
Sep 2, 2026
Merged

fix(rivetkit): retry get_or_create as get on allocated datacenter conflict#5627
abcxff merged 1 commit into
mainfrom
stack/fix-rivetkit-retry-get_or_create-as-get-on-allocated-datacenter-conflict-uosmqvxt

Conversation

@abcxff

@abcxff abcxff commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

No description provided.

@abcxff

abcxff commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Stack for rivet-dev/actors

Get stack: forklift get 5627
Push local edits: forklift submit
Merge when ready: forklift merge 5627

change uosmqvxt

@railway-app

railway-app Bot commented Sep 1, 2026

Copy link
Copy Markdown

🚅 Deployed to the actors-pr-5627 environment in rivet-frontend

Service Status Web Updated (UTC)
website ❌ Build Failed (View Logs) Web Sep 1, 2026 at 11:21 pm
kitchen-sink 😴 Sleeping (View Logs) Web Sep 1, 2026 at 7:11 pm
frontend-cloud 😴 Sleeping (View Logs) Web Sep 1, 2026 at 7:08 pm
frontend-inspector 😴 Sleeping (View Logs) Web Sep 1, 2026 at 7:06 pm
ladle ✅ Success (View Logs) Web Sep 1, 2026 at 7:04 pm
mcp-hub ✅ Success (View Logs) Web Sep 1, 2026 at 7:01 pm

@railway-app
railway-app Bot temporarily deployed to rivet-frontend / actors-pr-5627 September 1, 2026 19:01 Destroyed
@abcxff
abcxff force-pushed the stack/fix-rivetkit-retry-get_or_create-as-get-on-allocated-datacenter-conflict-uosmqvxt branch from 33ef6d1 to a1ee60a Compare September 1, 2026 23:21
@railway-app
railway-app Bot temporarily deployed to rivet-frontend / actors-pr-5627 September 1, 2026 23:21 Destroyed
@claude

claude Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Review

Small, focused fix: get_or_create_with_key now detects the actor.key_reserved_in_different_datacenter error from the PUT and falls back to a get_with_key lookup instead of surfacing the conflict as a hard failure. Good test coverage of the happy healing path (get_or_create_retries_as_get_on_datacenter_conflict).

Potential bugs

  1. remote_manager.rs:343 - The fallback is a single, un-retried get_with_key call. The comment says this heals after race, but the race window it is meant to heal is exactly the case where the winning reservation is visible (causing the PUT to fail with key_reserved_in_different_datacenter) while the actor row itself has not materialized yet via the list-by-key endpoint. If the fallback get_with_key lands inside that narrow window, it returns None and the whole call now hard-fails with actor key reserved in different datacenter but get by key found no actor, turning what should be a transient, self-healing race into a permanent error for the caller. Consider a short bounded retry/backoff around the fallback get_with_key rather than a single shot.

  2. Error context lost on fallback failure - self.get_with_key(name, key).await? propagates the raw error from the fallback GET (e.g. a transient network blip) with no mention of the original PUT conflict that triggered the fallback path. Wrapping with .context("retrying get_or_create as get after datacenter conflict") would make production debugging easier when this path is hit.

Style / convention

  1. remote_manager.rs:344 - anyhow!("actor key reserved in different datacenter but get by key found no actor") builds a fresh error via the anyhow! macro. Root CLAUDE.md asks to prefer .context() over anyhow! when constructing failable-function errors; pairing this with .context(...) on the original PUT failure info (per point 2) would satisfy both the convention and the debuggability concern in one change.

Overall a reasonable, narrowly-scoped fix. The main thing worth hardening before merge is the single-shot fallback lookup, since it can still leave a transient race as a hard error for the caller.

@abcxff
abcxff merged commit a1ee60a into main Sep 2, 2026
10 of 17 checks passed
@abcxff
abcxff deleted the stack/fix-rivetkit-retry-get_or_create-as-get-on-allocated-datacenter-conflict-uosmqvxt branch September 2, 2026 00:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant