Skip to content

chore: finalize kv structure for queue, workflows, traces, sqlite#4284

Merged
NathanFlurry merged 1 commit intomainfrom
02-23-chore_finalize_kv_structure_for_queue_workflows_traces_sqlite
Feb 24, 2026
Merged

chore: finalize kv structure for queue, workflows, traces, sqlite#4284
NathanFlurry merged 1 commit intomainfrom
02-23-chore_finalize_kv_structure_for_queue_workflows_traces_sqlite

Conversation

@NathanFlurry
Copy link
Member

Description

Please include a summary of the changes and the related issue. Please also include relevant motivation and context.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@railway-app railway-app bot temporarily deployed to rivet-frontend / rivet-pr-4284 February 24, 2026 07:14 Destroyed
@railway-app
Copy link

railway-app bot commented Feb 24, 2026

🚅 Deployed to the rivet-pr-4284 environment in rivet-frontend

Service Status Web Updated (UTC)
frontend-cloud 😴 Sleeping (View Logs) Web Feb 24, 2026 at 7:28 am
website 😴 Sleeping (View Logs) Web Feb 24, 2026 at 7:25 am
frontend-inspector 😴 Sleeping (View Logs) Web Feb 24, 2026 at 7:25 am
mcp-hub ✅ Success (View Logs) Web Feb 24, 2026 at 7:15 am
ladle ❌ Build Failed (View Logs) Web Feb 24, 2026 at 7:15 am

Copy link
Member Author

NathanFlurry commented Feb 24, 2026


How to use the Graphite Merge Queue

Add the label merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

@claude
Copy link

claude bot commented Feb 24, 2026

PR Review: chore: finalize kv structure for queue, workflows, traces, sqlite

This PR restructures the actor KV keyspace by adding storage versioning bytes and renumbering top-level namespaces. The intent is clearly to finalize the structure before it becomes immutable in production. The approach is directionally correct — versioned namespaces are the right design — but there are some issues worth flagging.


Breaking Change Without Migration

This is the most significant concern. The PR changes key paths for every major subsystem:

Subsystem Old key prefix New key prefix
Queue messages [5]{message_id} [5][1][2]{message_id}
Queue metadata [6] [5][1][1]
Workflow [7]{key} [6][1]{key}
Traces [8]{key} [7][1]{key}
SQLite [9][0/1, ...] [8][1][0/1, ...]

Any actor with existing data under the old scheme will silently lose access to it after this change. There is no migration path. If this is intentional (e.g., actors don't yet have meaningful production data at the old paths), that should be stated explicitly in the PR description to make the intent clear to future readers.


Documentation Inconsistency

The new ACTOR_KV_STRUCTURE.md tree view has this comment:

sqlite (8)/    # SQLite VFS namespace.
  v1 (1)/    # SQLite data version. Legacy pre-v1 SQLite keys are still resolved in sqlite-vfs/src/vfs.ts.

However, sqlite-vfs/src/vfs.ts contains no legacy key resolution logic. It only imports and uses the new getMetaKey / getChunkKey functions from kv.ts. This claim is inaccurate and should be removed or corrected.


Workflow Field VERSION Removal

WORKFLOW_FIELD.VERSION (previously [3, 4]) is removed, and INPUT is renumbered from [3, 5] to [3, 4]. Any workflow that previously stored version or input data at the old indices will silently read null or misinterpret the stored bytes after this change. Like the namespace renumbering, this is a hard break. Same comment applies: if intentional, state it explicitly.


rebuildMetadata Will Miss Old Queue Messages

In queue-manager.ts, #rebuildMetadata is called as a fallback when queue metadata can't be decoded. It scans for messages using QUEUE_MESSAGES_PREFIX (new path [5][1][2]). Old messages stored at [5]{message_id} won't be discovered during a rebuild, meaning the fallback recovery path does not actually recover pre-migration data.


Minor: Duplicate concatPrefix Implementations

concatPrefix is defined in both keys.ts and traces-driver.ts with identical implementations. The one in keys.ts could be exported and reused in traces-driver.ts.


No Tests

No new tests are added to cover the restructured key paths, the new versioned prefix functions (queueMetadataKey, queueMessagesPrefix, etc.), or the decodeQueueMessageKey prefix validation. The CLAUDE.md testing guidelines specify that driver tests should cover new functionality. This is especially important given that key encoding bugs would silently lose actor data.


Positive Notes

  • Adding explicit versioning bytes to all namespaces is the right design and makes future migrations possible.
  • The decodeQueueMessageKey prefix validation added in this PR is a good defensive check.
  • Reducing maxQueueMessageSize from 1 MiB to 64 KiB is the correct fix — it was previously possible to configure a value that would fail at KV write time without any error at configuration time.
  • Increasing KV storage limit from 1 GiB to 10 GiB and the simplified docs are reasonable.
  • The new ACTOR_KV_STRUCTURE.md tree view is much cleaner and more readable than the previous version.

@NathanFlurry NathanFlurry force-pushed the 02-23-chore_increase_actor_storage_size_to_10_gib branch from 2cb7629 to 1839ce7 Compare February 24, 2026 07:39
Base automatically changed from 02-23-chore_increase_actor_storage_size_to_10_gib to main February 24, 2026 07:39
@NathanFlurry NathanFlurry force-pushed the 02-23-chore_finalize_kv_structure_for_queue_workflows_traces_sqlite branch from 50ef521 to 88cf774 Compare February 24, 2026 07:39
@railway-app railway-app bot temporarily deployed to rivet-frontend / rivet-pr-4284 February 24, 2026 07:39 Destroyed
@NathanFlurry NathanFlurry merged commit cd2edc2 into main Feb 24, 2026
7 of 17 checks passed
@NathanFlurry NathanFlurry deleted the 02-23-chore_finalize_kv_structure_for_queue_workflows_traces_sqlite branch February 24, 2026 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant