test(artifact): add sync performance benchmarks - #3
Draft
maphew wants to merge 9 commits into
Draft
Conversation
Squashed follow-up changes: - fix(artifact): baseline curation of trashed sessions at opt-in - fix(artifact): roll back curation when metadata append fails - fix(artifact): delete corrupt bucket objects on pull so S3 self-heals - docs: design secure artifact S3 endpoints - docs: close S3 redirect downgrade gap - docs: plan secure artifact S3 endpoints - fix(artifact): secure S3 transport boundaries - test(artifact): tighten S3 security assertions - docs: explain S3 endpoint security policy - docs: clarify S3 corruption handling by scheme - docs: qualify S3 self-healing behavior - fix(artifact): pin insecure S3 override semantics - docs: design HTTP peer redirect rejection - docs: plan HTTP peer redirect rejection - fix(artifact): reject HTTP peer redirects - test(artifact): bound redirect source capture - docs: document peer redirect boundary and remove plans - fix(sync): harden artifact consistency boundaries - fix(sync): persist origin and heal invalid manifests - fix(sync): preserve provenance and retry state - fix(postgres): converge importer-first origin upgrades - fix(postgres): retain source state during consolidation - fix(sync): preserve remote history and purge durability - fix(server): serialize session lifecycle publication - fix(sync): harden artifact import and retention - fix(sync): cap artifact allocation amplification - fix(sync): bound nested artifact collections - fix(server): serialize curation publication - fix(artifact): harden sync boundaries - fix(sync): harden peer artifact convergence - fix(artifact): confine GC and recover deferred imports - fix(artifact): publish local sessions from HTTP peers Co-authored-by: Wes McKinney <wesmckinn+git@gmail.com>
Artifact imports can add metadata conflicts while a reader keeps the same session open. Caching conflict results only by session ID prevented live session refreshes from exposing those conflicts until navigation or reload. Keep asynchronous stale-response protection while treating each refreshed session snapshot as an opportunity to reload conflict state. VALID (fixed): #1 -- same-ID session refreshes now refetch metadata conflicts. INVALID (dismissed): none. PEDANTIC (skipped): none.
Watcher change bursts and startup already synchronize local session data, so repeating full discovery before artifact publication made incremental updates archive-sized. Keep the periodic floor as the coverage path for unwatched roots while preserving signal flushing on every export.\n\nPin metadata publication only needs a message ordinal and source UUID. A backend-parity point lookup prevents one pin or unpin from materializing the full transcript and its tool data while the lifecycle lock is held.
Periodic PG pushes previously scanned and allocated every historical artifact-import state before determining whether any sessions needed consideration. Large replicated archives therefore imposed work even on no-change watch ticks.\n\nDerive exact provenance keys only after the candidate window is assembled and resolve them through bounded primary-key batches. Empty candidate windows now avoid provenance I/O entirely.
Importer-first artifact convergence rewrites reverse references before deleting a legacy duplicate. Without reverse indexes, every duplicate forced full scans of sessions, tool calls, and tool-result events inside the push transaction.\n\nCreate the indexes after column migrations so both new and legacy schemas can accelerate consolidation safely.
Identity preparation previously issued serialized owner lookups for each candidate and repeated many of them during alias and relationship resolution. The latency dominated full pushes to remote PostgreSQL even when later fingerprint comparison skipped the session.\n\nPreload every candidate, legacy-prefix, canonical, and alias owner in one array query. Cache both rows and absences for the full push so out-of-window relationship lookups are queried at most once.
Owner
Author
|
Reference large-archive run on 2026-07-15:
These are comparison baselines rather than release thresholds. The target prints the effective fixture dimensions, and invalid explicit dimensions fail before timing. |
wesm
force-pushed
the
docs/local-first-multi-machine-sync
branch
3 times, most recently
from
July 22, 2026 20:34
cc88a44 to
c3be274
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds scalable, self-verifying artifact-sync benchmarks for initial export, initial import, converged no-op sync, and one-session incremental sync. The benchmark gate now includes the artifact package, explicit dimensions fail loudly, and output reports the effective fixture scale.
A canonical 16,000-message large-archive target and reference baseline make performance investigations repeatable. PostgreSQL is intentionally excluded so the harness remains self-contained and runnable without an external service.