Skip to content

perf: batch ingest paths through per-page write transactions - #138

Merged
steipete merged 2 commits into
mainfrom
steipete/priceless-bardeen-b8b715
Aug 7, 2026
Merged

perf: batch ingest paths through per-page write transactions#138
steipete merged 2 commits into
mainfrom
steipete/priceless-bardeen-b8b715

Conversation

@steipete

@steipete steipete commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Every ingest path except the provider ran one BEGIN IMMEDIATE transaction per message (~10 statements each, serialized through MaxOpenConns(1)). Benchmarking showed tx-per-page is ~3.4x faster. This converts the remaining paths to accumulate a page of messages into store.WriteBatch and commit per page, following the provider idiom:

  • slackapi: history pages and thread-reply pages batch per API page; thread syncs run after the page commits, and thread-skip/channel-skip sync-state bookkeeping stays outside the batch.
  • slackmcp: channel messages flush every 500; summary.Messages now comes from WriteBatchResult.MessagesWritten (identical semantics). Thread parents are batched separately from replies so the reply count stays exact.
  • slackdesktop: redux states and drafts batch per state/pass; legacy-draft deletions run after the batch commit.
  • cli/import: batches 500 messages per flush; validateImportMessageKeys replaces per-message point queries with a chunked bulk IN-list lookup (499/chunk, matching the store preflight precedent), preserving the exact error text.

New additive store flag MessageWrite.SkipWorkspaceCollision turns cross-workspace message collisions into clean in-batch skips. This is transactionally safe: in the batch flow a collision only surfaces after a zero-row upsert, before any writes for that message. WriteBatchResult.CollisionsSkipped reports the skipped keys so the Slack API sync replays each through skipMessageCollision, preserving the per-message warning logs #131 introduced — and a collided parent's thread is not queued for thread sync, matching #131's skip semantics. Desktop paths keep their historical silent skip. Covered by tests in internal/store/batch_test.go (skip-and-commit and default-abort cases).

Intentional semantic shift: a mid-page write failure now rolls back the whole page instead of leaving earlier messages committed — the intended atomicity of page-sized transactions.

Proof: go build ./..., go vet ./..., go test ./..., go test -race ./... all green on top of 10e287b.

@clawsweeper

clawsweeper Bot commented Aug 7, 2026

Copy link
Copy Markdown

ClawSweeper status: review started.

I am starting a fresh review of this pull request: perf: batch ingest paths through per-page write transactions This is item 1/1 in the current shard. Shard 0/1.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

@steipete
steipete merged commit b407a90 into main Aug 7, 2026
14 checks passed
@steipete
steipete deleted the steipete/priceless-bardeen-b8b715 branch August 7, 2026 02:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants