Skip to content

fix: intermittent Postgres deadlock for concurrent memory searches#2146

Open
supreme-gg-gg wants to merge 2 commits into
kagent-dev:mainfrom
supreme-gg-gg:fix/memory-query
Open

fix: intermittent Postgres deadlock for concurrent memory searches#2146
supreme-gg-gg wants to merge 2 commits into
kagent-dev:mainfrom
supreme-gg-gg:fix/memory-query

Conversation

@supreme-gg-gg

Copy link
Copy Markdown
Contributor

Close #2135

Fixes intermittent Postgres deadlocks when concurrent memory searches update overlapping rows' access_count (e.g. PrefetchMemoryTool fanning out per-sentence searches). Row locks are now acquired in id order, and access-count increments are best-effort so a bookkeeping failure no longer fails the search.

The root cause is described in #2135 but in short when two concurrent search requests return results like ["memory_1", "memory_2"] and ["memory_2", "memory_1"], it will result in a deadlock when trying to update the rows.

I've added a concurrency test that reproduced the error without this fix to the query and passed after the fix.

Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>
Copilot AI review requested due to automatic review settings July 3, 2026 20:02
@github-actions github-actions Bot added the bug Something isn't working label Jul 3, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses intermittent PostgreSQL deadlocks triggered by concurrent memory searches that increment memory.access_count on overlapping row sets. It makes row-lock acquisition deterministic (by id order) and changes access-count increments to be best-effort so memory search results are not blocked by bookkeeping failures.

Changes:

  • Update IncrementMemoryAccessCount to lock rows in a deterministic id order before updating to prevent deadlocks between concurrent overlapping increments.
  • Make SearchAgentMemory ignore (and log) access-count increment failures so searches still return results.
  • Add a concurrency-focused database test to catch regressions around concurrent search/access-count updates.

Reviewed changes

Copilot reviewed 3 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
go/core/internal/database/queries/memory.sql Changes access-count increment SQL to lock rows in id order before updating.
go/core/internal/database/gen/querier.go Regenerates sqlc interface to reflect the updated query/comments.
go/core/internal/database/gen/memory.sql.go Regenerates sqlc output with the updated SQL for access-count increments.
go/core/internal/database/client_postgres.go Makes access-count increments best-effort during memory search (no longer fail the search).
go/core/internal/database/client_test.go Adds a concurrent search test intended to reproduce/prevent the deadlock scenario.
Files not reviewed (2)
  • go/core/internal/database/gen/memory.sql.go: Generated file
  • go/core/internal/database/gen/querier.go: Generated file

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread go/core/internal/database/client_postgres.go
Comment thread go/core/internal/database/client_test.go Outdated
Comment thread go/core/internal/database/client_test.go Outdated
Comment thread go/core/internal/database/queries/memory.sql Outdated
Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Concurrent memory searches fail with deadlock

2 participants