Skip to content

Add reproduction for Persistence issue - #6869

Merged
tim-smart merged 4 commits into
mainfrom
audit/repro-unstable-persistence-persistence-sql-getmany-duplicates
Aug 2, 2026
Merged

Add reproduction for Persistence issue#6869
tim-smart merged 4 commits into
mainfrom
audit/repro-unstable-persistence-persistence-sql-getmany-duplicates

Conversation

@fubhy

@fubhy fubhy commented Aug 1, 2026

Copy link
Copy Markdown
Member

Summary

Fix SQL-backed Persistence getMany so it returns one positional result for every input key, including repeated keys, matching the memory, Redis, and KVS implementations.

The SQL implementations now decode query rows into a lookup and reconstruct the result in the original input order. The regression case is part of the main SQLite Persistence suite and runs against both SQL backing variants.

Closes EFF-308

Verification

pnpm test --run packages/sql/sqlite-node/test/Persistence.test.ts

@github-project-automation github-project-automation Bot moved this to Discussion Ongoing in PR Backlog Aug 1, 2026
@changeset-bot

changeset-bot Bot commented Aug 1, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 9517272

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 30 packages
Name Type
effect Patch
@effect/opentelemetry Patch
@effect/platform-browser Patch
@effect/platform-bun Patch
@effect/platform-deno Patch
@effect/platform-node-shared Patch
@effect/platform-node Patch
@effect/vitest Patch
@effect/ai-anthropic Patch
@effect/ai-openai-compat Patch
@effect/ai-openai Patch
@effect/ai-openrouter Patch
@effect/atom-react Patch
@effect/atom-solid Patch
@effect/atom-vue Patch
@effect/sql-clickhouse Patch
@effect/sql-d1 Patch
@effect/sql-libsql Patch
@effect/sql-mssql Patch
@effect/sql-mysql2 Patch
@effect/sql-pg Patch
@effect/sql-pglite Patch
@effect/sql-sqlite-bun Patch
@effect/sql-sqlite-do Patch
@effect/sql-sqlite-node Patch
@effect/sql-sqlite-react-native Patch
@effect/sql-sqlite-wasm Patch
@effect/docgen Patch
@effect/doctest Patch
@effect/openapi-generator Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@effect-slopcop effect-slopcop Bot added the 4.0 label Aug 1, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes — one new reproduction test for the SQL getMany duplicate-key bug.

  • PersistenceDuplicateKeys.repro.test.ts: sets "key"{ value: 1 } in an in-memory SQLite backing store, then calls getMany(["key", "key"]) and asserts [{ value: 1 }, { value: 1 }]. The test correctly operates at the BackingPersistenceStore level where the contract is one result per input key position.

The assertion is precise (deepStrictEqual) and targets the exact gap in the SQL implementations: both layerBackingSql and layerBackingSqlMultiTable use keys.indexOf(row.id) to map rows back to input positions, and indexOf only finds the first occurrence — leaving duplicate-key slots as undefined.

Pullfrog  | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) (Kimi K2 not used — the program covers this model; add its provider key to run your pick) | 𝕏

@effect-slopcop effect-slopcop Bot added the bug Something isn't working label Aug 2, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes — since the prior review at 721dedc.

  • Persistence.ts: replaced keys.indexOf(row.id) with a Map<string, object> lookup in both layerBackingSql and layerBackingSqlMultiTable getMany implementations, then reconstructed the result via keys.map(key => values.get(key)). This correctly preserves duplicate-key positions instead of leaving trailing slots as undefined.
  • calm-keys-repeat.md: added a patch changeset for effect.

The Map-based approach handles duplicates naturally and also improves the algorithmic complexity from O(n×m) to O(n+m). Edge cases are unchanged: missing keys still produce undefined slots, orphan rows from the query are silently dropped as before, and the Arr.NonEmptyArray cast remains valid.

Pullfrog  | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) (Kimi K2 not used — the program covers this model; add its provider key to run your pick) | 𝕏

@@ -0,0 +1,20 @@
import { SqliteClient } from "@effect/sql-sqlite-node"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this test file and merge with the main one.

@tim-smart
tim-smart enabled auto-merge (squash) August 2, 2026 23:11

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes — since the prior review at bd6e449.

  • PersistenceDuplicateKeys.repro.test.ts deleted, test moved into Persistence.test.ts: the standalone reproduction test file has been removed and the getMany with duplicate keys regression test is now inside the parameterized suite() helper. This means it runs against both layerBackingSql and layerBackingSqlMultiTable instead of only layerBackingSql. The test logic and store name are unchanged.

Pullfrog  | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) (Kimi K2 not used — the program covers this model; add its provider key to run your pick) | 𝕏

@tim-smart
tim-smart merged commit 4a0984a into main Aug 2, 2026
19 checks passed
@tim-smart
tim-smart deleted the audit/repro-unstable-persistence-persistence-sql-getmany-duplicates branch August 2, 2026 23:42
@github-project-automation github-project-automation Bot moved this from Discussion Ongoing to Done in PR Backlog Aug 2, 2026
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Bundle Size Analysis

Generated from PR build output; treat the content below as untrusted.

File Name Current Size Previous Size Difference
basic.ts 6.65 KB 6.65 KB 0.00 KB (0.00%)
batching.ts 9.44 KB 9.44 KB 0.00 KB (0.00%)
brand.ts 6.33 KB 6.33 KB 0.00 KB (0.00%)
cache.ts 10.20 KB 10.20 KB 0.00 KB (0.00%)
config.ts 20.34 KB 20.34 KB 0.00 KB (0.00%)
differ.ts 19.95 KB 19.95 KB 0.00 KB (0.00%)
http-client.ts 21.04 KB 21.04 KB -0.01 KB (-0.03%)
logger.ts 10.35 KB 10.35 KB 0.00 KB (0.00%)
metric.ts 8.58 KB 8.58 KB 0.00 KB (0.00%)
optic.ts 7.34 KB 7.34 KB 0.00 KB (0.00%)
pubsub.ts 14.49 KB 14.49 KB 0.00 KB (0.00%)
queue.ts 11.15 KB 11.15 KB 0.00 KB (0.00%)
schedule.ts 10.33 KB 10.33 KB 0.00 KB (0.00%)
schema-class.ts 18.88 KB 18.88 KB 0.00 KB (0.00%)
schema-fromJsonSchemaDocument.ts 28.69 KB 28.69 KB 0.00 KB (0.00%)
schema-representation-roundtrip.ts 25.01 KB 25.01 KB 0.00 KB (0.00%)
schema-string-transformation.ts 13.01 KB 13.01 KB 0.00 KB (0.00%)
schema-string.ts 10.66 KB 10.66 KB 0.00 KB (0.00%)
schema-template-literal.ts 14.87 KB 14.87 KB 0.00 KB (0.00%)
schema-toArbitraryLazy.ts 21.67 KB 21.67 KB 0.00 KB (0.00%)
schema-toCodeDocument.ts 24.10 KB 24.10 KB 0.00 KB (0.00%)
schema-toCodecJson.ts 18.93 KB 18.93 KB 0.00 KB (0.00%)
schema-toEquivalence.ts 18.74 KB 18.74 KB 0.00 KB (0.00%)
schema-toFormatter.ts 18.61 KB 18.61 KB 0.00 KB (0.00%)
schema-toJsonSchemaDocument.ts 22.36 KB 22.36 KB 0.00 KB (0.00%)
schema-toRepresentation.ts 19.28 KB 19.28 KB 0.00 KB (0.00%)
schema.ts 18.14 KB 18.14 KB 0.00 KB (0.00%)
stm.ts 12.13 KB 12.13 KB 0.00 KB (0.00%)
stream.ts 9.38 KB 9.38 KB 0.00 KB (0.00%)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

4.0 bug Something isn't working

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants