diff --git a/.changeset/engine-autonumber-resync.md b/.changeset/engine-autonumber-resync.md new file mode 100644 index 0000000000..975ad9a209 --- /dev/null +++ b/.changeset/engine-autonumber-resync.md @@ -0,0 +1,77 @@ +--- +"@objectstack/objectql": patch +--- + +fix(objectql): resync the engine's fallback autonumber counter instead of seeding it once (#6806) + +On the engine's **fallback** autonumber path — the one serving drivers that do +NOT declare `supports.autonumber` (driver-memory, driver-mongodb); SQL drivers +own a persistent sequence and are untouched — `applyAutonumbers` seeded +`object.field.` from the store **once** and then incremented purely in +memory. That is only the truth while the engine is the sole writer of the field, +and it never is. Two holes, closed from the two ends that can see them. + +**1. An exempt writer's record number now lifts the counter (free).** `isSystem` +seed replay, a `preserveAudit` historical import and a `beforeInsert` hook stamp +all reach the "respect an explicit value" branch — #5503's strip exempts exactly +those three — so each persisted a number the counter never saw. The counter kept +issuing from the one-time seed, *below* the store's real max, and every number +up to that max was a duplicate business identifier. The supplied value is now +parsed with the same #6468 anchoring rules the seeding scan uses (extracted as +one shared reader, so the two readings cannot drift) and the counter is lifted to +it. Cost: one string parse, **no extra query** — a warm counter now converges on +what a cold re-seed of the same store would answer. + +Adoption deliberately never throws (an exempt write was accepted before and +still is), never lowers a counter, never seeds an *unseeded* counter (that would +skip the seeding scan and answer from one row), and ignores a value outside the +record's own counter scope (a historical import into a past date scope cannot +burn today's band). + +**2. A collision now re-seeds and re-issues instead of burning numbers.** A +counter sitting below the real max — because a writer *outside* this process +took numbers the engine could not observe — collided on every insert until it +walked past that max one number at a time; each failed create surfaced the +driver's raw error *and* advanced the counter, so it never converged on its own. +A unique-constraint failure attributable to an autonumber the engine issued now +drops the counter, re-seeds from the store and re-issues, bounded to 3 attempts; +past that the write fails with `code: 'ERR_AUTONUMBER_COLLISION'` carrying the +driver's error as `cause`, rather than the raw driver error. A conflict on a +different column, and any non-unique failure, are rethrown untouched. + +**This half is storage-dependent, and the docs name the drivers.** It is +triggered by the store rejecting the duplicate, so it reaches only drivers that +take this fallback path *and* enforce uniqueness. Measured across all five +in-repo drivers: only **driver-memory** (`supports = {}`) and **driver-mongodb** +(bit absent) take the path at all — driver-sql declares `autonumber: true`, and +driver-sqlite-wasm and driver-turso inherit it via `extends SqlDriver`. Of those +two, only driver-mongodb can raise a violation (a single-field unique index, when +the field declares `unique`); **driver-memory never does** — its `create` is a +`table.push()` storing no constraints at all — so there a duplicate still lands +silently and this branch is unreachable. + +So the collision retry protects essentially one backend, and that is now stated +in those terms rather than as "the storage layer". It is not a new claim: it is +the reading already ruled and gated in `scripts/driver-memory-census.ledger.json` +for `autonumber-seed-cross-side-parity.integration.test.ts` — "InMemoryDriver +declares `supports = {}`, so the ENGINE's autonumber seeding owns the counter. No +SQL backend can stand in". The silent-duplicate outcome is pinned rather than +left implied, and driver-memory is covered by the adoption half above, which +waits for no rejection. Enforcing uniqueness in the driver is the remedy for the +remaining case and is not attempted here. + +A **batch** insert drops the stale counter but is never re-issued (`bulkCreate` +may be partially applied, so re-writing it could duplicate the rows that did +land). `insert(object, rows[])` and `insertMany` therefore reject with the +**driver's own** duplicate-key error — never `ERR_AUTONUMBER_COLLISION`, which is +the single-row identity for "re-issued and still refused" — and the guarantee a +batch does get is that the next write re-seeds, so a caller's retry converges. + +The unique-violation questions are asked of `@objectstack/types`' +`isUniqueViolationError` / `uniqueViolationColumn` (#6250 / #6544), never a +dialect word-list of the engine's own. #6114's read-failure discrimination is +unchanged and now also covers the re-seed: a missing table still seeds from 0, +every other read failure still propagates and writes nothing. + +The counter stays **global** (not tenant-partitioned) — that remains parked per +#5495's disposition. diff --git a/packages/objectql/src/engine-autonumber-resync.test.ts b/packages/objectql/src/engine-autonumber-resync.test.ts new file mode 100644 index 0000000000..7293f4f319 --- /dev/null +++ b/packages/objectql/src/engine-autonumber-resync.test.ts @@ -0,0 +1,771 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * #6806 — the engine's fallback autonumber counter must RESYNC, from both ends. + * + * `applyAutonumbers` seeds `object.field.` from the store once and then + * increments purely in memory. That is only the truth while the engine is the + * sole writer of the field, and it never is: + * + * 1. **Exempt writers go through this very method.** `isSystem` seed replay, a + * `preserveAudit` historical import and a `beforeInsert` hook stamp all + * reach the "respect an explicit value" branch (#5503's strip exempts + * exactly those three), so each persists a record number the counter never + * saw. The counter then keeps issuing from the one-time seed — BELOW the + * store's real max — and every number up to that max is a duplicate + * business identifier. This is the warm-DB shape #5495's PROBE1 measured. + * 2. **Writers outside this process cannot be observed at all.** Another + * instance, a direct driver write, a restore. Those surface only as a + * unique-constraint failure on the create — and there was no collision + * handling on this path, so the driver's raw error reached the caller while + * the in-memory counter had already advanced. Each failed create burned a + * number and the NEXT insert collided too, one number at a time, until the + * counter walked past the real max (#5495's PROBE3). + * + * The fix answers each from the end that can see it: + * + * - **Adopt** (`adoptExplicitAutonumber`): lift the counter from the value an + * exempt writer supplied, read by #6468's anchoring rules — the SAME reading + * the seeding scan performs, now shared as `readAutonumberCounter`. Costs one + * string parse and NO query, and makes the warm counter converge on what a + * cold re-seed of the same store would answer. + * - **Re-seed on collision** (`createWithAutonumberResync`): drop the stale + * counter, re-seed from the store, re-issue — bounded. Costs nothing until a + * collision actually happens. + * + * Neither subsumes the other, which is why both are here: adoption covers drift + * the engine can observe but the store cannot report; collision-resync covers + * drift the store reports but the engine could not observe. + * + * ## The collision half is STORAGE-DEPENDENT, and says so — by driver name + * + * It is triggered by the store REJECTING the duplicate, so it reaches only + * drivers that take this fallback path AND enforce uniqueness. Measured across + * all five in-repo drivers (`supports.autonumber` read from each): + * driver-memory (`supports = {}`) and driver-mongodb (absent) take this path; + * driver-sql declares `autonumber: true`, and driver-sqlite-wasm and + * driver-turso inherit it (`extends SqlDriver`; Turso spreads + * `...super.supports`), so none of the three ever reaches here. Of the two that + * do, only driver-mongodb can raise anything — a single-field unique index, + * when the field declares `unique`. **driver-memory never can**: `create` is a + * `table.push()` storing no constraints at all (#4065), so a duplicate lands + * SILENTLY and this branch is unreachable. + * + * That is the repo's EXISTING ruled reading, not a fresh claim by this file: + * `scripts/driver-memory-census.ledger.json` records it for + * `packages/runtime/src/autonumber-seed-cross-side-parity.integration.test.ts` + * as `ruled-permanent` («#6664 A, maintainer 2026-08-08 — inherits #5704 + * Q2 = B») — "InMemoryDriver declares `supports = {}`, so the ENGINE's + * autonumber seeding owns the counter. No SQL backend can stand in". Section + * (3b) pins the consequence rather than authoring a second answer to the same + * question (#6832's one-contract-two-numbers shape). What covers driver-memory + * is adoption, which waits for no rejection. + * + * NOTE this file imports no driver package — the rig below is a hand-rolled + * fake driver — so it adds no `driver-memory` consumer and + * `check:driver-memory-census` sees no unledgered arrival. + * + * ## What is deliberately NOT changed + * + * - **#6114 / #5979 read-failure discrimination.** A missing table still seeds + * from 0; every other read failure still propagates and writes NOTHING — + * including on a RE-seed, which is the new call site. Pinned below. + * - **A conflict on some OTHER unique field** is rethrown untouched (#5495's + * disposition: «非本字段的冲突原样上抛»). + * - **The batch path is re-seeded but never re-issued.** `bulkCreate` may be + * partially applied, so re-writing a batch could duplicate the rows that did + * land. The stale counter is dropped and the driver's error is rethrown as + * before. + * + * The unique-violation questions are asked of `@objectstack/types`' + * `isUniqueViolationError` / `uniqueViolationColumn` (#6250 / #6544) — never a + * word-list of the engine's own (PD #12, precedent #5841). + * + * These tests drive a fake DRIVER (not a fake engine), so no engine write-verb + * dispatch contract is involved. + */ + +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; +import { ObjectQL } from './engine'; +import { SchemaRegistry } from './registry'; +import type { IDataDriver } from '@objectstack/spec/contracts'; + +vi.mock('./registry', () => { + const instance: any = { + getObject: vi.fn(), + resolveObject: vi.fn((n: string) => instance.getObject(n)), + registerObject: vi.fn(), + getObjectOwner: vi.fn(), + registerNamespace: vi.fn(), + registerKind: vi.fn(), + registerItem: vi.fn(), + registerApp: vi.fn(), + installPackage: vi.fn(), + reset: vi.fn(), + metadata: { get: vi.fn(() => new Map()) }, + }; + function SchemaRegistry() { + return instance; + } + Object.assign(SchemaRegistry, instance); + return { + SchemaRegistry, + computeFQN: (_ns: string | undefined, name: string) => name, + parseFQN: (fqn: string) => ({ namespace: undefined, shortName: fqn }), + RESERVED_NAMESPACES: new Set(['base', 'system']), + }; +}); + +/** Date tokens render from the wall clock, so the clock is pinned. */ +const FIXED_NOW = new Date('2026-06-15T09:00:00Z'); + +type Row = Record; + +/** + * Evaluate only the operators the seeding walk actually emits. Anything else + * throws rather than being tolerated — silently ignoring an unknown operator + * would let a bad query pass as a good one. (Same rig as the #6468 suffix + * tests, which is the point: these two files must agree about what the scan + * is allowed to send.) + */ +function matches(row: Row, where: any): boolean { + if (where == null) return true; + for (const [key, cond] of Object.entries(where)) { + if (key === '$and') { + if (!(cond as any[]).every((w) => matches(row, w))) return false; + continue; + } + if (key.startsWith('$')) throw new Error(`fake driver: unsupported logical operator ${key}`); + const v = row[key]; + if (cond !== null && typeof cond === 'object' && !Array.isArray(cond)) { + for (const [op, operand] of Object.entries(cond as Record)) { + if (op === '$startsWith') { + if (typeof v !== 'string' || !v.startsWith(String(operand))) return false; + } else if (op === '$gt') { + if (!(String(v) > String(operand))) return false; + } else if (op === '$eq') { + if (v !== operand) return false; + } else { + throw new Error(`fake driver: unsupported operator ${op}`); + } + } + } else if (v !== cond) { + return false; + } + } + return true; +} + +/* -------------------------------------------------------------------------- + * Collision fixtures — the shapes the two fallback drivers actually raise, + * plus the SQL shapes a third-party fallback driver would. + * ----------------------------------------------------------------------- */ + +/** + * driver-mongodb's duplicate-key error. `code` is 11000 (matched by no + * signature limb) and the message names the INDEX, never the column — so + * `isUniqueViolationError` says yes on the `duplicate key` limb while + * `uniqueViolationColumn` answers `undefined`. That combination is exactly why + * the resync treats an unnamed column as attributable: MongoDB is the ONE + * in-repo fallback driver that can raise a collision at all (driver-memory has + * no uniqueness constraints), and demanding a named column would make the + * resync unreachable on it. + */ +const mongoDuplicate = (field: string, value: string) => + Object.assign( + new Error( + `E11000 duplicate key error collection: app.doc index: ${field}_1 dup key: { ${field}: "${value}" }`, + ), + { code: 11000 }, + ); + +/** Postgres names the conflicting COLUMN in its DETAIL line — `uniqueViolationColumn` reads it. */ +const postgresDuplicate = (column: string, value: string) => + Object.assign(new Error(`duplicate key value violates unique constraint "doc_${column}_key"`), { + code: '23505', + detail: `Key (${column})=(${value}) already exists.`, + }); + +interface DriverOpts { + /** Reject a create whose autonumber value is already stored. */ + uniqueOn?: string; + /** Build the rejection. Defaults to the MongoDB shape. */ + duplicateError?: (field: string, value: string) => unknown; + /** Reject EVERY create as a duplicate, regardless of the store. */ + alwaysDuplicate?: boolean; +} + +/** + * A driver over a LIVE store: `create` appends, so a later seeding scan sees + * what earlier inserts wrote — and a test can push rows in directly to play the + * writer this engine cannot observe. + */ +function makeDriver(rows: Row[], opts: DriverOpts = {}) { + const created: Row[] = []; + const queries: any[] = []; + let findFails: (() => unknown) | null = null; + const dup = opts.duplicateError ?? mongoDuplicate; + const driver: any = { + name: 'memory', + version: '0.0.0', + // No `autonumber` support — this is exactly the engine fallback path. + supports: {}, + connect: vi.fn().mockResolvedValue(undefined), + disconnect: vi.fn().mockResolvedValue(undefined), + checkHealth: vi.fn().mockResolvedValue(true), + execute: vi.fn(), + find: vi.fn(async (_obj: string, ast: any) => { + if (findFails) throw findFails(); + queries.push(JSON.parse(JSON.stringify({ where: ast?.where, orderBy: ast?.orderBy, fields: ast?.fields }))); + let out = rows.filter((r) => matches(r, ast?.where)); + const orderBy = ast?.orderBy; + if (Array.isArray(orderBy) && orderBy.length > 0) { + const { field, order } = orderBy[0]; + out = [...out].sort((a, b) => { + const av = String(a[field] ?? ''); + const bv = String(b[field] ?? ''); + const cmp = av < bv ? -1 : av > bv ? 1 : 0; + return order === 'desc' ? -cmp : cmp; + }); + } + if (typeof ast?.limit === 'number') out = out.slice(0, ast.limit); + return out.map((r) => ({ ...r })); + }), + findOne: vi.fn(), + create: vi.fn(async (_obj: string, row: Row) => { + const field = opts.uniqueOn; + if (field) { + const value = String(row[field] ?? ''); + if (opts.alwaysDuplicate || rows.some((r) => r[field] === row[field])) { + created.push({ ...row, __rejected: true }); + throw dup(field, value); + } + } + created.push({ ...row }); + const stored = { id: `new${created.length}`, ...row }; + rows.push(stored); + return stored; + }), + update: vi.fn(), + delete: vi.fn(), + count: vi.fn(), + }; + driver.created = created; + driver.queries = queries; + /** Make every subsequent seeding read fail with `make()`; `null` restores it. */ + driver.breakReads = (make: (() => unknown) | null) => { + findFails = make; + }; + return driver as IDataDriver & { + created: Row[]; + queries: any[]; + breakReads: (make: (() => unknown) | null) => void; + }; +} + +function schemaWith(field: string, format?: string, extra?: Record) { + return { + name: 'doc', + fields: { + title: { type: 'text' }, + ...(extra ?? {}), + ...(format === undefined + ? { [field]: { type: 'autonumber', required: true } } + : { [field]: { type: 'autonumber', required: true, format } }), + }, + }; +} + +const rowId = (n: number) => `r${String(n).padStart(6, '0')}`; + +/** Stored rows carrying pre-existing record numbers, in insertion order. */ +function storedRows(field: string, values: string[]): Row[] { + return values.map((v, i) => ({ id: rowId(i + 1), [field]: v })); +} + +function makeRig(schema: any, rows: Row[], opts: DriverOpts = {}) { + vi.mocked(SchemaRegistry.getObject).mockReturnValue(schema as any); + const driver = makeDriver(rows, opts); + const engine = new ObjectQL(); + engine.registerDriver(driver, true); + return { engine, driver, rows }; +} + +/** Seeding reads carry the projection `['id', ]` — that is how they are told apart. */ +const seedReads = (driver: { queries: any[] }, field: string) => + driver.queries.filter((q) => Array.isArray(q.fields) && q.fields.includes(field)); + +describe('ObjectQL autonumber resync (#6806)', () => { + beforeEach(() => { + vi.clearAllMocks(); + vi.useFakeTimers({ toFake: ['Date'] }); + vi.setSystemTime(FIXED_NOW); + }); + + afterEach(() => { + vi.useRealTimers(); + }); + + /* ====================================================================== * + * (1) Adoption — an exempt writer's number lifts the counter (PROBE1) + * ==================================================================== */ + + describe('an exempt writer\'s record number lifts the seeded counter', () => { + const SCHEMA = schemaWith('doc_no', 'D-{0000}'); + + it('isSystem seed replay — the counter follows the replayed number', async () => { + const { engine, driver } = makeRig(SCHEMA, storedRows('doc_no', ['D-0003'])); + await engine.init(); + + // Seeds from the store (max 3) and issues 4. The counter is now warm. + expect((await engine.insert('doc', { title: 'first' })).doc_no).toBe('D-0004'); + + // A seed replay writes D-0009 straight through: `isSystem` skips #5503's + // strip entirely, so the value reaches `applyAutonumbers` intact. + const replayed = await engine.insert( + 'doc', + { title: 'replayed', doc_no: 'D-0009' }, + { context: { isSystem: true } } as any, + ); + expect(replayed.doc_no).toBe('D-0009'); + + // The defect: the counter stayed at 4 and re-issued D-0005..D-0009, + // duplicating a business identifier five times over. + expect((await engine.insert('doc', { title: 'after' })).doc_no).toBe('D-0010'); + + // ...and it did it by ADOPTING, not by re-reading: one seeding scan for + // the whole sequence. The cost of this resync is a string parse. + expect(seedReads(driver, 'doc_no')).toHaveLength(1); + }); + + it('preserveAudit historical import — same lift, same free cost', async () => { + const { engine, driver } = makeRig(SCHEMA, storedRows('doc_no', ['D-0003'])); + await engine.init(); + + expect((await engine.insert('doc', { title: 'first' })).doc_no).toBe('D-0004'); + + const imported = await engine.insert( + 'doc', + { title: 'legacy', doc_no: 'D-0021' }, + { context: { preserveAudit: true } } as any, + ); + expect(imported.doc_no).toBe('D-0021'); + + expect((await engine.insert('doc', { title: 'after' })).doc_no).toBe('D-0022'); + expect(seedReads(driver, 'doc_no')).toHaveLength(1); + }); + + it('a beforeInsert hook stamp — the third exempt writer (#6339)', async () => { + const { engine } = makeRig(SCHEMA, storedRows('doc_no', ['D-0003'])); + await engine.init(); + + expect((await engine.insert('doc', { title: 'first' })).doc_no).toBe('D-0004'); + + // A hook-written key is not caller-supplied, so the strip keeps it. + engine.registerHook( + 'beforeInsert', + async (ctx: any) => { + if (ctx.input.data.title === 'stamped') ctx.input.data.doc_no = 'D-0030'; + }, + { object: 'doc' }, + ); + expect((await engine.insert('doc', { title: 'stamped' })).doc_no).toBe('D-0030'); + + expect((await engine.insert('doc', { title: 'after' })).doc_no).toBe('D-0031'); + }); + + it('adopts across a batch of exempt rows', async () => { + const { engine } = makeRig(SCHEMA, storedRows('doc_no', ['D-0003'])); + await engine.init(); + + expect((await engine.insert('doc', { title: 'first' })).doc_no).toBe('D-0004'); + + await engine.insert( + 'doc', + [ + { title: 'a', doc_no: 'D-0011' }, + { title: 'b', doc_no: 'D-0014' }, + { title: 'c', doc_no: 'D-0012' }, + ], + { context: { isSystem: true } } as any, + ); + + // The MAX of the batch wins, not the last row seen. + expect((await engine.insert('doc', { title: 'after' })).doc_no).toBe('D-0015'); + }); + }); + + /* ====================================================================== * + * (2) Adoption's four refusals — the ways it could do harm instead + * ==================================================================== */ + + describe('adoption never harms', () => { + const SCHEMA = schemaWith('doc_no', 'D-{0000}'); + + it('never LOWERS a counter that has already issued numbers', async () => { + const { engine } = makeRig(SCHEMA, storedRows('doc_no', ['D-0003'])); + await engine.init(); + + expect((await engine.insert('doc', { title: 'first' })).doc_no).toBe('D-0004'); + + // A legacy import reinstating an OLD number must not rewind the counter + // onto numbers it has already issued. + await engine.insert('doc', { title: 'old', doc_no: 'D-0001' }, { context: { preserveAudit: true } } as any); + + expect((await engine.insert('doc', { title: 'after' })).doc_no).toBe('D-0005'); + }); + + it('does not adopt into an UNSEEDED counter — the seeding scan still runs', async () => { + // The store already holds D-0050. If the exempt value were written in as + // the seed, the scan would be skipped and the next number (D-0010) would + // duplicate an existing row — the very defect, arrived at from the fix. + const { engine, driver } = makeRig(SCHEMA, storedRows('doc_no', ['D-0050'])); + await engine.init(); + + await engine.insert('doc', { title: 'replay', doc_no: 'D-0009' }, { context: { isSystem: true } } as any); + + expect((await engine.insert('doc', { title: 'first' })).doc_no).toBe('D-0051'); + expect(seedReads(driver, 'doc_no')).toHaveLength(1); + }); + + it('does not let another SCOPE\'s value lift this scope\'s counter', async () => { + const DATED = schemaWith('doc_no', 'AD{YYYYMMDD}-{0000}'); + const { engine } = makeRig(DATED, storedRows('doc_no', ['AD20260615-0003'])); + await engine.init(); + + expect((await engine.insert('doc', { title: 'first' })).doc_no).toBe('AD20260615-0004'); + + // A historical import into a PAST day. Its counter is a different key; + // reading 900 into today's would burn most of today's band. + await engine.insert( + 'doc', + { title: 'legacy', doc_no: 'AD20240101-0900' }, + { context: { preserveAudit: true } } as any, + ); + + expect((await engine.insert('doc', { title: 'after' })).doc_no).toBe('AD20260615-0005'); + }); + + it('never throws on a value it cannot parse or place', async () => { + const SCOPED = schemaWith('doc_no', '{island_zone}-{000}', { island_zone: { type: 'text' } }); + const { engine } = makeRig(SCOPED, storedRows('doc_no', ['A-003'])); + await engine.init(); + + expect((await engine.insert('doc', { title: 'first', island_zone: 'A' })).doc_no).toBe('A-004'); + + // The interpolated field is empty, so no scope can be rendered. Learning + // nothing is the answer; rejecting an exempt writer's row is not — it was + // accepted before this resync existed. + const odd = await engine.insert( + 'doc', + { title: 'odd', doc_no: 'not-a-number' }, + { context: { isSystem: true } } as any, + ); + expect(odd.doc_no).toBe('not-a-number'); + + expect((await engine.insert('doc', { title: 'after', island_zone: 'A' })).doc_no).toBe('A-005'); + }); + + it('still REFUSES to generate when an interpolated field is empty (control)', async () => { + const SCOPED = schemaWith('doc_no', '{island_zone}-{000}', { island_zone: { type: 'text' } }); + const { engine } = makeRig(SCOPED, []); + await engine.init(); + + // The generating branch is untouched: only the adopt branch went quiet. + await expect(engine.insert('doc', { title: 'no zone' })).rejects.toThrow(/island_zone/); + }); + }); + + /* ====================================================================== * + * (3) Collision — re-seed and re-issue instead of burning numbers (PROBE3) + * ==================================================================== */ + + describe('a collision re-seeds and re-issues rather than burning numbers', () => { + const SCHEMA = schemaWith('doc_no', 'D-{0000}'); + + it('MongoDB E11000: the write SUCCEEDS on the number the re-seed found', async () => { + const { engine, driver, rows } = makeRig(SCHEMA, storedRows('doc_no', ['D-0003']), { + uniqueOn: 'doc_no', + }); + await engine.init(); + + expect((await engine.insert('doc', { title: 'first' })).doc_no).toBe('D-0004'); + + // A writer this engine cannot observe takes 0005..0007. + rows.push(...storedRows('doc_no', ['D-0005', 'D-0006', 'D-0007']).map((r, i) => ({ ...r, id: `x${i}` }))); + + const recovered = await engine.insert('doc', { title: 'second' }); + + // Attempt 1 collided on D-0005; the re-seed read the real max (7) back. + expect(recovered.doc_no).toBe('D-0008'); + expect(driver.create).toHaveBeenCalledTimes(3); // 1 (first insert) + 2 (collide, re-issue) + expect(seedReads(driver, 'doc_no')).toHaveLength(2); + }); + + it('converges — the NEXT insert continues from the re-seeded counter', async () => { + const { engine, driver, rows } = makeRig(SCHEMA, storedRows('doc_no', ['D-0003']), { + uniqueOn: 'doc_no', + }); + await engine.init(); + await engine.insert('doc', { title: 'first' }); + rows.push(...storedRows('doc_no', ['D-0005', 'D-0006', 'D-0007']).map((r, i) => ({ ...r, id: `x${i}` }))); + await engine.insert('doc', { title: 'second' }); + + // The defect's real cost was that it never converged: the stale counter + // survived the failure, so 0006 and 0007 collided in turn — one burned + // number and one raw driver error per insert until it walked past the max. + expect((await engine.insert('doc', { title: 'third' })).doc_no).toBe('D-0009'); + expect(seedReads(driver, 'doc_no')).toHaveLength(2); + }); + + it('Postgres naming the autonumber COLUMN is attributed the same way', async () => { + const { engine, rows } = makeRig(SCHEMA, storedRows('doc_no', ['D-0003']), { + uniqueOn: 'doc_no', + duplicateError: postgresDuplicate, + }); + await engine.init(); + await engine.insert('doc', { title: 'first' }); + rows.push({ id: 'x1', doc_no: 'D-0005' }); + + expect((await engine.insert('doc', { title: 'second' })).doc_no).toBe('D-0006'); + }); + + it('a conflict on a DIFFERENT column is rethrown untouched, with no re-issue', async () => { + // «非本字段的冲突原样上抛» — #5495's disposition. A duplicate email is the + // caller's business error; re-issuing a record number cannot fix it, and + // swallowing it into an engine error would hide what actually failed. + const { engine, driver } = makeRig(SCHEMA, storedRows('doc_no', ['D-0003']), { + uniqueOn: 'doc_no', + alwaysDuplicate: true, + duplicateError: () => postgresDuplicate('email', 'a@b.example'), + }); + await engine.init(); + + await expect(engine.insert('doc', { title: 'first' })).rejects.toMatchObject({ + code: '23505', + detail: 'Key (email)=(a@b.example) already exists.', + }); + expect(driver.create).toHaveBeenCalledTimes(1); + }); + + it('a non-unique driver failure is rethrown untouched, with no re-issue', async () => { + const { engine, driver } = makeRig(SCHEMA, storedRows('doc_no', ['D-0003']), { + uniqueOn: 'doc_no', + alwaysDuplicate: true, + duplicateError: () => Object.assign(new Error('deadlock detected'), { code: '40P01' }), + }); + await engine.init(); + + await expect(engine.insert('doc', { title: 'first' })).rejects.toThrow(/deadlock detected/); + expect(driver.create).toHaveBeenCalledTimes(1); + }); + + it('refuses with a NAMED error after the bounded attempts, keeping the driver error as cause', async () => { + const { engine, driver } = makeRig(SCHEMA, storedRows('doc_no', ['D-0003']), { + uniqueOn: 'doc_no', + alwaysDuplicate: true, + }); + await engine.init(); + + const failure = await engine.insert('doc', { title: 'first' }).then( + () => { throw new Error('expected the insert to be refused'); }, + (e) => e as any, + ); + + // An identity a caller can branch on — not the driver's raw error, and + // not a bare throw. + expect(failure.code).toBe('ERR_AUTONUMBER_COLLISION'); + expect(failure.message).toMatch(/doc_no/); + expect(failure.message).toMatch(/No record was written/); + // The driver's own diagnosis is preserved rather than replaced. + expect(String((failure.cause as Error)?.message)).toMatch(/E11000/); + // Bounded: three attempts, not a spin against a live competitor. + expect(driver.create).toHaveBeenCalledTimes(3); + }); + + it('burns no numbers when it refuses — the next insert starts from the store', async () => { + const rows = storedRows('doc_no', ['D-0003']); + let rejecting = true; + const { engine } = makeRig(SCHEMA, rows, { + uniqueOn: 'doc_no', + get alwaysDuplicate() { return rejecting; }, + } as DriverOpts); + await engine.init(); + + await expect(engine.insert('doc', { title: 'doomed' })).rejects.toMatchObject({ + code: 'ERR_AUTONUMBER_COLLISION', + }); + + // Three failed creates advanced the in-memory counter three times before + // this fix. The refusal drops the counter instead, so the store — still + // holding only D-0003 — decides the next number. + rejecting = false; + expect((await engine.insert('doc', { title: 'after' })).doc_no).toBe('D-0004'); + }); + + it('does not re-issue for a value the ENGINE did not issue', async () => { + const { engine, driver } = makeRig(SCHEMA, storedRows('doc_no', ['D-0003']), { + uniqueOn: 'doc_no', + }); + await engine.init(); + + // An exempt writer replaying a number that is already taken. The engine + // issued nothing on this row, so there is nothing of its own to re-issue + // and the collision is the writer's to see. + await expect( + engine.insert('doc', { title: 'replay', doc_no: 'D-0003' }, { context: { isSystem: true } } as any), + ).rejects.toThrow(/E11000/); + expect(driver.create).toHaveBeenCalledTimes(1); + }); + + it('a batch collision drops the counter but does NOT re-issue the batch', async () => { + // `bulkCreate` may be partially applied, so re-writing is worse than the + // collision. What must not survive is the stale counter. + const rows = storedRows('doc_no', ['D-0003']); + const { engine, driver } = makeRig(SCHEMA, rows, { uniqueOn: 'doc_no' }); + await engine.init(); + await engine.insert('doc', { title: 'first' }); // D-0004, counter warm at 4 + // The unobservable writer takes a whole BAND, not one number: dropping the + // counter and merely advancing it past the collision are otherwise + // indistinguishable here, and only the drop reaches the real max. + rows.push(...storedRows('doc_no', ['D-0005', 'D-0006', 'D-0007', 'D-0008', 'D-0009']).map((r, i) => ({ ...r, id: `x${i}` }))); + + // What an author actually gets on a batch: the DRIVER's own error, not + // the single-row path's `ERR_AUTONUMBER_COLLISION` — because nothing was + // re-issued, so "re-issued and still refused" would be a false statement. + const failure = await engine.insert('doc', [{ title: 'a' }]).then( + () => { throw new Error('expected the batch to be refused'); }, + (e) => e as any, + ); + expect(failure.message).toMatch(/E11000/); + expect(failure.code).toBe(11000); + expect(failure.code).not.toBe('ERR_AUTONUMBER_COLLISION'); + const createsAfterBatch = (driver.create as any).mock.calls.length; + + // The counter was dropped, so the next single insert RE-SEEDS and lands + // above the whole band (max 9). Merely advancing the stale counter would + // hand back D-0006 — a number the unobservable writer already took. That + // is the guarantee a batch DOES get: the caller's retry converges. + expect((await engine.insert('doc', { title: 'after' })).doc_no).toBe('D-0010'); + expect((driver.create as any).mock.calls.length).toBe(createsAfterBatch + 1); + }); + + it('insertMany reports the same way — driver error, counter dropped', async () => { + const rows = storedRows('doc_no', ['D-0003']); + const { engine } = makeRig(SCHEMA, rows, { uniqueOn: 'doc_no' }); + await engine.init(); + await engine.insert('doc', { title: 'first' }); + rows.push(...storedRows('doc_no', ['D-0005', 'D-0006', 'D-0007', 'D-0008', 'D-0009']).map((r, i) => ({ ...r, id: `x${i}` }))); + + // Partial-row mode culls rows that fail PREPARATION; a driver write that + // fails is still a whole-call rejection, so this is the same contract. + await expect(engine.insertMany('doc', [{ title: 'a' }])).rejects.toMatchObject({ code: 11000 }); + + expect((await engine.insert('doc', { title: 'after' })).doc_no).toBe('D-0010'); + }); + }); + + /* ====================================================================== * + * (3b) The collision half is STORAGE-DEPENDENT — name which driver gives + * which guarantee, rather than implying one that is not delivered + * ==================================================================== */ + + describe('what a driver with no uniqueness constraint gets (driver-memory)', () => { + const SCHEMA = schemaWith('doc_no', 'D-{0000}'); + + it('a duplicate lands SILENTLY — the collision branch cannot be reached', async () => { + // `InMemoryDriver.create` is a `table.push()` storing no constraints of + // any kind (its own docstring since #4065 — it calls itself a WEAK + // oracle). So a duplicate raises nothing, there is no error to catch, and + // the re-issue this file pins elsewhere never runs. Recorded rather than + // papered over (PD #10): "collisions are handled" would be FALSE on one + // of the two drivers this fallback path serves — and the fallback path is + // only those two of five, the other three inheriting SqlDriver's + // `autonumber: true`. So the retry protects ONE backend, and this is the + // other one. + // + // No `uniqueOn` — this rig is the memory driver's shape exactly. + const rows = storedRows('doc_no', ['D-0003']); + const { engine, driver } = makeRig(SCHEMA, rows); + await engine.init(); + await engine.insert('doc', { title: 'first' }); // D-0004 + + // The writer this engine cannot observe takes D-0005. + rows.push({ id: 'x1', doc_no: 'D-0005' }); + + const written = await engine.insert('doc', { title: 'second' }); + + // The honest outcome: the number is issued a second time, the write + // SUCCEEDS, and nothing anywhere says so. Fixing this needs uniqueness in + // the driver — `packages/drivers/**` is under the #5499 freeze, and a + // pre-issue existence probe in the engine would cost a query per insert + // and still be racy. Reported as a follow-up, not implemented here. + expect(written.doc_no).toBe('D-0005'); + expect(rows.filter((r) => r.doc_no === 'D-0005')).toHaveLength(2); + // One create attempt: with no rejection there is nothing to retry. + expect(driver.create).toHaveBeenCalledTimes(2); // 'first' + 'second' + }); + + it('...but ADOPTION still holds there — it needs no constraint at all', async () => { + // The half that does cover driver-memory: drift the engine can observe + // is fixed without waiting for anyone to reject anything. + const { engine } = makeRig(SCHEMA, storedRows('doc_no', ['D-0003'])); + await engine.init(); + + expect((await engine.insert('doc', { title: 'first' })).doc_no).toBe('D-0004'); + await engine.insert('doc', { title: 'replay', doc_no: 'D-0009' }, { context: { isSystem: true } } as any); + + expect((await engine.insert('doc', { title: 'after' })).doc_no).toBe('D-0010'); + }); + }); + + /* ====================================================================== * + * (4) #6114 / #5979 — read-failure discrimination survives the new call site + * ==================================================================== */ + + describe('read-failure discrimination survives (#6114 / #5979)', () => { + const SCHEMA = schemaWith('doc_no', 'D-{0000}'); + + it('a missing table still seeds from 0 (benign, unchanged)', async () => { + const { engine, driver } = makeRig(SCHEMA, []); + driver.breakReads(() => Object.assign(new Error('relation "doc" does not exist'), { code: '42P01' })); + await engine.init(); + + expect((await engine.insert('doc', { title: 'first' })).doc_no).toBe('D-0001'); + }); + + it('an outage during the FIRST seed propagates and writes nothing (unchanged)', async () => { + const { engine, driver } = makeRig(SCHEMA, storedRows('doc_no', ['D-0007'])); + driver.breakReads(() => Object.assign(new Error('connect ECONNREFUSED 127.0.0.1:5432'), { code: 'ECONNREFUSED' })); + await engine.init(); + + await expect(engine.insert('doc', { title: 'first' })).rejects.toThrow(/ECONNREFUSED/); + expect(driver.create).not.toHaveBeenCalled(); + }); + + it('an outage during the RE-seed propagates — never swallowed into 0 or a stale value', async () => { + // The new call site. If the re-seed answered 0 the way the pre-#5979 + // bare catch did, this insert would come back D-0001 against a store + // holding D-0007 — the #5979 regression line, arrived at through #6806. + const rows = storedRows('doc_no', ['D-0003']); + const { engine, driver } = makeRig(SCHEMA, rows, { uniqueOn: 'doc_no' }); + await engine.init(); + await engine.insert('doc', { title: 'first' }); // D-0004 + rows.push(...storedRows('doc_no', ['D-0005', 'D-0006', 'D-0007']).map((r, i) => ({ ...r, id: `x${i}` }))); + + driver.breakReads(() => Object.assign(new Error('Connection terminated unexpectedly'), { code: '08006' })); + const createsBefore = (driver.create as any).mock.calls.length; + + await expect(engine.insert('doc', { title: 'second' })).rejects.toThrow(/Connection terminated/); + // Exactly ONE create was attempted after the outage began: the collision, + // whose re-seed then failed. No number was forged from data never read. + expect((driver.create as any).mock.calls.length).toBe(createsBefore + 1); + + // And the failed re-seed poisoned nothing: once the store is readable the + // counter comes back from the real max, not from 0 and not from the + // burned in-memory value. + driver.breakReads(null); + expect((await engine.insert('doc', { title: 'third' })).doc_no).toBe('D-0008'); + }); + }); +}); diff --git a/packages/objectql/src/engine.ts b/packages/objectql/src/engine.ts index 48d5d1a653..6ef68f7e87 100644 --- a/packages/objectql/src/engine.ts +++ b/packages/objectql/src/engine.ts @@ -87,6 +87,14 @@ import { resolveAllowDriverConnectFailure } from '@objectstack/types'; // vocabulary of "benign driver error" is the exact debt that module exists to // retire, and `check:durability-log-level` exempts only this declared name. import { isMissingTableError } from '@objectstack/metadata/errors'; +// [#6806] The ONE shared "is this driver error a unique-constraint violation?" +// predicate, and its narrower companion "which column conflicted" (#6250 / +// #6544, both landed in `@objectstack/types`). The engine's autonumber +// collision resync asks exactly the two questions those exports were named to +// answer, so it asks THEM — hand-writing a fifth dialect word-list inside the +// engine is the consumer-side tolerant parsing PD #12 forbids and precedent +// #5841 retired. +import { isUniqueViolationError, uniqueViolationColumn } from '@objectstack/types'; /** * Per-row outcome of {@link ObjectQL.insertMany} (framework#3172). One entry @@ -312,6 +320,76 @@ const ENGINE_AGGREGATE_OPTION_KEYS: ReadonlySet = new Set([ */ const AUTONUMBER_SEED_PAGE_SIZE = 5000; +/** + * How many times one insert may re-seed and re-issue after a unique-constraint + * collision on an engine-issued autonumber (#6806) before it refuses. + * + * Bounded on purpose. The FIRST re-issue is the one that matters: the collision + * proves the in-memory counter sits below the store's real max, and the re-seed + * that follows reads that max back, so attempt 2 is issued from the truth. A + * further collision means another writer took the number in between — real + * concurrency, which more spinning does not resolve (each attempt costs a full + * scope scan). Two spare attempts absorb a burst; past that the write fails + * loudly rather than looping against a live competitor. + */ +const AUTONUMBER_COLLISION_ATTEMPTS = 3; + +/** + * One autonumber the ENGINE issued on a row, paired with the counter it came + * from (#6806). Only engine-issued values are listed: a value an exempt writer + * supplied is the caller's, and a collision on it is the caller's to see. + */ +interface IssuedAutonumber { + /** Field the value was written to. */ + readonly field: string; + /** `object.field.` key in {@link ObjectQL.autonumberCounters}. */ + readonly counterKey: string; +} + +/** + * Read the counter out of ONE stored autonumber value, under #6468's anchoring + * rules. Shared by the seeding scan and by the adopt-on-exempt-write resync + * (#6806) so both readings can never drift apart — a divergence here is a + * duplicate record number, which is the harm the whole family is about. + * + * `prefix` and `suffix` are `renderAutonumber`'s own declared output; this + * function derives no format understanding of its own (see `seedAutonumber`'s + * "Locating the counter inside a stored value" section for the full rationale): + * + * - **Either one declared ⇒ the slot is ANCHORED**: the counter is the digit + * run at the START of what follows the prefix, after removing the declared + * suffix when this row carries it. The suffix is stripped when it matches, + * never required to match — a dynamic suffix renders differently per row + * while the counter scope is the rendered PREFIX, so those rows share this + * counter and must still be read. + * - **Neither declared ⇒ UNANCHORED**: the legacy reading — the LAST digit + * run of the whole value. + * + * A value outside the scope (it does not carry the rendered prefix) reads as + * `undefined`: it belongs to another counter and must not lift this one. + * + * Both branches use linear `/\d+/` forms — a backtracking lookahead here is a + * polynomial-ReDoS sink on stored values full of zeros (CodeQL + * js/polynomial-redos). + */ +function readAutonumberCounter(value: string, prefix: string, suffix: string): number | undefined { + if (prefix && !value.startsWith(prefix)) return undefined; + const anchored = prefix !== '' || suffix !== ''; + let digits: string | undefined; + if (anchored) { + let core = value.slice(prefix.length); + if (suffix && core.endsWith(suffix)) core = core.slice(0, core.length - suffix.length); + const head = core.match(/^\d+/); + digits = head ? head[0] : undefined; + } else { + const runs = value.match(/\d+/g); + digits = runs ? runs[runs.length - 1] : undefined; + } + if (!digits) return undefined; + const n = parseInt(digits, 10); + return Number.isFinite(n) ? n : undefined; +} + /** Tombstoned option keys: rejected with the spec's own removal notice. */ const ENGINE_RETIRED_OPTION_MESSAGES: Record = { cursor: QUERY_CURSOR_REMOVED, @@ -2343,29 +2421,68 @@ export class ObjectQL implements IObjectQLEngine { * interpolation (`{island_zone}{000}`) and per-scope reset behave identically * to the SQL driver's persistent sequence (#1603). NOTE: this in-memory seeding * is single-instance. + * + * # Keeping the seeded counter in sync (#6806) + * + * Seeding "once per counter key" is only the truth while the engine is the + * ONLY writer of the field. It is not: the `continue` below is reached by + * every exempt writer (`isSystem` seed replay, a `preserveAudit` historical + * import, a `beforeInsert` hook stamp), and each of those persists a record + * number the counter never saw. The counter then keeps issuing from where the + * one-time seed left it — below the store's real max — and every number it + * issues up to that max is a duplicate business identifier. Two resyncs + * close that, from opposite ends: + * + * - **Adopt, here.** An exempt value passes through this very loop, so the + * counter can be lifted from it for FREE — one string parse, no query. + * {@link adoptExplicitAutonumber} does it, which makes the warm in-memory + * counter converge on what a cold re-seed of the same store would answer. + * This is the whole fix for in-process drift, and it costs nothing on the + * generating path (a caller-supplied value never reaches this method + * unless the writer is exempt). + * - **Re-seed on collision**, at the write. Adoption cannot see a writer + * outside this process (another instance, a direct driver write, a + * restore). Those surface as a unique-constraint failure on the create, + * and {@link createWithAutonumberResync} answers it by dropping the stale + * counter, re-seeding from the store and re-issuing — bounded. It costs + * nothing until a collision actually happens. + * + * Both are required and neither subsumes the other: adoption covers drift the + * engine can observe but the store cannot report, collision-resync covers + * drift the store reports but the engine could not observe. + * + * @returns the autonumbers this call ISSUED, in field order — the input + * {@link createWithAutonumberResync} needs to decide whether a unique + * violation is one of its own numbers. An adopted (caller-supplied) value is + * deliberately NOT listed: it is not the engine's to re-issue. */ private async applyAutonumbers( object: string, record: Record, execCtx?: ExecutionContext, driverOwnsAutonumber?: boolean, - ): Promise { - if (driverOwnsAutonumber) return; // driver generates persistently in create() + ): Promise { + if (driverOwnsAutonumber) return []; // driver generates persistently in create() const fields = (this.getSchema(object) as any)?.fields; - if (!fields || typeof fields !== 'object' || Array.isArray(fields)) return; + if (!fields || typeof fields !== 'object' || Array.isArray(fields)) return []; const now = new Date(); const timezone = execCtx?.timezone; + const issued: IssuedAutonumber[] = []; for (const [name, def] of Object.entries(fields)) { if ((def as any)?.type !== 'autonumber') continue; - const current = record[name]; - // Respect an explicit value — reachable only for an EXEMPT writer now - // (isSystem / preserveAudit / a hook stamp): #5503's strip removed every - // other caller's value before this method was called. - if (current != null && current !== '') continue; // Honor either the spec-canonical `autonumberFormat` or the shorthand // `format` (both appear in metadata; the driver reads both too) — #1603. const fmt = (def as any).autonumberFormat ?? (def as any).format; const tokens = parseAutonumberFormat(typeof fmt === 'string' ? fmt : ''); + const current = record[name]; + // Respect an explicit value — reachable only for an EXEMPT writer now + // (isSystem / preserveAudit / a hook stamp): #5503's strip removed every + // other caller's value before this method was called. Respecting it is + // unchanged; what is new is that the counter LEARNS from it (#6806). + if (current != null && current !== '') { + this.adoptExplicitAutonumber(object, name, tokens, record, String(current), now, timezone); + continue; + } // Refuse to generate when an interpolated `{field}` is empty — it would // render to an empty prefix and merge this record into the wrong counter // scope. Mirror the SQL driver so both paths fail identically (#1603). @@ -2387,7 +2504,228 @@ export class ObjectQL implements IObjectQLEngine { next += 1; this.autonumberCounters.set(counterKey, next); record[name] = renderAutonumber({ tokens, seq: next, record, now, timezone }).value; + issued.push({ field: name, counterKey }); } + return issued; + } + + /** + * Lift the in-memory counter to a record number an EXEMPT writer supplied + * (#6806) — the free half of the resync, and the one that closes the shape + * #5495's PROBE1 measured on a warm database. + * + * The value is parsed with {@link readAutonumberCounter}, i.e. by exactly the + * anchoring rules #6468 gave the seeding scan, against the prefix/suffix this + * record's own format renders. So adopting is the same reading a cold re-seed + * would perform over the same row — which is the invariant to hold on to: a + * warm counter must answer what a restart would answer. + * + * Four deliberate refusals, each one a way this could do harm: + * + * - **Never throws.** An exempt write is not the engine's to reject, and it + * was not rejected before this method existed. A format whose `{field}` + * interpolation is empty, or a value that parses to nothing, simply + * teaches the counter nothing. + * - **Only LIFTS a counter that is already seeded.** With no seed in hand, + * writing this value in would SKIP the seeding scan and answer from one + * row — below the real max whenever the store holds a higher number, i.e. + * the duplicate-number defect itself. Doing nothing is correct: the row is + * persisted, so the first generating insert's own scan reads it. + * - **Never lowers.** A counter that has already issued numbers must not go + * back over them; the max is a floor that only rises. + * - **Only within this record's scope.** `readAutonumberCounter` returns + * `undefined` for a value that does not carry the rendered prefix, so a + * historical import into last month's date scope cannot lift THIS + * month's counter. Its own scope's counter is left untouched, which is + * harmless: a scope is derived from the write instant, so a past scope's + * counter is not one a subsequent generating insert can reach. + * + * Adoption runs BEFORE the driver write, so an exempt insert that then fails + * leaves the counter lifted over a number nobody took — a gap. Gaps are + * already the documented cost of this path (a failed attempt consumes its + * value; the counter is "resilient to deletions"), and the direction is the + * safe one: a gap is a cosmetic surprise, a duplicate is a corrupted business + * identifier. + */ + private adoptExplicitAutonumber( + object: string, + field: string, + tokens: ReturnType, + record: Record, + value: string, + now: Date, + timezone?: string, + ): void { + let probe: ReturnType; + try { + // An empty `{field}` would render an empty prefix and point at the wrong + // counter — the same hazard the generating branch throws on. Here it is a + // reason to learn nothing, never a reason to fail the caller's write. + if (missingFieldValues(tokens, record).length > 0) return; + probe = renderAutonumber({ tokens, seq: 0, record, now, timezone }); + } catch { + return; + } + const counterKey = `${object}.${field}.${probe.scope}`; + const seeded = this.autonumberCounters.get(counterKey); + if (seeded == null) return; // not seeded yet — the first seed scan will read this row + const supplied = readAutonumberCounter(value, probe.prefix, probe.suffix); + if (supplied == null || supplied <= seeded) return; + this.autonumberCounters.set(counterKey, supplied); + this.logger.debug('Autonumber counter lifted to an externally supplied value', { + object, field, counterKey, from: seeded, to: supplied, + }); + } + + /** + * Create ONE record, re-seeding and re-issuing when the driver rejects an + * autonumber the engine itself issued as a duplicate (#6806). + * + * # What this is for + * + * A counter that sits below the store's real max — because a writer outside + * this process took numbers the engine could not observe — collides on every + * insert until it has walked past that max one number at a time. Before this, + * each of those inserts failed with the driver's raw error AND advanced the + * counter, so a warm-database storm burned a number per failed create and + * never converged on its own (#5495's PROBE3). Dropping the counter on the + * collision is what converges it: the re-seed reads the true max back, and + * the next attempt is issued from it. + * + * # Which failures qualify + * + * - The engine must have ISSUED an autonumber on this row. A row whose + * numbers all came from an exempt writer has nothing here to re-issue. + * - The error must be a unique violation, per `isUniqueViolationError` + * (#6250) — never a word-list of this method's own. + * - When the dialect names the conflicting COLUMN (`uniqueViolationColumn`, + * #6544), it must be one of the fields the engine issued. A conflict on + * some other unique field is the caller's business error and is rethrown + * untouched, exactly as #5495's disposition ruled («非本字段的冲突原样上抛»). + * When the dialect names no determinable column the attribution falls back + * to "the engine issued a number on this row, and the row was refused as a + * duplicate" — deliberately, because `uniqueViolationColumn` answers + * `undefined` for every index-named dialect, which includes MongoDB's + * `E11000 ... index: doc_no_1`, i.e. the ONE fallback driver that can + * raise this at all. Requiring a named column would make the resync + * unreachable on exactly the driver that needs it. The cost of the + * fallback is a wasted re-issue when an unrelated unique field is what + * actually conflicted: the second attempt fails the same way, and the + * original error is what the caller finally sees. + * + * # ⚠ The guarantee is STORAGE-DEPENDENT — say so, do not imply otherwise + * + * This whole branch is triggered by the storage layer REJECTING the duplicate, + * so it reaches only drivers that (1) take this fallback path at all and + * (2) enforce uniqueness. Measured across all five in-repo drivers: + * + * | driver | `supports.autonumber` | fallback path? | uniqueness on the column | a collision appears as | + * |:---|:---|:---|:---|:---| + * | driver-memory | `supports = {}` | **yes** | **none, ever** | **nothing** — a silent duplicate | + * | driver-mongodb | absent (`{ batchSchemaSync: true }`) | **yes** | single-field unique index when the field declares `unique` | `E11000 duplicate key` → re-seed + re-issue, here | + * | driver-sql | `autonumber: true` | no | — | — | + * | driver-sqlite-wasm | inherited (`extends SqlDriver`, no `supports` override) | no | — | — | + * | driver-turso | inherited (`...super.supports`) | no | — | — | + * + * So the retry protects essentially ONE backend: driver-mongodb with a + * `unique` autonumber field. That is not a new claim — it is the reading the + * repo already ruled and gates, in + * `scripts/driver-memory-census.ledger.json`'s disposition for + * `packages/runtime/src/autonumber-seed-cross-side-parity.integration.test.ts` + * (axis `ruled-permanent`, «#6664 A, maintainer 2026-08-08 — inherits #5704 + * Q2 = B»), which states it as: "InMemoryDriver declares `supports = {}`, so + * the ENGINE's autonumber seeding owns the counter. No SQL backend can stand + * in — SqlDriver advertises the capability and its own sequence bootstrap + * answers instead". This comment cites that ruling rather than restating it: + * a second answer to "who owns the autonumber counter" is the same + * one-contract-two-numbers defect this lane keeps closing (#6832). + * + * `InMemoryDriver.create` is a `table.push()` and it stores no constraints of + * any kind — its own docstring says so since #4065, and calls itself a WEAK + * oracle for exactly this reason. So on driver-memory an out-of-process + * duplicate cannot raise anything for this method to catch, and the number + * lands twice in the rendered field with no error anywhere. + * + * That is stated rather than papered over (PD #10: never advertise a + * capability the runtime does not deliver). What covers driver-memory is the + * OTHER half of this resync — {@link adoptExplicitAutonumber} — which needs no + * constraint at all because it never waits for a rejection. Between them: drift + * the engine can observe is fixed on every driver; drift only the store can + * report is fixed wherever the store reports it. + * + * ⛔ The remedy for the silent-duplicate row is uniqueness enforcement in the + * driver, NOT a pre-issue existence probe here: a probe costs a query on every + * insert (the cost this resync was designed to avoid) and is still racy, so it + * would trade a silent duplicate for a rarer silent duplicate at double the + * read cost. `packages/drivers/**` is under the #5499 investment freeze, so + * that work is not this change's to do. + * + * # And when it does not converge + * + * After {@link AUTONUMBER_COLLISION_ATTEMPTS} the write fails with a named + * engine error (`code: 'ERR_AUTONUMBER_COLLISION'`) carrying the driver's + * error as `cause`. The raw driver error is deliberately NOT the contract + * here: "your record number collided three times after re-seeding" is an + * engine-level condition a caller can act on, and the driver's prose is + * preserved rather than replaced. + */ + private async createWithAutonumberResync( + driver: any, + object: string, + row: Record, + driverOptions: any, + issued: IssuedAutonumber[], + execCtx: ExecutionContext | undefined, + driverOwnsAutonumber: boolean, + ): Promise { + let attempt = 1; + for (;;) { + try { + return await driver.create(object, row, driverOptions); + } catch (error) { + if (!this.isIssuedAutonumberCollision(error, issued)) throw error; + // Whatever happens next, the stale counter must not survive this call: + // leaving it in place is what turned one collision into a storm. + for (const one of issued) this.autonumberCounters.delete(one.counterKey); + if (attempt >= AUTONUMBER_COLLISION_ATTEMPTS) { + const fields = issued.map((one) => one.field).join(', '); + throw Object.assign( + new Error( + `Autonumber collision on '${object}' field(s) [${fields}]: the record number was ` + + `re-seeded from the store and re-issued ${AUTONUMBER_COLLISION_ATTEMPTS} times and the ` + + `driver rejected each one as a duplicate. No record was written.`, + ), + { code: 'ERR_AUTONUMBER_COLLISION', cause: error }, + ); + } + attempt += 1; + this.logger.warn('Autonumber collided — re-seeding the counter and re-issuing', { + object, fields: issued.map((one) => one.field), attempt, + }); + // Clear the slots so `applyAutonumbers` treats them as empty again — + // leaving the burned value in place would read as an exempt writer's + // and be adopted rather than re-issued. + for (const one of issued) delete row[one.field]; + issued = await this.applyAutonumbers(object, row, execCtx, driverOwnsAutonumber); + // Nothing left to re-issue (the field vanished from the schema + // mid-flight) — the next failure is the caller's to see. + if (issued.length === 0) return await driver.create(object, row, driverOptions); + } + } + } + + /** + * Whether `error` is a unique violation attributable to one of the + * autonumbers this insert issued (#6806). See + * {@link createWithAutonumberResync} for why an unnamed column counts as + * attributable and a differently-named one does not. + */ + private isIssuedAutonumberCollision(error: unknown, issued: IssuedAutonumber[]): boolean { + if (issued.length === 0) return false; + if (!isUniqueViolationError(error)) return false; + const column = uniqueViolationColumn(error); + if (column === undefined) return true; + return issued.some((one) => one.field === column); } /** @@ -2417,6 +2755,10 @@ export class ObjectQL implements IObjectQLEngine { * bare trailing counter, and values predating any format have no anchor to * read from, so this stays exactly as it was. * + * That reading is {@link readAutonumberCounter}, module-level rather than + * inline, because #6806's resync must read an exempt writer's supplied value + * by exactly these rules. + * * The suffix is *stripped when it matches*, never *required* to match: a * dynamic suffix renders differently per row (`{000}-{YYYY}` is `-2025` on * last year's rows) while the counter scope is the rendered PREFIX — here `''` @@ -2487,29 +2829,17 @@ export class ObjectQL implements IObjectQLEngine { }, ); let max = 0; - // Anchored when the format declares text on EITHER side of the slot; see - // the "Locating the counter" section above. - const anchored = prefix !== '' || suffix !== ''; for await (const page of walk.pages()) { for (const r of page) { const v = r?.[field]; if (v == null) continue; - const s = String(v); - if (prefix && !s.startsWith(prefix)) continue; - // Both branches use the linear /\d+/ forms — a backtracking lookahead - // here is a polynomial-ReDoS sink on stored values full of zeros - // (CodeQL js/polynomial-redos). - let digits: string | undefined; - if (anchored) { - let core = s.slice(prefix.length); - if (suffix && core.endsWith(suffix)) core = core.slice(0, core.length - suffix.length); - const head = core.match(/^\d+/); - digits = head ? head[0] : undefined; - } else { - const runs = s.match(/\d+/g); - digits = runs ? runs[runs.length - 1] : undefined; - } - if (digits) max = Math.max(max, parseInt(digits, 10) || 0); + // The reading itself lives in `readAutonumberCounter` (the section + // above describes it) because #6806's adopt-on-exempt-write resync + // must read a supplied value by the SAME rules this scan reads a + // stored one — two copies of it would drift into two different + // answers for one row, which is a duplicate record number. + const counter = readAutonumberCounter(String(v), prefix, suffix); + if (counter != null) max = Math.max(max, counter); } } // The walk is unbounded (no `max`), so truncation here means the scan @@ -6156,10 +6486,15 @@ export class ObjectQL implements IObjectQLEngine { // autonumber assigns nothing here — so no validation rule can depend on // the value, making this reorder safe. In partial mode dead rows are // skipped, so they never consume a sequence value either. + // [#6806] What each row's autonumbers were ISSUED from, so a + // unique-violation on the write below can be attributed to a counter + // and answered by re-seeding it. Empty for every row when the driver + // owns autonumber, and for any row the engine numbered nothing on. + const issuedPerRow: IssuedAutonumber[][] = new Array(rows.length).fill(null).map(() => []); for (let i = 0; i < rows.length; i++) { if (rowErrors[i] !== undefined) continue; try { - await this.applyAutonumbers(object, rows[i], opCtx.context, driverOwnsAutonumber); + issuedPerRow[i] = await this.applyAutonumbers(object, rows[i], opCtx.context, driverOwnsAutonumber); } catch (e) { if (!partialMode) throw e; rowErrors[i] = e; @@ -6173,14 +6508,50 @@ export class ObjectQL implements IObjectQLEngine { if (isBatch) { if (liveRows.length === 0) { result = []; - } else if (driver.bulkCreate) { - result = await driver.bulkCreate(object, liveRows, driverOptions); } else { - // Fallback loop - result = await Promise.all(liveRows.map((item) => driver.create(object, item, driverOptions))); + // [#6806] A batch is re-seeded but never re-issued. `bulkCreate` may + // be partially applied by a driver without a transaction, so + // re-writing the batch could DUPLICATE the rows that did land — + // strictly worse than the collision. What must not survive is the + // stale counter: leaving it is what makes the very next insert + // collide too, one number at a time, which is the storm. So the + // counters this batch drew on are dropped and the driver's error is + // rethrown UNCHANGED (a batch caller — bulkWrite's per-row + // degradation — reads these errors, and this is not the place to + // change what it reads). + // + // What an author gets, stated plainly: `insert(object, rows[])` and + // `insertMany` both REJECT with the driver's own duplicate-key + // error — never `ERR_AUTONUMBER_COLLISION`, which is the + // single-row path's identity for "re-issued and still refused". + // Whether any row was written is the driver's answer, not this + // method's. The one thing the engine guarantees is that the NEXT + // write re-seeds instead of walking into the same collision, so a + // retry by the caller converges. Pinned in + // engine-autonumber-resync.test.ts. + try { + if (driver.bulkCreate) { + result = await driver.bulkCreate(object, liveRows, driverOptions); + } else { + // Fallback loop + result = await Promise.all(liveRows.map((item) => driver.create(object, item, driverOptions))); + } + } catch (error) { + const batchIssued = liveIndexes.flatMap((i) => issuedPerRow[i]); + if (this.isIssuedAutonumberCollision(error, batchIssued)) { + for (const one of batchIssued) this.autonumberCounters.delete(one.counterKey); + this.logger.warn('Autonumber collided in a batch insert — counter dropped, batch not re-issued', { + object, fields: [...new Set(batchIssued.map((one) => one.field))], + }); + } + throw error; + } } } else { - result = await driver.create(object, liveRows[0], driverOptions); + result = await this.createWithAutonumberResync( + driver, object, liveRows[0], driverOptions, issuedPerRow[liveIndexes[0]], + opCtx.context, driverOwnsAutonumber, + ); } // Driver-result contract guard (framework#3151): a batch write must