diff --git a/.changeset/import-historical-fsm.md b/.changeset/import-historical-fsm.md new file mode 100644 index 000000000..cdf656f6d --- /dev/null +++ b/.changeset/import-historical-fsm.md @@ -0,0 +1,32 @@ +--- +"@objectstack/spec": patch +"@objectstack/objectql": patch +"@objectstack/rest": patch +"@objectstack/platform-objects": patch +--- + +feat(rest): `treatAsHistorical` import option — skip the state machine for historical-data migration (#3479) + +Sibling of #3433 (seed exemption), one entry point over. #3165's `initialStates` enforced +the FSM entry point on every INSERT, so importing established historical facts — +a batch of already-`closed` tickets, `closed_won` deals, `completed` projects — +was rejected row-by-row with `invalid_initial_state`, blocking the core +data-migration path. Unlike the seed case it was visible (per-row errors), but it +still functionally blocked a legitimate use. + +- **spec**: `ExecutionContext.skipStateMachine` — a general, server-set flag (the + seed-specific `seedReplay`'s sibling) that skips the `state_machine` rule for a + write; `ImportRequestSchema.treatAsHistorical` (default `false`) — the user-facing + import option. +- **objectql**: the engine now skips the state machine for `seedReplay` OR + `skipStateMachine` (one helper), covering both seed replay and historical import. +- **rest**: the import runner sets `skipStateMachine` on the write context iff the + request opts into `treatAsHistorical`; default off, so a normal import still walks + the FSM (the strict behavior is the default). Import **undo** now also carries + `skipStateMachine`, since restoring a prior snapshot re-writes an earlier state + that need not be a legal transition from where the row is now. +- **platform-objects**: `sys_import_job.treat_as_historical` audit column (additive). + +Scope is identical to the seed exemption: ONLY the `state_machine` rule is skipped; +field shape, `format`, `cross_field`, `script` all still run. The objectui import +wizard checkbox is a separate follow-up. diff --git a/content/docs/protocol/objectql/state-machine.mdx b/content/docs/protocol/objectql/state-machine.mdx index dc1b46d7d..83321ae06 100644 --- a/content/docs/protocol/objectql/state-machine.mdx +++ b/content/docs/protocol/objectql/state-machine.mdx @@ -100,6 +100,7 @@ transitions: { - The check is **lenient where it cannot reason**: if the prior state is not described by the table (e.g. legacy or externally-written data), it does not block. - Only a rule with `severity: 'error'` (the default) blocks the write; `warning`/`info` are logged. - **Seed writes are exempt** (#3433). Curated seed data — package bootstrap fixtures, marketplace templates, per-org replay, all loaded by `SeedLoaderService` — is a snapshot of established facts, not a record walking its lifecycle, so it bypasses the `state_machine` rule entirely: a seed may be born mid-lifecycle (a `completed` project, a `closed_won` opportunity) and neither `initialStates` (insert) nor `transitions` (update) is enforced. Every *other* validation still runs, so a seed must still satisfy field shape, `format`, `script`, and the rest. `os lint` warns when a seeded value is not a state the machine declares, so a typo is still caught before boot. +- **A "historical" data import is exempt too** (#3479). Migrating established facts — a batch of already-`closed` tickets, `closed_won` deals — is the same "snapshot, not a lifecycle event" situation. Set `treatAsHistorical: true` on the import request (default **off**) and the runner puts `skipStateMachine` on the write context, so `initialStates` doesn't reject those mid-lifecycle rows. A normal import leaves it off and still walks the FSM — the strict behavior is the default, so the exemption is always an explicit opt-in. ### Conditional transitions diff --git a/content/docs/references/api/export.mdx b/content/docs/references/api/export.mdx index ea7e1f7c9..8891b52df 100644 --- a/content/docs/references/api/export.mdx +++ b/content/docs/references/api/export.mdx @@ -82,6 +82,7 @@ const result = CreateExportJobRequest.parse(data); | **writeMode** | `Enum<'insert' \| 'update' \| 'upsert'>` | ✅ | insert / update / upsert semantics | | **matchFields** | `string[]` | optional | Fields that identify an existing record (required for update/upsert) | | **runAutomations** | `boolean` | ✅ | Fire triggers/hooks for each imported row (off by default for bulk) | +| **treatAsHistorical** | `boolean` | ✅ | Import as established historical facts: skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates (#3479). Off by default so a normal import still walks the FSM. | | **trimWhitespace** | `boolean` | ✅ | Trim leading/trailing whitespace from string cells | | **nullValues** | `string[]` | optional | Strings treated as null/blank (besides empty string) | | **createMissingOptions** | `boolean` | ✅ | Keep unmatched select values instead of failing the row | @@ -370,6 +371,7 @@ Type: `{ sourceField: string; targetField: string; targetLabel?: string; transfo | **writeMode** | `Enum<'insert' \| 'update' \| 'upsert'>` | ✅ | insert / update / upsert semantics | | **matchFields** | `string[]` | optional | Fields that identify an existing record (required for update/upsert) | | **runAutomations** | `boolean` | ✅ | Fire triggers/hooks for each imported row (off by default for bulk) | +| **treatAsHistorical** | `boolean` | ✅ | Import as established historical facts: skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates (#3479). Off by default so a normal import still walks the FSM. | | **trimWhitespace** | `boolean` | ✅ | Trim leading/trailing whitespace from string cells | | **nullValues** | `string[]` | optional | Strings treated as null/blank (besides empty string) | | **createMissingOptions** | `boolean` | ✅ | Keep unmatched select values instead of failing the row | diff --git a/content/docs/references/kernel/execution-context.mdx b/content/docs/references/kernel/execution-context.mdx index 8f1955cc6..c81b916a2 100644 --- a/content/docs/references/kernel/execution-context.mdx +++ b/content/docs/references/kernel/execution-context.mdx @@ -68,6 +68,7 @@ const result = ExecutionContext.parse(data); | **skipTriggers** | `boolean` | optional | | | **skipAutomations** | `boolean` | optional | | | **seedReplay** | `boolean` | optional | | +| **skipStateMachine** | `boolean` | optional | | | **oauthScopes** | `string[]` | optional | | | **accessToken** | `string` | optional | | | **transaction** | `any` | optional | | diff --git a/packages/objectql/src/engine.test.ts b/packages/objectql/src/engine.test.ts index d4610bbb7..cf04a7c8a 100644 --- a/packages/objectql/src/engine.test.ts +++ b/packages/objectql/src/engine.test.ts @@ -907,6 +907,14 @@ describe('ObjectQL Engine', () => { expect(mockDriver.create).toHaveBeenCalledTimes(1); }); + it('admits the same INSERT when the context carries skipStateMachine (#3479 historical import)', async () => { + // The general flag — set by the REST import runner for a "historical" + // import — takes the same engine path as seedReplay. + vi.mocked(SchemaRegistry.getObject).mockReturnValue(approvalObject as any); + await engine.insert('seed_approval', { status: 'approved' }, { context: { skipStateMachine: true } as any }); + expect(mockDriver.create).toHaveBeenCalledTimes(1); + }); + it('still enforces non-state_machine rules under seedReplay (scoped exemption)', async () => { vi.mocked(SchemaRegistry.getObject).mockReturnValue({ ...approvalObject, diff --git a/packages/objectql/src/engine.ts b/packages/objectql/src/engine.ts index 88d6b35b1..b3d3a11ba 100644 --- a/packages/objectql/src/engine.ts +++ b/packages/objectql/src/engine.ts @@ -219,6 +219,18 @@ function mergeReadContext( return { ...fromQuery, ...fromOptions }; } +/** + * True when this write is exempt from the `state_machine` validation rule — + * both the insert `initialStates` entry check and the update `transitions` + * check are skipped for it. Either the seed-specific `seedReplay` flag (#3433) + * or the general `skipStateMachine` flag (#3479, set by the REST import runner + * for a "historical" import) turns it off. Both are server-set, never + * client-supplied. + */ +function shouldSkipStateMachine(ctx?: ExecutionContext): boolean { + return ctx?.seedReplay === true || ctx?.skipStateMachine === true; +} + /** * Engine Middleware (Onion model) */ @@ -2569,7 +2581,7 @@ export class ObjectQL implements IDataEngine { try { normalizeMultiValueFields(schemaForValidation, rows[i]); validateRecord(schemaForValidation, rows[i], 'insert'); - evaluateValidationRules(schemaForValidation as any, rows[i], 'insert', { logger: this.logger, currentUser: this.buildEvalUser(opCtx.context), skipStateMachine: opCtx.context?.seedReplay === true }); + evaluateValidationRules(schemaForValidation as any, rows[i], 'insert', { logger: this.logger, currentUser: this.buildEvalUser(opCtx.context), skipStateMachine: shouldSkipStateMachine(opCtx.context) }); } catch (e) { if (!partialMode) throw e; rowErrors[i] = e; @@ -2859,7 +2871,7 @@ export class ObjectQL implements IDataEngine { hookContext.input.data = stripReadonlyFields(updateSchema as any, preRo, suppliedKeys, this.logger) as any; reportDroppedFields(preRo, hookContext.input.data as Record, 'readonly'); } - evaluateValidationRules(updateSchema as any, hookContext.input.data as Record, 'update', { previous: priorRecord, logger: this.logger, currentUser: this.buildEvalUser(opCtx.context), skipStateMachine: opCtx.context?.seedReplay === true }); + evaluateValidationRules(updateSchema as any, hookContext.input.data as Record, 'update', { previous: priorRecord, logger: this.logger, currentUser: this.buildEvalUser(opCtx.context), skipStateMachine: shouldSkipStateMachine(opCtx.context) }); result = await driver.update(object, hookContext.input.id as string, hookContext.input.data as Record, hookContext.input.options as any); } else if (options?.multi && driver.updateMany) { await this.encryptSecretFields(object, hookContext.input.data as Record, opCtx.context, hookContext.input.options); @@ -2927,7 +2939,7 @@ export class ObjectQL implements IDataEngine { if (rulesNeedRows) { for (const row of priorRows ?? []) { try { - evaluateValidationRules(updateSchema as any, hookContext.input.data as Record, 'update', { previous: row, logger: this.logger, currentUser: bulkEvalUser, skipStateMachine: opCtx.context?.seedReplay === true }); + evaluateValidationRules(updateSchema as any, hookContext.input.data as Record, 'update', { previous: row, logger: this.logger, currentUser: bulkEvalUser, skipStateMachine: shouldSkipStateMachine(opCtx.context) }); } catch (err) { if (err instanceof ValidationError && row?.id != null) { throw new ValidationError(err.fields.map((f) => ({ ...f, message: `${f.message} (record ${String(row.id)})` }))); @@ -2936,7 +2948,7 @@ export class ObjectQL implements IDataEngine { } } } else { - evaluateValidationRules(updateSchema as any, hookContext.input.data as Record, 'update', { previous: null, logger: this.logger, currentUser: bulkEvalUser, skipStateMachine: opCtx.context?.seedReplay === true }); + evaluateValidationRules(updateSchema as any, hookContext.input.data as Record, 'update', { previous: null, logger: this.logger, currentUser: bulkEvalUser, skipStateMachine: shouldSkipStateMachine(opCtx.context) }); } result = await driver.updateMany(object, ast, hookContext.input.data as Record, hookContext.input.options as any); } else { diff --git a/packages/platform-objects/src/audit/sys-import-job.object.ts b/packages/platform-objects/src/audit/sys-import-job.object.ts index 69b4e445e..7ae82cde8 100644 --- a/packages/platform-objects/src/audit/sys-import-job.object.ts +++ b/packages/platform-objects/src/audit/sys-import-job.object.ts @@ -64,6 +64,7 @@ export const SysImportJob = ObjectSchema.create({ ), dry_run: Field.boolean({ label: 'Dry Run', required: false, defaultValue: false, group: 'Request' }), run_automations: Field.boolean({ label: 'Run Automations', required: false, defaultValue: false, group: 'Request' }), + treat_as_historical: Field.boolean({ label: 'Treat As Historical', required: false, defaultValue: false, group: 'Request' }), // ── outcome ── error: Field.textarea({ label: 'Fatal Error', required: false, group: 'Outcome' }), diff --git a/packages/rest/src/import-prepare.ts b/packages/rest/src/import-prepare.ts index 843848e4b..c3434164d 100644 --- a/packages/rest/src/import-prepare.ts +++ b/packages/rest/src/import-prepare.ts @@ -173,6 +173,9 @@ export interface PreparedImport { matchFields: string[]; dryRun: boolean; runAutomations: boolean; + /** #3479 — import established historical facts: skip the state_machine rule + * so mid-lifecycle rows are not rejected by `initialStates`. */ + treatAsHistorical: boolean; trimWhitespace: boolean; nullValues?: string[]; createMissingOptions: boolean; @@ -256,6 +259,11 @@ export async function prepareImportRequest( // flag until #2922), so opt-out must be explicit — matches platform // convention (Salesforce runs triggers on import by default). const runAutomations = body?.runAutomations !== false; + // Default OFF (opt-in): a normal import must still walk the state machine — + // only an explicit "historical" import skips it (#3479), so mid-lifecycle + // rows aren't rejected by `initialStates`. Unlike `runAutomations`, the safe + // default is the strict one. + const treatAsHistorical = body?.treatAsHistorical === true; const trimWhitespace = body?.trimWhitespace !== false; const nullValues: string[] | undefined = Array.isArray(body?.nullValues) ? body.nullValues.filter((v: any) => typeof v === 'string') @@ -392,6 +400,7 @@ export async function prepareImportRequest( ok: true, prepared: { rows, metaMap, writeMode, matchFields, dryRun, runAutomations, + treatAsHistorical, trimWhitespace, nullValues, createMissingOptions, skipBlankMatchKey, }, }; diff --git a/packages/rest/src/import-runner-historical.test.ts b/packages/rest/src/import-runner-historical.test.ts new file mode 100644 index 000000000..fa5673b72 --- /dev/null +++ b/packages/rest/src/import-runner-historical.test.ts @@ -0,0 +1,89 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * #3479 — a "historical" import carries curated established facts (a batch of + * already-`closed` tickets, `closed_won` deals), so it must skip the object's + * `state_machine` rule: otherwise `initialStates` rejects every mid-lifecycle + * row on insert. runImport owns the option→context mapping — it sets + * `skipStateMachine` on the write context iff `treatAsHistorical` is set. A + * normal import must NOT set it (the FSM still applies). + * + * The engine's actual exemption behavior (skipStateMachine → skip the rule) is + * tested in @objectstack/objectql (engine.test.ts); this pins the wiring the + * import runner owns. + */ + +import { describe, it, expect, vi } from 'vitest'; +import { runImport, type ImportProtocolLike } from './import-runner'; +import type { ExportFieldMeta } from './export-format.js'; + +const metaMap = new Map([['name', { name: 'name', type: 'text' }]]); + +const baseOpts = { + objectName: 'ticket', + metaMap, + writeMode: 'insert' as const, + matchFields: [] as string[], + dryRun: false, + runAutomations: false, + trimWhitespace: true, + createMissingOptions: false, + skipBlankMatchKey: false, +}; + +/** Provider mock that records the write context every persist call received. */ +function makeProvider() { + const contexts: any[] = []; + let idc = 0; + const p: ImportProtocolLike = { + findData: vi.fn(async () => []), + createData: vi.fn(async (args: any) => { + contexts.push(args.context); + return { id: `d${++idc}`, ...args.data }; + }), + updateData: vi.fn(async (args: any) => { + contexts.push(args.context); + return { id: args.id, ...args.data }; + }), + createManyData: vi.fn(async (args: any) => { + contexts.push(args.context); + return { records: args.records.map((r: any) => ({ id: `d${++idc}`, ...r })) }; + }), + }; + return { p, contexts }; +} + +describe('runImport — historical import skips the state machine (#3479)', () => { + it('sets skipStateMachine on the write context when treatAsHistorical is true', async () => { + const { p, contexts } = makeProvider(); + const summary = await runImport({ + ...baseOpts, p, rows: [{ name: 'a' }, { name: 'b' }], treatAsHistorical: true, + }); + expect(summary.created).toBe(2); + expect(contexts.length).toBeGreaterThan(0); + for (const ctx of contexts) expect(ctx?.skipStateMachine).toBe(true); + }); + + it('does NOT set skipStateMachine for a normal import (the FSM still applies)', async () => { + const { p, contexts } = makeProvider(); + await runImport({ ...baseOpts, p, rows: [{ name: 'a' }], treatAsHistorical: false }); + expect(contexts.length).toBeGreaterThan(0); + for (const ctx of contexts) expect(ctx?.skipStateMachine).toBeUndefined(); + }); + + it('defaults to off when the option is omitted (safe default — walk the FSM)', async () => { + const { p, contexts } = makeProvider(); + await runImport({ ...baseOpts, p, rows: [{ name: 'a' }] }); + expect(contexts.length).toBeGreaterThan(0); + for (const ctx of contexts) expect(ctx?.skipStateMachine).toBeUndefined(); + }); + + it('leaves the automation toggle independent (skipAutomations still tracks runAutomations)', async () => { + const { p, contexts } = makeProvider(); + await runImport({ ...baseOpts, p, rows: [{ name: 'a' }], treatAsHistorical: true, runAutomations: true }); + for (const ctx of contexts) { + expect(ctx?.skipStateMachine).toBe(true); + expect(ctx?.skipAutomations).toBe(false); // runAutomations:true ⇒ don't skip + } + }); +}); diff --git a/packages/rest/src/import-runner.ts b/packages/rest/src/import-runner.ts index be889a0f0..3239cf433 100644 --- a/packages/rest/src/import-runner.ts +++ b/packages/rest/src/import-runner.ts @@ -102,6 +102,11 @@ export interface RunImportOptions { matchFields: string[]; dryRun: boolean; runAutomations: boolean; + /** #3479 — treat rows as established historical facts: the write context + * carries `skipStateMachine`, so mid-lifecycle values aren't rejected by the + * object's `state_machine` `initialStates` (insert) / `transitions` (update). + * Optional here (runner default is off); `prepareImportRequest` always sets it. */ + treatAsHistorical?: boolean; trimWhitespace: boolean; nullValues?: string[]; createMissingOptions: boolean; @@ -163,7 +168,7 @@ const yieldToEventLoop = (): Promise => export function runImport(opts: RunImportOptions): Promise { const { p, objectName, environmentId, context, rows, metaMap, - writeMode, matchFields, dryRun, runAutomations, + writeMode, matchFields, dryRun, runAutomations, treatAsHistorical, trimWhitespace, nullValues, createMissingOptions, skipBlankMatchKey, onProgress, shouldCancel, captureUndo, } = opts; @@ -262,7 +267,14 @@ export function runImport(opts: RunImportOptions): Promise { return recs[0]; }; - const writeCtx = { ...(context ?? {}), skipAutomations: !runAutomations }; + const writeCtx = { + ...(context ?? {}), + skipAutomations: !runAutomations, + // #3479 — a "historical" import carries curated established facts, so the + // engine skips the state_machine rule for these writes (initialStates on + // insert, transitions on update). Default off: a normal import walks the FSM. + ...(treatAsHistorical ? { skipStateMachine: true } : {}), + }; // Sparse-indexed by row position `i` (not push-only): CREATE rows are // resolved immediately but their write is deferred to a later batch flush, diff --git a/packages/rest/src/rest-server.ts b/packages/rest/src/rest-server.ts index b11ce914a..3bce55c3d 100644 --- a/packages/rest/src/rest-server.ts +++ b/packages/rest/src/rest-server.ts @@ -3831,6 +3831,7 @@ export class RestServer { write_mode: prepared.writeMode, dry_run: prepared.dryRun, run_automations: prepared.runAutomations, + treat_as_historical: prepared.treatAsHistorical, created_at: createdAt, ...(createdBy ? { created_by: createdBy } : {}), }; @@ -4003,7 +4004,11 @@ export class RestServer { const objectName = String(row.object_name ?? ''); const log = parseUndoLog(row.undo_log)!; // Undo automations too: reversing writes shouldn't re-fire triggers. - const writeCtx = { ...(context ?? {}), skipAutomations: true }; + // Skip the state machine as well (#3479): restoring a prior snapshot + // re-writes the row's earlier state, which need not be a legal + // transition from where it is now — an undo reinstates an established + // fact, it does not walk the FSM. + const writeCtx = { ...(context ?? {}), skipAutomations: true, skipStateMachine: true }; let deleted = 0, restored = 0, failed = 0; // Delete created records first (they didn't exist before). diff --git a/packages/spec/src/api/export.zod.ts b/packages/spec/src/api/export.zod.ts index 36b338600..c913f24a0 100644 --- a/packages/spec/src/api/export.zod.ts +++ b/packages/spec/src/api/export.zod.ts @@ -319,6 +319,8 @@ export const ImportRequestSchema = lazySchema(() => z.object({ .describe('Fields that identify an existing record (required for update/upsert)'), runAutomations: z.boolean().default(false) .describe('Fire triggers/hooks for each imported row (off by default for bulk)'), + treatAsHistorical: z.boolean().default(false) + .describe('Import as established historical facts: skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates (#3479). Off by default so a normal import still walks the FSM.'), trimWhitespace: z.boolean().default(true) .describe('Trim leading/trailing whitespace from string cells'), nullValues: z.array(z.string()).optional() diff --git a/packages/spec/src/kernel/execution-context.zod.ts b/packages/spec/src/kernel/execution-context.zod.ts index 0888fcddd..43b4b491b 100644 --- a/packages/spec/src/kernel/execution-context.zod.ts +++ b/packages/spec/src/kernel/execution-context.zod.ts @@ -218,6 +218,25 @@ export const ExecutionContextSchema = lazySchema(() => z.object({ */ seedReplay: z.boolean().optional(), + /** + * Skip the object's `state_machine` validation rule for this write — both the + * `initialStates` entry check on insert and the `transitions` check on update + * (#3479). Same engine behavior as {@link seedReplay}, but a GENERAL, + * server-set flag for any CURATED write of established facts that is not seed + * replay: notably a data import flagged "historical" (migrating a batch of + * already-`closed` tickets or `closed_won` deals), where the FSM entry point + * would otherwise reject every mid-lifecycle row. + * + * `seedReplay` is the seed-specific specialization (it also implies + * {@link skipTriggers} semantics at its callers); this is the plain + * skip-the-state-machine intent. The engine treats either flag identically. + * Server-constructed only, never client-supplied — the REST import runner sets + * it from the request's explicit `treatAsHistorical` option, gated so a normal + * import still walks the FSM. SCOPE is identical to `seedReplay`: only the + * `state_machine` rule is skipped; every other validation still runs. + */ + skipStateMachine: z.boolean().optional(), + /** * OAuth 2.1 scopes granted to the access token that authenticated this * request, when the principal was resolved from an OAuth bearer token