diff --git a/CHANGELOG.md b/CHANGELOG.md index ad049dd..98a0117 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,9 +2,24 @@ ## Unreleased +Target stable specification release: `v0.3.0` (not published). + +Until a protected or signed `v0.3.0` tag targets this work, implementations MUST identify it with `specification_status: release_candidate`, `specification_release: null`, and the exact immutable specification commit. The entries below do not constitute a stable release. + - Prepared `openstatspec-specification` 0.2.1 with the reusable, fail-closed Dolt adapter declaration validator and packaged schema resources. This package update adds no concrete Dolt support declaration. +- Added backward-compatible Transformation Plan and SPSS Syntax Frontend 0.2 + contracts for sequential numeric `COMPUTE` and conditional `IF`, bounded + comparison/`AND`/`OR` expressions, `FORMATS`, `VARIABLE LEVEL`, and + `EXECUTE`. +- Defined exact SQL three-valued missing semantics and deterministic complete + plan JSON hashing while preserving every 0.1 contract, fixture, and hash. +- Kept create-target apply fail-closed on MySQL, MariaDB, and Dolt; these + profiles require a separately versioned physical-and-catalog provisioning + action followed by an existing-target apply with no automatic Dolt commit. +- Added independent synthetic golden fixtures, hashes, examples, and repository + validation gates for the new contracts. ## v0.2.0 - 2026-07-31 diff --git a/README.md b/README.md index c6202a1..1db09e7 100644 --- a/README.md +++ b/README.md @@ -37,15 +37,18 @@ permits a transformation to mutate a core import. That separate derived-data profile is not used by the SPSS-like in-place frontend below. -The [Transformation Plan Profile 0.1](docs/transformation-plan-profile-0.1.md) -adds a canonical, language-neutral operation plan. The -[SPSS Syntax Frontend Profile 0.1](docs/spss-syntax-frontend-profile-0.1.md) -lowers a deliberately small `RECODE`, `VARIABLE LABELS`, and `VALUE LABELS` -subset into that plan. The -[in-place binding](docs/transformation-plan-sql-binding-0.1.md) applies the plan +The release-candidate [Transformation Plan Profile 0.2](docs/transformation-plan-profile-0.2.md) +adds sequential bounded numeric assignment and conditional assignment to the +unchanged 0.1 operations. The +[SPSS Syntax Frontend Profile 0.2](docs/spss-syntax-frontend-profile-0.2.md) +lowers `COMPUTE`, `IF`, `FORMATS`, `VARIABLE LEVEL`, and `EXECUTE` +alongside the 0.1 `RECODE`, `VARIABLE LABELS`, and `VALUE LABELS` subset. +Programs using only the 0.1 subset retain exact 0.1 plan identity and hash. The +[in-place binding](docs/transformation-plan-sql-binding-0.2.md) applies the plan to the same dataset and same physical wide table on supported SQL profiles. It creates no derived dataset, data copy, or OpenStatSpec undo layer; Dolt-specific -history and commits remain Dolt's. +history and commits remain Dolt's. MySQL, MariaDB, and Dolt require a new target +to be provisioned separately before an in-place transformation apply. ## Repository layout @@ -62,17 +65,19 @@ history and commits remain Dolt's. - `transformation/plan-0.1.schema.json` — canonical transformation-plan schema. - `sql/transformation-plan-profile-schema.sql` — compact in-place apply audit; it is not a dataset-version catalog. -- `conformance/transformation-plan-0.1.json` and - `conformance/spss-syntax-frontend-0.1.json` — plan and frontend conformance - cases with canonical hashes; `conformance/in-place-transformation-0.1.json` +- `conformance/transformation-plan-0.2.json` and + `conformance/spss-syntax-frontend-0.2.json` — additive plan and frontend + conformance cases with independent golden hashes; `conformance/in-place-transformation-0.2.json` fixes the same-dataset/same-table execution invariants and the additional - controlled Dolt context. + controlled Dolt context. The 0.1 schemas and fixtures remain unchanged. - `examples/` — small illustrative mapping fixtures. ## Conformance principle Implementations must preflight target capabilities before import. If the target cannot faithfully create one wide table because of column, identifier, string, or row limits, import must fail atomically with a machine-readable capability diagnostic. It must never silently truncate, drop, split, transpose, pivot, or transform source data. +This specification repository publishes only normative Transformation Plan and SPSS syntax frontend schemas, documentation, and declarative fixtures. Executable validation or behavioral conformance code, including parser, SQL-engine, transaction, and adapter behavior, belongs in each implementation or adapter repository. + ## Who it is for OpenStatSpec is for people and projects that need to move statistical datasets between SPSS and relational databases without turning the data into a different model. It is especially relevant to: diff --git a/VERSIONING.md b/VERSIONING.md index 5d946e4..938359b 100644 --- a/VERSIONING.md +++ b/VERSIONING.md @@ -11,6 +11,15 @@ core source profile. Adding or revising an optional profile does not authorize changes to core datasets. A breaking workflow-profile change increments that profile's major version even when the core version is unchanged. +Transformation Plan and frontend contracts are selected by their exact +contract identifiers. An implementation may support multiple versions +simultaneously. It MUST preserve the canonical bytes and hashes of accepted +older plans and MUST NOT silently reinterpret or rewrite an older plan under a +newer schema. A frontend that accepts a newer request contract but receives a +program entirely inside the 0.1 command subset emits the exact 0.1 plan +contract. Explicit recompilation to a newer plan is a new identified operation, +not migration of the old plan or audit row. + The SPSS SAV/ZSAV profile is versioned independently in its document and manifest. A conforming implementation must publish the exact profile version, immutable specification commit, specification status, supported directions, supported SQL profiles, and engine capability declaration it tested. The immutable commit is mandatory for both release candidates and stable releases. `specification_release` is additive provenance, not an alternative to the commit: diff --git a/conformance/in-place-transformation-0.2.json b/conformance/in-place-transformation-0.2.json new file mode 100644 index 0000000..370c5e4 --- /dev/null +++ b/conformance/in-place-transformation-0.2.json @@ -0,0 +1,481 @@ +{ + "manifest_version": "0.2", + "profile": "OpenStatSpec In-Place Transformation Binding 0.2", + "contract": "openstatspec-in-place-transformation-v0.2", + "audit_schema": "../sql/transformation-plan-profile-schema.sql", + "cases": [ + { + "id": "dolt-preprovisioned-target-sequential-null-semantics", + "database_profile": "dolt", + "actor": "conformance-runner", + "expected_branch": "feature/recode", + "expected_head": "provisioning-commit", + "working_set_clean": true, + "applied_plan_case": "sequential-conditional-binary-existing-target", + "applied_frontend_case": "compute-if-labels-format-level-execute-existing-target", + "before": { + "dataset_id": "11111111-1111-4111-8111-111111111111", + "physical_table_name": "data_synthetic", + "physical_table_schema": null, + "dataset_count": 1, + "persistent_data_table_count": 1, + "case_count": 4, + "target_provisioning": { + "physical_column": true, + "catalog_variable": true, + "dolt_commit": "provisioning-commit" + }, + "target_identity": {"variable_id": "33333333-3333-4333-8333-333333333333", "ordinal": 3}, + "target_metadata": { + "missing_values": [ + { + "ordinal": 1, + "rule_kind": "discrete", + "code_kind": "numeric", + "numeric_value": -9 + } + ] + }, + "dolt_branch": "feature/recode", + "dolt_head": "provisioning-commit", + "working_set_clean": true, + "rows": [ + { + "__case_ordinal": 1, + "source_a": 1, + "source_b": 1, + "target": null + }, + { + "__case_ordinal": 2, + "source_a": 1, + "source_b": null, + "target": null + }, + { + "__case_ordinal": 3, + "source_a": null, + "source_b": 1, + "target": null + }, + { + "__case_ordinal": 4, + "source_a": 0, + "source_b": 1, + "target": null + } + ] + }, + "after": { + "dataset_id": "11111111-1111-4111-8111-111111111111", + "physical_table_name": "data_synthetic", + "dataset_count": 1, + "persistent_data_table_count": 1, + "physical_table_schema": null, + "target_identity": {"variable_id": "33333333-3333-4333-8333-333333333333", "ordinal": 3}, + "dolt_branch": "feature/recode", + "dolt_head": "provisioning-commit", + "case_count": 4, + "dolt_commit_performed": false, + "working_set_clean": false, + "rows": [ + { + "__case_ordinal": 1, + "source_a": 1, + "source_b": 1, + "target": 1 + }, + { + "__case_ordinal": 2, + "source_a": 1, + "source_b": null, + "target": 0 + }, + { + "__case_ordinal": 3, + "source_a": null, + "source_b": 1, + "target": 0 + }, + { + "__case_ordinal": 4, + "source_a": 0, + "source_b": 1, + "target": 0 + } + ], + "target_metadata": { + "variable_label": "Synthetic conjunction", + "value_labels": [ + [ + 0, + "No" + ], + [ + 1, + "Yes" + ] + ], + "format": "F1.0", + "measurement_level": "nominal", + "missing_values": [ + { + "ordinal": 1, + "rule_kind": "discrete", + "code_kind": "numeric", + "numeric_value": -9 + } + ] + } + }, + "expected_audit": { + "contract_id": "openstatspec-in-place-transformation-v0.2", + "database_profile": "dolt", + "status": "succeeded", + "dataset_id": "11111111-1111-4111-8111-111111111111", + "physical_table_schema": null, + "physical_table_name": "data_synthetic", + "plan_hash": "8e9f72759f6ed53aa6957dc288bdc49894ca2e08a8313ca957da6b4dfd11acbd", "canonical_plan_json": "{\"contract\":\"openstatspec-transformation-plan-v0.2\",\"input_alias\":\"parent\",\"operations\":[{\"op\":\"assign\",\"target\":\"target\",\"target_mode\":\"replace\",\"value\":{\"kind\":\"literal\",\"value\":{\"bits\":\"0000000000000000\",\"type\":\"binary64\"}}},{\"condition\":{\"expression\":\"boolean\",\"operands\":[{\"expression\":\"comparison\",\"left\":{\"kind\":\"variable\",\"variable\":\"source_a\"},\"operator\":\"=\",\"right\":{\"kind\":\"literal\",\"value\":{\"bits\":\"3ff0000000000000\",\"type\":\"binary64\"}}},{\"expression\":\"comparison\",\"left\":{\"kind\":\"variable\",\"variable\":\"source_b\"},\"operator\":\"=\",\"right\":{\"kind\":\"literal\",\"value\":{\"bits\":\"3ff0000000000000\",\"type\":\"binary64\"}}}],\"operator\":\"and\"},\"op\":\"conditional_assign\",\"target\":\"target\",\"value\":{\"kind\":\"literal\",\"value\":{\"bits\":\"3ff0000000000000\",\"type\":\"binary64\"}}},{\"label\":\"Synthetic conjunction\",\"op\":\"set_variable_label\",\"variable\":\"target\"},{\"labels\":[{\"label\":\"No\",\"value\":{\"bits\":\"0000000000000000\",\"type\":\"binary64\"}},{\"label\":\"Yes\",\"value\":{\"bits\":\"3ff0000000000000\",\"type\":\"binary64\"}}],\"op\":\"replace_value_labels\",\"variable\":\"target\"},{\"decimals\":0,\"family\":\"F\",\"op\":\"set_format\",\"variable\":\"target\",\"width\":1},{\"level\":\"nominal\",\"op\":\"set_measurement_level\",\"variable\":\"target\"},{\"op\":\"execute\"}]}", + "source_hash": "02bede796f1f0c8dcc22eb616d4afa48540ea8f09f637179842368af8a576481", + "operation_count": 7, + "dolt_branch": "feature/recode", + "dolt_head_before": "provisioning-commit", + "dolt_head_after": "provisioning-commit", + "actor": "conformance-runner" + }, + "forbidden_artifacts": [ + "derived_dataset_row", + "persistent_output_table", + "full_table_copy", + "staging_dataset", + "staging_relation", + "snapshot_table", + "rollback_table", + "dataset_version_row", + "retirement_or_recovery_row", + "temporary_object_residue" + ], + "required_audit_fields": [ + "apply_id", + "contract_id", + "database_profile", + "dataset_id", + "physical_table_schema", + "physical_table_name", + "plan_hash", + "canonical_plan_json", + "source_hash", + "actor", + "status", + "dolt_branch", + "dolt_head_before", + "dolt_head_after", + "operation_count", + "started_at", + "completed_at" + ], + "expected_error": null + }, + { + "id": "dolt-preprovisioned-target-or-null-semantics", + "database_profile": "dolt", + "actor": "conformance-runner", + "expected_branch": "feature/recode", + "expected_head": "provisioning-commit", + "working_set_clean": true, + "applied_plan_case": "binding-or-null-existing-target", + "applied_frontend_case": "binding-or-null-existing-target", + "dataset_id": "11111111-1111-4111-8111-111111111111", + "physical_table_schema": null, + "physical_table_name": "data_synthetic", + "target_preprovisioned": true, + "condition": "source_a = 1 OR source_b = 1", + "before_rows": [ + { + "__case_ordinal": 1, + "source_a": null, + "source_b": 1, + "target": 0 + }, + { + "__case_ordinal": 2, + "source_a": null, + "source_b": 0, + "target": 0 + } + ], + "after_rows": [ + { + "__case_ordinal": 1, + "source_a": null, + "source_b": 1, + "target": 1 + }, + { + "__case_ordinal": 2, + "source_a": null, + "source_b": 0, + "target": 0 + } + ], + "expected_error": null, + "expected_audit": {"contract_id": "openstatspec-in-place-transformation-v0.2", "database_profile": "dolt", "dataset_id": "11111111-1111-4111-8111-111111111111", "physical_table_schema": null, "physical_table_name": "data_synthetic", "actor": "conformance-runner", "status": "succeeded", "operation_count": 1, "plan_hash": "7fa8a017ed2476606c20f7b6bc9df09b7bbff208891e03335dfa393c7c260742", "canonical_plan_json": "{\"contract\":\"openstatspec-transformation-plan-v0.2\",\"input_alias\":\"parent\",\"operations\":[{\"condition\":{\"expression\":\"boolean\",\"operands\":[{\"expression\":\"comparison\",\"left\":{\"kind\":\"variable\",\"variable\":\"source_a\"},\"operator\":\"=\",\"right\":{\"kind\":\"literal\",\"value\":{\"bits\":\"3ff0000000000000\",\"type\":\"binary64\"}}},{\"expression\":\"comparison\",\"left\":{\"kind\":\"variable\",\"variable\":\"source_b\"},\"operator\":\"=\",\"right\":{\"kind\":\"literal\",\"value\":{\"bits\":\"3ff0000000000000\",\"type\":\"binary64\"}}}],\"operator\":\"or\"},\"op\":\"conditional_assign\",\"target\":\"target\",\"value\":{\"kind\":\"literal\",\"value\":{\"bits\":\"3ff0000000000000\",\"type\":\"binary64\"}}}]}", "source_hash": "dc936264d68bdf22741a8476d650a4e58fc03a63ac47f86093cbbb732f31594a", "dolt_branch": "feature/recode", "dolt_head_before": "provisioning-commit", "dolt_head_after": "provisioning-commit"} + }, + { + "id": "dolt-create-target-fails-before-mutation", + "database_profile": "dolt", + "actor": "conformance-runner", + "expected_branch": "feature/recode", + "expected_head": "provisioning-commit", + "working_set_clean": true, + "target_mode": "create", + "expected_error": "schema_change_not_atomic", + "mutation_started": false + }, + { + "id": "mysql-create-target-fails-before-mutation", + "database_profile": "mysql", + "actor": "conformance-runner", + "target_mode": "create", + "expected_error": "schema_change_not_atomic", + "mutation_started": false + }, + { + "id": "mariadb-create-target-fails-before-mutation", + "database_profile": "mariadb", + "actor": "conformance-runner", + "target_mode": "create", + "expected_error": "schema_change_not_atomic", + "mutation_started": false + }, + { + "id": "dolt-preprovisioned-target-variable-missing-propagation", + "database_profile": "dolt", + "actor": "conformance-runner", + "expected_branch": "feature/recode", + "expected_head": "provisioning-commit", + "working_set_clean": true, + "applied_plan_case": "binding-variable-missing-existing-target", + "applied_frontend_case": "binding-variable-missing-existing-target", + "dataset_id": "11111111-1111-4111-8111-111111111111", + "physical_table_schema": null, + "physical_table_name": "data_synthetic", + "target_preprovisioned": true, + "assign": { + "target": "target", + "value_variable": "source_a" + }, + "before_rows": [ + { + "__case_ordinal": 1, + "source_a": null, + "target": 7 + }, + { + "__case_ordinal": 2, + "source_a": 2, + "target": 7 + } + ], + "after_rows": [ + { + "__case_ordinal": 1, + "source_a": null, + "target": null + }, + { + "__case_ordinal": 2, + "source_a": 2, + "target": 2 + } + ], + "expected_error": null, + "expected_audit": {"contract_id": "openstatspec-in-place-transformation-v0.2", "database_profile": "dolt", "dataset_id": "11111111-1111-4111-8111-111111111111", "physical_table_schema": null, "physical_table_name": "data_synthetic", "actor": "conformance-runner", "status": "succeeded", "operation_count": 1, "plan_hash": "2b52cbd2989c99aac69dbd10e0f3de225ca84f3902193e05931bc3bf9c3e8649", "canonical_plan_json": "{\"contract\":\"openstatspec-transformation-plan-v0.2\",\"input_alias\":\"parent\",\"operations\":[{\"op\":\"assign\",\"target\":\"target\",\"target_mode\":\"replace\",\"value\":{\"kind\":\"variable\",\"variable\":\"source_a\"}}]}", "source_hash": "febe1c066647e5e50aa6c115a7ef33ade99592c4929a2d8a7411a7dec0ff4c15", "dolt_branch": "feature/recode", "dolt_head_before": "provisioning-commit", "dolt_head_after": "provisioning-commit"} + }, + { + "id": "dolt-preprovisioned-target-conditional-variable-missing-propagation", + "database_profile": "dolt", + "actor": "conformance-runner", + "expected_branch": "feature/recode", + "expected_head": "provisioning-commit", + "working_set_clean": true, + "applied_plan_case": "binding-conditional-missing-existing-target", + "applied_frontend_case": "binding-conditional-missing-existing-target", + "dataset_id": "11111111-1111-4111-8111-111111111111", + "physical_table_schema": null, + "physical_table_name": "data_synthetic", + "target_preprovisioned": true, + "conditional_assign": { + "target": "target", + "value_variable": "source_a", + "predicate": "source_b = 1" + }, + "before_rows": [ + { + "__case_ordinal": 1, + "source_a": null, + "source_b": 1, + "target": 7 + }, + { + "__case_ordinal": 2, + "source_a": null, + "source_b": 0, + "target": 7 + } + ], + "after_rows": [ + { + "__case_ordinal": 1, + "source_a": null, + "source_b": 1, + "target": null + }, + { + "__case_ordinal": 2, + "source_a": null, + "source_b": 0, + "target": 7 + } + ], + "expected_error": null, + "expected_audit": {"contract_id": "openstatspec-in-place-transformation-v0.2", "database_profile": "dolt", "dataset_id": "11111111-1111-4111-8111-111111111111", "physical_table_schema": null, "physical_table_name": "data_synthetic", "actor": "conformance-runner", "status": "succeeded", "operation_count": 1, "plan_hash": "aa42695ecf8d84d487fe1cdc1b11dd9d9e5d048df670877ed562e9b9fbd23053", "canonical_plan_json": "{\"contract\":\"openstatspec-transformation-plan-v0.2\",\"input_alias\":\"parent\",\"operations\":[{\"condition\":{\"expression\":\"comparison\",\"left\":{\"kind\":\"variable\",\"variable\":\"source_b\"},\"operator\":\"=\",\"right\":{\"kind\":\"literal\",\"value\":{\"bits\":\"3ff0000000000000\",\"type\":\"binary64\"}}},\"op\":\"conditional_assign\",\"target\":\"target\",\"value\":{\"kind\":\"variable\",\"variable\":\"source_a\"}}]}", "source_hash": "9fb6357cd8b436b69c523d86be0ce1415dc0fded15a425f737e6f3e37bd08b76", "dolt_branch": "feature/recode", "dolt_head_before": "provisioning-commit", "dolt_head_after": "provisioning-commit"} + }, + { + "id": "sqlite-inequality-boundary-semantics", + "database_profile": "sqlite", + "actor": "conformance-runner", + "applied_plan_case": "binding-inequality-boundaries-existing-targets", + "applied_frontend_case": "binding-inequality-boundaries-existing-targets", + "dataset_id": "44444444-4444-4444-8444-444444444444", + "physical_table_schema": null, + "physical_table_name": "data_inequality", + "target_preprovisioned": true, + "boundary": 1, + "before_rows": [{"__case_ordinal": 1, "source": 0, "target_lt": 0, "target_le": 0, "target_gt": 0, "target_ge": 0}, {"__case_ordinal": 2, "source": 1, "target_lt": 0, "target_le": 0, "target_gt": 0, "target_ge": 0}, {"__case_ordinal": 3, "source": 2, "target_lt": 0, "target_le": 0, "target_gt": 0, "target_ge": 0}], + "operations": [{"op": "conditional_assign", "condition": "source < 1", "target": "target_lt", "value": 1}, {"op": "conditional_assign", "condition": "source <= 1", "target": "target_le", "value": 1}, {"op": "conditional_assign", "condition": "source > 1", "target": "target_gt", "value": 1}, {"op": "conditional_assign", "condition": "source >= 1", "target": "target_ge", "value": 1}], + "after_rows": [{"__case_ordinal": 1, "source": 0, "target_lt": 1, "target_le": 1, "target_gt": 0, "target_ge": 0}, {"__case_ordinal": 2, "source": 1, "target_lt": 0, "target_le": 1, "target_gt": 0, "target_ge": 1}, {"__case_ordinal": 3, "source": 2, "target_lt": 0, "target_le": 0, "target_gt": 1, "target_ge": 1}], + "expected_error": null, + "expected_audit": {"contract_id": "openstatspec-in-place-transformation-v0.2", "database_profile": "sqlite", "dataset_id": "44444444-4444-4444-8444-444444444444", "physical_table_schema": null, "physical_table_name": "data_inequality", "actor": "conformance-runner", "status": "succeeded", "operation_count": 4, "plan_hash": "cc0bde9b2adbbe76614c51a6122a5c9c17c9e610a63241eb1769956c2e543690", "canonical_plan_json": "{\"contract\":\"openstatspec-transformation-plan-v0.2\",\"input_alias\":\"parent\",\"operations\":[{\"condition\":{\"expression\":\"comparison\",\"left\":{\"kind\":\"variable\",\"variable\":\"source\"},\"operator\":\"<\",\"right\":{\"kind\":\"literal\",\"value\":{\"bits\":\"3ff0000000000000\",\"type\":\"binary64\"}}},\"op\":\"conditional_assign\",\"target\":\"target_lt\",\"value\":{\"kind\":\"literal\",\"value\":{\"bits\":\"3ff0000000000000\",\"type\":\"binary64\"}}},{\"condition\":{\"expression\":\"comparison\",\"left\":{\"kind\":\"variable\",\"variable\":\"source\"},\"operator\":\"<=\",\"right\":{\"kind\":\"literal\",\"value\":{\"bits\":\"3ff0000000000000\",\"type\":\"binary64\"}}},\"op\":\"conditional_assign\",\"target\":\"target_le\",\"value\":{\"kind\":\"literal\",\"value\":{\"bits\":\"3ff0000000000000\",\"type\":\"binary64\"}}},{\"condition\":{\"expression\":\"comparison\",\"left\":{\"kind\":\"variable\",\"variable\":\"source\"},\"operator\":\">\",\"right\":{\"kind\":\"literal\",\"value\":{\"bits\":\"3ff0000000000000\",\"type\":\"binary64\"}}},\"op\":\"conditional_assign\",\"target\":\"target_gt\",\"value\":{\"kind\":\"literal\",\"value\":{\"bits\":\"3ff0000000000000\",\"type\":\"binary64\"}}},{\"condition\":{\"expression\":\"comparison\",\"left\":{\"kind\":\"variable\",\"variable\":\"source\"},\"operator\":\">=\",\"right\":{\"kind\":\"literal\",\"value\":{\"bits\":\"3ff0000000000000\",\"type\":\"binary64\"}}},\"op\":\"conditional_assign\",\"target\":\"target_ge\",\"value\":{\"kind\":\"literal\",\"value\":{\"bits\":\"3ff0000000000000\",\"type\":\"binary64\"}}}]}", "source_hash": "1208507803e9b69cb523585e83dd60643e61d3082f9b1d76fe4bca33b1db0011"} + }, + { + "id": "dolt-context-changed-after-mutation-rolls-back", + "database_profile": "dolt", + "target_preprovisioned": true, + "actor": "conformance-runner", + "expected_context": {"branch": "feature/recode", "head": "provisioning-commit", "working_set_clean": true}, + "completion_context": {"branch": "feature/recode", "head": "concurrent-commit"}, + "failure_point": "after_data_and_metadata_before_audit", + "before": {"rows": [{"__case_ordinal": 1, "target": 0}], "target_metadata": {"variable_label": "Before"}, "audit_row_count": 0}, + "after_failure": {"rows": [{"__case_ordinal": 1, "target": 0}], "target_metadata": {"variable_label": "Before"}, "audit_row_count": 0}, + "mutation_started": true, + "expected_error": "dolt_context_changed" + }, + { + "id": "dolt-empty-actor-fails-before-mutation", + "database_profile": "dolt", + "actor": "", + "expected_error": "actor_required", + "mutation_started": false + }, + { + "id": "sqlite-create-target-atomic-success", + "database_profile": "sqlite", + "applied_plan_case": "sqlite-create-target-source-copy", + "applied_frontend_case": "sqlite-create-target-source-copy", + "target_mode": "create", + "operation": { + "op": "assign", + "target": "target", + "value_variable": "source_a" + }, + "before": { + "dataset_id": "22222222-2222-4222-8222-222222222222", + "physical_table_schema": null, + "physical_table_name": "data_atomic_create", + "dataset_count": 1, + "persistent_data_table_count": 1, + "variable_count": 2, + "case_count": 2, + "target": { + "physical_column_present": false, + "catalog_variable_present": false + }, + "rows": [ + { + "__case_ordinal": 1, + "source_a": 2, + "source_b": 11 + }, + { + "__case_ordinal": 2, + "source_a": null, + "source_b": 22 + } + ] + }, + "after": { + "dataset_id": "22222222-2222-4222-8222-222222222222", + "physical_table_schema": null, + "physical_table_name": "data_atomic_create", + "dataset_count": 1, + "persistent_data_table_count": 1, + "variable_count": 3, + "case_count": 2, + "target": { + "physical_column_present": true, + "catalog_variable_present": true, + "ordinal": 3, + "initial_metadata": { + "variable_label": null, + "value_labels": [], + "format": null, + "measurement_level": null, + "missing_values": [] + } + }, + "rows": [ + { + "__case_ordinal": 1, + "source_a": 2, + "source_b": 11, + "target": 2 + }, + { + "__case_ordinal": 2, + "source_a": null, + "source_b": 22, + "target": null + } + ] + }, + "transaction_boundary": [ + "physical_schema", + "row_values", + "catalog", + "compact_audit" + ], + "rollback_probe": { + "failure_point": "after_catalog_create_before_audit", + "physical_column_present": false, + "catalog_variable_present": false, + "row_values_changed": false, + "audit_row_present": false + }, + "expected_error": null, + "actor": "conformance-runner", + "expected_audit": { + "contract_id": "openstatspec-in-place-transformation-v0.2", + "database_profile": "sqlite", + "dataset_id": "22222222-2222-4222-8222-222222222222", + "physical_table_schema": null, + "physical_table_name": "data_atomic_create", + "actor": "conformance-runner", + "status": "succeeded", + "operation_count": 1, + "plan_hash": "25b815bb706bd8692667d251f819c731176c5728205c98b51aa2ae509f39a2c2", "canonical_plan_json": "{\"contract\":\"openstatspec-transformation-plan-v0.2\",\"input_alias\":\"parent\",\"operations\":[{\"op\":\"assign\",\"target\":\"target\",\"target_mode\":\"create\",\"value\":{\"kind\":\"variable\",\"variable\":\"source_a\"}}]}", + "source_hash": "febe1c066647e5e50aa6c115a7ef33ade99592c4929a2d8a7411a7dec0ff4c15" + } + } + ] +} diff --git a/conformance/spss-syntax-frontend-0.2.json b/conformance/spss-syntax-frontend-0.2.json new file mode 100644 index 0000000..553b9d0 --- /dev/null +++ b/conformance/spss-syntax-frontend-0.2.json @@ -0,0 +1,1106 @@ +{ + "manifest_version": "0.2", + "profile": "OpenStatSpec SPSS-like Syntax Frontend 0.2", + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "plan_contracts": [ + "openstatspec-transformation-plan-v0.1", + "openstatspec-transformation-plan-v0.2" + ], + "request_schema": "../transformation/spss-syntax-frontend-0.2.schema.json", + "plan_schemas": { + "openstatspec-transformation-plan-v0.1": "../transformation/plan-0.1.schema.json", + "openstatspec-transformation-plan-v0.2": "../transformation/plan-0.2.schema.json" + }, + "cases": [ + { + "id": "compute-if-labels-format-level-execute-existing-target", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "COMPUTE target = 0.\nIF (source_a = 1 AND source_b = 1) target = 1.\nVARIABLE LABELS target 'Synthetic conjunction'.\nVALUE LABELS target 0 'No' 1 'Yes'.\nFORMATS target (F1.0).\nVARIABLE LEVEL target (NOMINAL).\nEXECUTE." + }, + "expected_plan_case": "sequential-conditional-binary-existing-target", + "expected_plan_hash": "8e9f72759f6ed53aa6957dc288bdc49894ca2e08a8313ca957da6b4dfd11acbd", + "expected_source_hash": "02bede796f1f0c8dcc22eb616d4afa48540ea8f09f637179842368af8a576481", + "expected_error": null + }, + { + "id": "three-term-and-flattens-source-order", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "source_c", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "IF (source_a = 1 AND source_b = 1 AND source_c = 1) target = 1.\nEXECUTE." + }, + "expected_plan_case": "three-term-and-flattens-source-order", + "expected_plan_hash": "d6f94edd5de261552e79157cd29ff7668e824476827fae7f0fe9f56ba45704ea", + "expected_source_hash": "518cb0fa9537e7dff65f43c7a8de2d7e51c1e15b8c856c57e1875d2cfeb5646e", + "expected_error": null + }, + { + "id": "parenthesized-three-term-and-flattens-source-order", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "source_c", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "IF ((source_a = 1 AND source_b = 1) AND source_c = 1) target = 1.\nEXECUTE." + }, + "expected_plan_case": "three-term-and-flattens-source-order", + "expected_plan_hash": "d6f94edd5de261552e79157cd29ff7668e824476827fae7f0fe9f56ba45704ea", + "expected_source_hash": "fdad0805734da0d8b356d67ed561c3ddfa48f0c4a942a60b1d6ca98dab3321d6", + "expected_error": null + }, + { + "id": "old-subset-retains-v0.1-plan", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "q1", + "storage_kind": "numeric" + } + ] + }, + "source_text": "RECODE q1 (1 = 0)." + }, + "expected_plan_contract": "openstatspec-transformation-plan-v0.1", + "expected_plan_case_0_1": "in-place-recode-default-copy", + "expected_plan_hash": "091a2daeb490deb3369cba83b5197d1c43767432c9a04cbf3e55bd69296b67a8", + "expected_source_hash": "68c05b0cae01476642c70956ddcaa693311be1a653ead1d1caf8b006843ed55b", + "expected_error": null + }, + { + "id": "nested-or-inequality-variable-operands-create", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "source_c", + "storage_kind": "numeric" + } + ] + }, + "source_text": "COMPUTE target = source_a.\nIF (source_a >= 0 OR (source_b < 5 AND source_c <= 9)) target = source_b.\nEXECUTE." + }, + "expected_plan_case": "nested-or-inequalities-variable-operands-create", + "expected_plan_hash": "926f2cd93ed0bfa330bdb2d0b31b241f552c86c116dd35a315a20408bf997daf", + "expected_source_hash": "4b3b35660b1ec57de0b8b0485dcc85bb51a05796df9e9624f4f1790d78e2bfeb", + "expected_error": null + }, + { + "id": "sqlite-create-target-source-copy", + "request": {"contract": "openstatspec-spss-syntax-frontend-v0.2", "input_alias": "parent", "input_schema": {"variables": [{"name": "source_a", "storage_kind": "numeric"}, {"name": "source_b", "storage_kind": "numeric"}]}, "source_text": "COMPUTE target = source_a."}, + "expected_plan_case": "sqlite-create-target-source-copy", + "expected_plan_hash": "25b815bb706bd8692667d251f819c731176c5728205c98b51aa2ae509f39a2c2", + "expected_source_hash": "febe1c066647e5e50aa6c115a7ef33ade99592c4929a2d8a7411a7dec0ff4c15", + "expected_error": null + }, + { + "id": "mixed-recode-and-compute-promotes-v0.2", + "request": {"contract": "openstatspec-spss-syntax-frontend-v0.2", "input_alias": "parent", "input_schema": {"variables": [{"name": "q1", "storage_kind": "numeric"}]}, "source_text": "RECODE q1 (1 = 0).\nCOMPUTE target = q1."}, + "expected_plan_case": "mixed-recode-and-compute-promotes-v0.2", + "expected_plan_hash": "c90b19239ea7c816e19575a16e4ea14186a2c73762d6d447c369a4b9d2f1e35f", + "expected_source_hash": "80e14ec0c4dd0e5013099ed12e47a2dd4d6b0ab0e93f3f611168744699a7e688", + "expected_error": null + }, + { + "id": "binding-or-null-existing-target", + "request": {"contract": "openstatspec-spss-syntax-frontend-v0.2", "input_alias": "parent", "input_schema": {"variables": [{"name": "source_a", "storage_kind": "numeric"}, {"name": "source_b", "storage_kind": "numeric"}, {"name": "target", "storage_kind": "numeric"}]}, "source_text": "IF (source_a = 1 OR source_b = 1) target = 1."}, + "expected_plan_case": "binding-or-null-existing-target", + "expected_plan_hash": "7fa8a017ed2476606c20f7b6bc9df09b7bbff208891e03335dfa393c7c260742", + "expected_source_hash": "dc936264d68bdf22741a8476d650a4e58fc03a63ac47f86093cbbb732f31594a", + "expected_error": null + }, + { + "id": "binding-variable-missing-existing-target", + "request": {"contract": "openstatspec-spss-syntax-frontend-v0.2", "input_alias": "parent", "input_schema": {"variables": [{"name": "source_a", "storage_kind": "numeric"}, {"name": "target", "storage_kind": "numeric"}]}, "source_text": "COMPUTE target = source_a."}, + "expected_plan_case": "binding-variable-missing-existing-target", + "expected_plan_hash": "2b52cbd2989c99aac69dbd10e0f3de225ca84f3902193e05931bc3bf9c3e8649", + "expected_source_hash": "febe1c066647e5e50aa6c115a7ef33ade99592c4929a2d8a7411a7dec0ff4c15", + "expected_error": null + }, + { + "id": "binding-conditional-missing-existing-target", + "request": {"contract": "openstatspec-spss-syntax-frontend-v0.2", "input_alias": "parent", "input_schema": {"variables": [{"name": "source_a", "storage_kind": "numeric"}, {"name": "source_b", "storage_kind": "numeric"}, {"name": "target", "storage_kind": "numeric"}]}, "source_text": "IF (source_b = 1) target = source_a."}, + "expected_plan_case": "binding-conditional-missing-existing-target", + "expected_plan_hash": "aa42695ecf8d84d487fe1cdc1b11dd9d9e5d048df670877ed562e9b9fbd23053", + "expected_source_hash": "9fb6357cd8b436b69c523d86be0ce1415dc0fded15a425f737e6f3e37bd08b76", + "expected_error": null + }, + { + "id": "binding-inequality-boundaries-existing-targets", + "request": {"contract": "openstatspec-spss-syntax-frontend-v0.2", "input_alias": "parent", "input_schema": {"variables": [{"name": "source", "storage_kind": "numeric"}, {"name": "target_lt", "storage_kind": "numeric"}, {"name": "target_le", "storage_kind": "numeric"}, {"name": "target_gt", "storage_kind": "numeric"}, {"name": "target_ge", "storage_kind": "numeric"}]}, "source_text": "IF (source < 1) target_lt = 1.\nIF (source <= 1) target_le = 1.\nIF (source > 1) target_gt = 1.\nIF (source >= 1) target_ge = 1."}, + "expected_plan_case": "binding-inequality-boundaries-existing-targets", + "expected_plan_hash": "cc0bde9b2adbbe76614c51a6122a5c9c17c9e610a63241eb1769956c2e543690", + "expected_source_hash": "1208507803e9b69cb523585e83dd60643e61d3082f9b1d76fe4bca33b1db0011", + "expected_error": null + }, + { + "id": "reject-string-expression", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "color", + "storage_kind": "string" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "COMPUTE target = color." + }, + "expected_source_hash": "9d44d6aa5554a716d55ecbcf04e9744f59a9a4ecf0281652ab713a7482d581c4", + "expected_error": "expression_type_unsupported" + }, + { + "id": "reject-missing-conditional-target", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + } + ] + }, + "source_text": "IF (source_a = 1) missing_target = 1." + }, + "expected_source_hash": "7a7956e282fb7e2de12daa1f1a3af5cd89e20193e0c0269ba5c63a27ac81e06f", + "expected_error": "conditional_target_missing" + }, + { + "id": "reject-invalid-format", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "FORMATS target (F2.2)." + }, + "expected_source_hash": "307aee8e2ca8a03939d4d487a3c9303d78abfc82f55f6434cab92e710defe8d5", + "expected_error": "invalid_format" + }, + { + "id": "unparenthesized-mixed-default-precedence", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "source_c", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "IF (source_a = 1 OR source_b = 1 AND source_c = 1) target = 1.\nEXECUTE." + }, + "expected_plan_case": "mixed-default-and-precedence", + "expected_plan_hash": "5fc4c6e5c41661f6b0f7806d3e4b5054e8282d0f4f3f4ae9da60eae68b7f9d43", + "expected_source_hash": "5dff61244ba2461bc64b1cbef374ebcbb751e03792cf92efcca439ad693f8943", + "expected_error": null + }, + { + "id": "parenthesized-mixed-override-precedence", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "source_c", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "IF ((source_a = 1 OR source_b = 1) AND source_c = 1) target = 1.\nEXECUTE." + }, + "expected_plan_case": "mixed-parentheses-override-precedence", + "expected_plan_hash": "dd283eacad64052470bc44b6fb69817db94a95aa6391b34b93f522dcbc63a279", + "expected_source_hash": "e655fe1d0aa6829dde510bdb1e3253110c723e1fdf23c539cd20bee4b3e07660", + "expected_error": null + }, + { + "id": "finite-number-rounding-exponent-underflow", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "source_c", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "COMPUTE target = 1.00000000000000011102230246251565404236316680908203125.\nCOMPUTE target = 1.00000000000000033306690738754696212708950042724609375.\nCOMPUTE target = -1.25e+2.\nCOMPUTE target = 1e-4000.\nEXECUTE." + }, + "expected_plan_case": "finite-number-rounding-exponent-underflow", + "expected_plan_hash": "5abd09242071c929f3fe7e597ad605c9e2cf06a6e2c72fe2b62584dd4ccccb05", + "expected_source_hash": "bf1cbb16ed58a0fb37ac437aee7008d336d8efd024a35c654d31698cabaca21d", + "expected_error": null + }, + { + "id": "reject-numeric-overflow", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "source_c", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "COMPUTE target = 1e4000." + }, + "expected_source_hash": "9cd7389c9841eb18f95a2a377cc05e1f31a8d9212ed9afa92cce6d9da6b7c0f1", + "expected_error": "spss_syntax_error" + }, + { + "id": "reject-leading-decimal-point", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "source_c", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "COMPUTE target = .5." + }, + "expected_source_hash": "b89b9a5cbdd4ff8c6e4a00926788b5b9fc79d0124d7a5bdf913e6382933bbee0", + "expected_error": "spss_syntax_error" + }, + { + "id": "reject-trailing-decimal-point", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "source_c", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "COMPUTE target = 1.." + }, + "expected_source_hash": "a2d5ced231ee88930405e9e36f8f68ab86f57ba720fd02579e28616d9d71f1cf", + "expected_error": "spss_syntax_error" + }, + { + "id": "three-term-or-flattens-source-order", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "source_c", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "IF (source_a = 1 OR source_b = 1 OR source_c = 1) target = 1.\nEXECUTE." + }, + "expected_plan_case": "three-term-or-flattens-source-order", + "expected_plan_hash": "e61850fd0e65726dbb13af184091b726ed70ead948300ae1d6163df2c23663f9", + "expected_source_hash": "6aeb46935565cb16b84c503e6df4481d0542b095a68b9788e1c0f0a7a9fcf318", + "expected_error": null + }, + { + "id": "parenthesized-three-term-or-flattens-source-order", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "source_c", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "IF ((source_a = 1 OR source_b = 1) OR source_c = 1) target = 1.\nEXECUTE." + }, + "expected_plan_case": "three-term-or-flattens-source-order", + "expected_plan_hash": "e61850fd0e65726dbb13af184091b726ed70ead948300ae1d6163df2c23663f9", + "expected_source_hash": "26f683849987d5489d66822c8315f1b7be07c6583cfd88c8a8449cfe3ac9c3ea", + "expected_error": null + }, + { + "id": "strict-greater-than", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "source_c", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "IF (source_a > 1) target = 1.\nEXECUTE." + }, + "expected_plan_case": "strict-greater-than", + "expected_plan_hash": "9fd5d95f078924be3ce0809c985f47ffedb1c5fbbecadc739591d37df6297625", + "expected_source_hash": "f8e21c1e96b872c65395a7319dcfb4c93b60798a3e1c578fb7ba63825391afcf", + "expected_error": null + }, + { + "id": "comparison-operand-permutations", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "source_c", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "IF (source_a = source_b AND 1 < source_c AND 1 = 1) target = 1.\nEXECUTE." + }, + "expected_plan_case": "comparison-operand-permutations", + "expected_plan_hash": "cd8e04360419bc5b5d6db9a8800f4c58e267d2a29f0c2926cfba8d266892f021", + "expected_source_hash": "7d2a5903474103a5c317e0bf2e9eb4a1710bb5ca504b8083be6249bb8df323c6", + "expected_error": null + }, + { + "id": "reject-string-predicate", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "color", + "storage_kind": "string" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "IF (color = 'R') target = 1." + }, + "expected_source_hash": "97dfc44da12bff2666ecc2c0d91cffa8142e3789e6f11bfe47c00c537f116920", + "expected_error": "expression_type_unsupported" + }, + { + "id": "reject-arithmetic-expression", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "source_c", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "COMPUTE target = source_a + 1." + }, + "expected_source_hash": "eeeb5f24bc311284afb72a350b58a2787f2634465b9d6f453bd2c8817784484a", + "expected_error": "expression_type_unsupported" + }, + { + "id": "reject-unsupported-command", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "source_c", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "SORT CASES BY source_a." + }, + "expected_source_hash": "d0aedc01794f58e2233d2ae6207490b62edc4af7f80ff53cd41845d6a38925f5", + "expected_error": "unsupported_spss_command" + }, + { + "id": "reject-leading-plus", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "source_c", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "COMPUTE target = +1." + }, + "expected_source_hash": "7c54709d7262ef4a751c1b8b2b4909ae7b803f4f9e50d5b269d17deff572a9ba", + "expected_error": "spss_syntax_error" + }, + { + "id": "reject-leading-zero", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "source_c", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "COMPUTE target = 01." + }, + "expected_source_hash": "f2513185d45ff2051dc6b1184df1a7d61abdd12aee2772975b0d0a94e00f76f2", + "expected_error": "spss_syntax_error" + }, + { + "id": "reject-nan-token", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "source_c", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "COMPUTE target = NaN." + }, + "expected_source_hash": "73519477685fa98ea68625926a28d7dd4cde76587a755bb754bd8983ccd09ad2", + "expected_error": "spss_syntax_error" + }, + { + "id": "reject-infinity-token", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "source_c", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "COMPUTE target = Infinity." + }, + "expected_source_hash": "940c30da3b6232307e976bc96a0ea64056dc9605cf717faae3580e5b4f3f0752", + "expected_error": "spss_syntax_error" + }, + { + "id": "minimum-nonzero-subnormal", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "source_c", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "COMPUTE target = 5e-324.\nEXECUTE." + }, + "expected_plan_case": "minimum-nonzero-subnormal", + "expected_plan_hash": "fec5cad101adf699cfac0cbfbc1e0371f512f8e674345269d1de8d3a8a4538de", + "expected_source_hash": "d64f70960fbc1e42014c4049fdddf9c466c77e3dc896447de7a02ccaa2a8c191", + "expected_error": null + }, + { + "id": "grouped-format-and-measurement-level-order", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "source_b", + "storage_kind": "numeric" + }, + { + "name": "source_c", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "FORMATS source_a (F8.2) source_b (F10.3).\nVARIABLE LEVEL source_a source_b (ORDINAL) / source_c target (SCALE).\nEXECUTE." + }, + "expected_plan_case": "grouped-format-and-measurement-level-order", + "expected_plan_hash": "4b5e8d6c2e8043e25c1483b455dc497910cb66835916f6cdf7afa1eba8d334b8", + "expected_source_hash": "c9ba1298688876632af6cb435c8f363ae9d56f2475f4adf3d8ad63862e7aae20", + "expected_error": null + }, + { + "id": "reject-if-without-parentheses", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "IF source_a = 1 target = 1." + }, + "expected_source_hash": "543af0de6ddedd40be3b64e372a28fc897bb80db7cc7b0ec883dd24aa3de5eae", + "expected_error": "spss_syntax_error" + }, + { + "id": "reject-if-else-form", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "IF (source_a = 1) target = 1 ELSE target = 0." + }, + "expected_source_hash": "ea118668e92255019e7f70ea60c7c6aee2e1a45333d424b48a9f6c3ed3b8660a", + "expected_error": "spss_syntax_error" + }, + { + "id": "reject-non-f-format", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "FORMATS target (A8)." + }, + "expected_source_hash": "37a5fd0884cddbba67253ed76b5bc3f47a1dcd6b6acd75f207aa7f2bec40232b", + "expected_error": "invalid_format" + }, + { + "id": "reject-invalid-measurement-level", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "VARIABLE LEVEL target (UNKNOWN)." + }, + "expected_source_hash": "2bf027ec059026ac05177003ae52f9a4fb1bb994d16ba899dcebcb2df347ee61", + "expected_error": "spss_syntax_error" + }, + { + "id": "negative-zero-canonical-positive-zero", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "target", + "storage_kind": "numeric" + } + ] + }, + "source_text": "COMPUTE target = -0.\nEXECUTE." + }, + "expected_plan_case": "negative-zero-canonical-positive-zero", + "expected_plan_hash": "0e881516aef24bbd5e3c6e2afe12a2f7f712d58f33fcba2bce2b74691bfe5a9c", + "expected_source_hash": "3bce3721dc895b4f58cf180cc72792494936cbaf4e1995b8c09a5ab50fee98a5", + "expected_error": null + }, + { + "id": "reject-string-target-compute", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "color", + "storage_kind": "string" + } + ] + }, + "source_text": "COMPUTE color = 1." + }, + "expected_source_hash": "9e4d4b5f23e024b0d1dd70e03465a964f38aa17879e9701fe537c553645dda89", + "expected_error": "expression_type_unsupported" + }, + { + "id": "reject-string-target-conditional", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "color", + "storage_kind": "string" + } + ] + }, + "source_text": "IF (source_a = 1) color = 1." + }, + "expected_source_hash": "a0c3d748f308793c5c529e5b7b67743979bd1a9ae5e2c0235a66951202930eb1", + "expected_error": "expression_type_unsupported" + }, + { + "id": "reject-string-target-format", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "source_a", + "storage_kind": "numeric" + }, + { + "name": "color", + "storage_kind": "string" + } + ] + }, + "source_text": "FORMATS color (F8.2)." + }, + "expected_source_hash": "84514670d918d0bb42619c32c75aa85225ad9e4f24ac2a0df1cd584f1deb7d97", + "expected_error": "expression_type_unsupported" + }, + { + "id": "metadata-only-value-labels-retains-v0.1-plan", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "a", + "storage_kind": "numeric" + }, + { + "name": "b", + "storage_kind": "numeric" + }, + { + "name": "color", + "storage_kind": "string" + } + ] + }, + "source_text": "VALUE LABELS a b 0 'No' 1 'Yes' / color 'R' 'Red'." + }, + "expected_plan_contract": "openstatspec-transformation-plan-v0.1", + "expected_plan_case_0_1": "value-label-varlists-and-groups", + "expected_plan_hash": "632e2075874257f7bf2400d6a2642f9336a9fc4ddf9664a20150a333d32414ad", + "expected_source_hash": "94954e5cdeada2bdee1f862078cb89e620bea3a7cd7c0e24d8c735786d469174", + "expected_error": null + }, + { + "id": "metadata-only-variable-labels-retains-v0.1-plan", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "a", + "storage_kind": "numeric" + }, + { + "name": "b", + "storage_kind": "numeric" + } + ] + }, + "source_text": "VARIABLE LABELS a 'Alpha' b 'Beta'." + }, + "expected_plan_contract": "openstatspec-transformation-plan-v0.1", + "expected_plan_0_1": { + "contract": "openstatspec-transformation-plan-v0.1", + "input_alias": "parent", + "operations": [ + { + "op": "set_variable_label", + "variable": "a", + "label": "Alpha" + }, + { + "op": "set_variable_label", + "variable": "b", + "label": "Beta" + } + ] + }, + "expected_plan_hash": "a663a7160957447ab125cbe8a106ea3419b6eb0855ea0287dea5ce71b6b08f20", + "expected_source_hash": "b70575af343e328875753476282962fca4224de98e46c656f84a415dc033b162", + "expected_error": null + }, + { + "id": "string-measurement-level-target", + "request": { + "contract": "openstatspec-spss-syntax-frontend-v0.2", + "input_alias": "parent", + "input_schema": { + "variables": [ + { + "name": "color", + "storage_kind": "string" + } + ] + }, + "source_text": "VARIABLE LEVEL color (NOMINAL).\nEXECUTE." + }, + "expected_plan_case": "string-measurement-level-target", + "expected_plan_hash": "5693460d700531e6e9162a9cf112bd276b475f9d414bd58c283c6f2db8fdbf9d", + "expected_source_hash": "0887befe1d3a49a4e196976eb5925688cd402bbbfb6efae602a27a1fe0c8196e", + "expected_error": null + } + ] +} diff --git a/conformance/transformation-plan-0.2.json b/conformance/transformation-plan-0.2.json new file mode 100644 index 0000000..734c56f --- /dev/null +++ b/conformance/transformation-plan-0.2.json @@ -0,0 +1,1104 @@ +{ + "manifest_version": "0.2", + "profile": "OpenStatSpec Transformation Plan 0.2", + "contract": "openstatspec-transformation-plan-v0.2", + "schema": "../transformation/plan-0.2.schema.json", + "canonicalization": "restricted-rfc8785-utf8-sha256", + "cases": [ + { + "id": "sequential-conditional-binary-existing-target", + "plan": { + "contract": "openstatspec-transformation-plan-v0.2", + "input_alias": "parent", + "operations": [ + { + "op": "assign", + "target": "target", + "target_mode": "replace", + "value": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "0000000000000000" + } + } + }, + { + "op": "conditional_assign", + "condition": { + "expression": "boolean", + "operator": "and", + "operands": [ + { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_a" + }, + "operator": "=", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + }, + { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_b" + }, + "operator": "=", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + } + ] + }, + "target": "target", + "value": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + }, + { + "op": "set_variable_label", + "variable": "target", + "label": "Synthetic conjunction" + }, + { + "op": "replace_value_labels", + "variable": "target", + "labels": [ + { + "value": { + "type": "binary64", + "bits": "0000000000000000" + }, + "label": "No" + }, + { + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + }, + "label": "Yes" + } + ] + }, + { + "op": "set_format", + "variable": "target", + "family": "F", + "width": 1, + "decimals": 0 + }, + { + "op": "set_measurement_level", + "variable": "target", + "level": "nominal" + }, + { + "op": "execute" + } + ] + }, + "expected_plan_hash": "8e9f72759f6ed53aa6957dc288bdc49894ca2e08a8313ca957da6b4dfd11acbd", + "expected_error": null + }, + { + "id": "three-term-and-flattens-source-order", + "plan": { + "contract": "openstatspec-transformation-plan-v0.2", + "input_alias": "parent", + "operations": [ + { + "op": "conditional_assign", + "condition": { + "expression": "boolean", + "operator": "and", + "operands": [ + { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_a" + }, + "operator": "=", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + }, + { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_b" + }, + "operator": "=", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + }, + { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_c" + }, + "operator": "=", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + } + ] + }, + "target": "target", + "value": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + }, + { + "op": "execute" + } + ] + }, + "expected_plan_hash": "d6f94edd5de261552e79157cd29ff7668e824476827fae7f0fe9f56ba45704ea", + "expected_error": null + }, + { + "id": "reject-string-predicate", + "plan": { + "contract": "openstatspec-transformation-plan-v0.2", + "input_alias": "parent", + "operations": [ + { + "op": "conditional_assign", + "condition": { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "color" + }, + "operator": "=", + "right": { + "kind": "literal", + "value": { + "type": "string", + "value": "R" + } + } + }, + "target": "target", + "value": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + } + ] + }, + "expected_plan_hash": null, + "expected_error": "expression_type_unsupported" + }, + { + "id": "nested-or-inequalities-variable-operands-create", + "plan": { + "contract": "openstatspec-transformation-plan-v0.2", + "input_alias": "parent", + "operations": [ + { + "op": "assign", + "target": "target", + "target_mode": "create", + "value": { + "kind": "variable", + "variable": "source_a" + } + }, + { + "op": "conditional_assign", + "condition": { + "expression": "boolean", + "operator": "or", + "operands": [ + { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_a" + }, + "operator": ">=", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "0000000000000000" + } + } + }, + { + "expression": "boolean", + "operator": "and", + "operands": [ + { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_b" + }, + "operator": "<", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "4014000000000000" + } + } + }, + { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_c" + }, + "operator": "<=", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "4022000000000000" + } + } + } + ] + } + ] + }, + "target": "target", + "value": { + "kind": "variable", + "variable": "source_b" + } + }, + { + "op": "execute" + } + ] + }, + "expected_plan_hash": "926f2cd93ed0bfa330bdb2d0b31b241f552c86c116dd35a315a20408bf997daf", + "expected_error": null + }, + { + "id": "sqlite-create-target-source-copy", + "plan": {"contract": "openstatspec-transformation-plan-v0.2", "input_alias": "parent", "operations": [{"op": "assign", "target": "target", "target_mode": "create", "value": {"kind": "variable", "variable": "source_a"}}]}, + "expected_plan_hash": "25b815bb706bd8692667d251f819c731176c5728205c98b51aa2ae509f39a2c2", + "expected_error": null + }, + { + "id": "mixed-recode-and-compute-promotes-v0.2", + "plan": {"contract": "openstatspec-transformation-plan-v0.2", "input_alias": "parent", "operations": [{"op": "recode", "source": "q1", "target": "q1", "target_mode": "replace", "rules": [{"match": {"kind": "values", "values": [{"type": "binary64", "bits": "3ff0000000000000"}]}, "result": {"kind": "literal", "value": {"type": "binary64", "bits": "0000000000000000"}}}], "unmatched": {"kind": "copy"}}, {"op": "assign", "target": "target", "target_mode": "create", "value": {"kind": "variable", "variable": "q1"}}]}, + "expected_plan_hash": "c90b19239ea7c816e19575a16e4ea14186a2c73762d6d447c369a4b9d2f1e35f", + "expected_error": null + }, + { + "id": "binding-or-null-existing-target", + "plan": {"contract": "openstatspec-transformation-plan-v0.2", "input_alias": "parent", "operations": [{"op": "conditional_assign", "condition": {"expression": "boolean", "operator": "or", "operands": [{"expression": "comparison", "left": {"kind": "variable", "variable": "source_a"}, "operator": "=", "right": {"kind": "literal", "value": {"type": "binary64", "bits": "3ff0000000000000"}}}, {"expression": "comparison", "left": {"kind": "variable", "variable": "source_b"}, "operator": "=", "right": {"kind": "literal", "value": {"type": "binary64", "bits": "3ff0000000000000"}}}]}, "target": "target", "value": {"kind": "literal", "value": {"type": "binary64", "bits": "3ff0000000000000"}}}]}, + "expected_plan_hash": "7fa8a017ed2476606c20f7b6bc9df09b7bbff208891e03335dfa393c7c260742", + "expected_error": null + }, + { + "id": "binding-variable-missing-existing-target", + "plan": {"contract": "openstatspec-transformation-plan-v0.2", "input_alias": "parent", "operations": [{"op": "assign", "target": "target", "target_mode": "replace", "value": {"kind": "variable", "variable": "source_a"}}]}, + "expected_plan_hash": "2b52cbd2989c99aac69dbd10e0f3de225ca84f3902193e05931bc3bf9c3e8649", + "expected_error": null + }, + { + "id": "binding-conditional-missing-existing-target", + "plan": {"contract": "openstatspec-transformation-plan-v0.2", "input_alias": "parent", "operations": [{"op": "conditional_assign", "condition": {"expression": "comparison", "left": {"kind": "variable", "variable": "source_b"}, "operator": "=", "right": {"kind": "literal", "value": {"type": "binary64", "bits": "3ff0000000000000"}}}, "target": "target", "value": {"kind": "variable", "variable": "source_a"}}]}, + "expected_plan_hash": "aa42695ecf8d84d487fe1cdc1b11dd9d9e5d048df670877ed562e9b9fbd23053", + "expected_error": null + }, + { + "id": "binding-inequality-boundaries-existing-targets", + "plan": {"contract": "openstatspec-transformation-plan-v0.2", "input_alias": "parent", "operations": [{"op": "conditional_assign", "condition": {"expression": "comparison", "left": {"kind": "variable", "variable": "source"}, "operator": "<", "right": {"kind": "literal", "value": {"type": "binary64", "bits": "3ff0000000000000"}}}, "target": "target_lt", "value": {"kind": "literal", "value": {"type": "binary64", "bits": "3ff0000000000000"}}}, {"op": "conditional_assign", "condition": {"expression": "comparison", "left": {"kind": "variable", "variable": "source"}, "operator": "<=", "right": {"kind": "literal", "value": {"type": "binary64", "bits": "3ff0000000000000"}}}, "target": "target_le", "value": {"kind": "literal", "value": {"type": "binary64", "bits": "3ff0000000000000"}}}, {"op": "conditional_assign", "condition": {"expression": "comparison", "left": {"kind": "variable", "variable": "source"}, "operator": ">", "right": {"kind": "literal", "value": {"type": "binary64", "bits": "3ff0000000000000"}}}, "target": "target_gt", "value": {"kind": "literal", "value": {"type": "binary64", "bits": "3ff0000000000000"}}}, {"op": "conditional_assign", "condition": {"expression": "comparison", "left": {"kind": "variable", "variable": "source"}, "operator": ">=", "right": {"kind": "literal", "value": {"type": "binary64", "bits": "3ff0000000000000"}}}, "target": "target_ge", "value": {"kind": "literal", "value": {"type": "binary64", "bits": "3ff0000000000000"}}}]}, + "expected_plan_hash": "cc0bde9b2adbbe76614c51a6122a5c9c17c9e610a63241eb1769956c2e543690", + "expected_error": null + }, + { + "id": "reject-invalid-format", + "plan": { + "contract": "openstatspec-transformation-plan-v0.2", + "input_alias": "parent", + "operations": [ + { + "op": "set_format", + "variable": "target", + "family": "F", + "width": 2, + "decimals": 2 + } + ] + }, + "expected_plan_hash": null, + "expected_error": "invalid_format" + }, + { + "id": "reject-reserved-assignment-target", + "plan": { + "contract": "openstatspec-transformation-plan-v0.2", + "input_alias": "parent", + "operations": [ + { + "op": "assign", + "target": "__target", + "target_mode": "create", + "value": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "0000000000000000" + } + } + } + ] + }, + "expected_plan_hash": null, + "expected_error": "reserved_target_name" + }, + { + "id": "reject-duplicate-value-label", + "plan": { + "contract": "openstatspec-transformation-plan-v0.2", + "input_alias": "parent", + "operations": [ + { + "op": "replace_value_labels", + "variable": "target", + "labels": [ + { + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + }, + "label": "One" + }, + { + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + }, + "label": "Duplicate one" + } + ] + } + ] + }, + "expected_plan_hash": null, + "expected_error": "duplicate_value_label" + }, + { + "id": "mixed-default-and-precedence", + "plan": { + "contract": "openstatspec-transformation-plan-v0.2", + "input_alias": "parent", + "operations": [ + { + "op": "conditional_assign", + "condition": { + "expression": "boolean", + "operator": "or", + "operands": [ + { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_a" + }, + "operator": "=", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + }, + { + "expression": "boolean", + "operator": "and", + "operands": [ + { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_b" + }, + "operator": "=", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + }, + { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_c" + }, + "operator": "=", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + } + ] + } + ] + }, + "target": "target", + "value": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + }, + { + "op": "execute" + } + ] + }, + "expected_error": null, + "expected_plan_hash": "5fc4c6e5c41661f6b0f7806d3e4b5054e8282d0f4f3f4ae9da60eae68b7f9d43" + }, + { + "id": "mixed-parentheses-override-precedence", + "plan": { + "contract": "openstatspec-transformation-plan-v0.2", + "input_alias": "parent", + "operations": [ + { + "op": "conditional_assign", + "condition": { + "expression": "boolean", + "operator": "and", + "operands": [ + { + "expression": "boolean", + "operator": "or", + "operands": [ + { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_a" + }, + "operator": "=", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + }, + { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_b" + }, + "operator": "=", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + } + ] + }, + { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_c" + }, + "operator": "=", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + } + ] + }, + "target": "target", + "value": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + }, + { + "op": "execute" + } + ] + }, + "expected_error": null, + "expected_plan_hash": "dd283eacad64052470bc44b6fb69817db94a95aa6391b34b93f522dcbc63a279" + }, + { + "id": "finite-number-rounding-exponent-underflow", + "plan": { + "contract": "openstatspec-transformation-plan-v0.2", + "input_alias": "parent", + "operations": [ + { + "op": "assign", + "target": "target", + "target_mode": "replace", + "value": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + }, + { + "op": "assign", + "target": "target", + "target_mode": "replace", + "value": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000002" + } + } + }, + { + "op": "assign", + "target": "target", + "target_mode": "replace", + "value": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "c05f400000000000" + } + } + }, + { + "op": "assign", + "target": "target", + "target_mode": "replace", + "value": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "0000000000000000" + } + } + }, + { + "op": "execute" + } + ] + }, + "expected_error": null, + "expected_plan_hash": "5abd09242071c929f3fe7e597ad605c9e2cf06a6e2c72fe2b62584dd4ccccb05" + }, + { + "id": "three-term-or-flattens-source-order", + "plan": { + "contract": "openstatspec-transformation-plan-v0.2", + "input_alias": "parent", + "operations": [ + { + "op": "conditional_assign", + "condition": { + "expression": "boolean", + "operator": "or", + "operands": [ + { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_a" + }, + "operator": "=", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + }, + { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_b" + }, + "operator": "=", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + }, + { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_c" + }, + "operator": "=", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + } + ] + }, + "target": "target", + "value": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + }, + { + "op": "execute" + } + ] + }, + "expected_error": null, + "expected_plan_hash": "e61850fd0e65726dbb13af184091b726ed70ead948300ae1d6163df2c23663f9" + }, + { + "id": "strict-greater-than", + "plan": { + "contract": "openstatspec-transformation-plan-v0.2", + "input_alias": "parent", + "operations": [ + { + "op": "conditional_assign", + "condition": { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_a" + }, + "operator": ">", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + }, + "target": "target", + "value": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + }, + { + "op": "execute" + } + ] + }, + "expected_error": null, + "expected_plan_hash": "9fd5d95f078924be3ce0809c985f47ffedb1c5fbbecadc739591d37df6297625" + }, + { + "id": "comparison-operand-permutations", + "plan": { + "contract": "openstatspec-transformation-plan-v0.2", + "input_alias": "parent", + "operations": [ + { + "op": "conditional_assign", + "condition": { + "expression": "boolean", + "operator": "and", + "operands": [ + { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_a" + }, + "operator": "=", + "right": { + "kind": "variable", + "variable": "source_b" + } + }, + { + "expression": "comparison", + "left": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + }, + "operator": "<", + "right": { + "kind": "variable", + "variable": "source_c" + } + }, + { + "expression": "comparison", + "left": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + }, + "operator": "=", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + } + ] + }, + "target": "target", + "value": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + }, + { + "op": "execute" + } + ] + }, + "expected_plan_hash": "cd8e04360419bc5b5d6db9a8800f4c58e267d2a29f0c2926cfba8d266892f021", + "expected_error": null + }, + { + "id": "minimum-nonzero-subnormal", + "plan": { + "contract": "openstatspec-transformation-plan-v0.2", + "input_alias": "parent", + "operations": [ + { + "op": "assign", + "target": "target", + "target_mode": "replace", + "value": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "0000000000000001" + } + } + }, + { + "op": "execute" + } + ] + }, + "expected_error": null, + "expected_plan_hash": "fec5cad101adf699cfac0cbfbc1e0371f512f8e674345269d1de8d3a8a4538de" + }, + { + "id": "grouped-format-and-measurement-level-order", + "plan": { + "contract": "openstatspec-transformation-plan-v0.2", + "input_alias": "parent", + "operations": [ + { + "op": "set_format", + "variable": "source_a", + "family": "F", + "width": 8, + "decimals": 2 + }, + { + "op": "set_format", + "variable": "source_b", + "family": "F", + "width": 10, + "decimals": 3 + }, + { + "op": "set_measurement_level", + "variable": "source_a", + "level": "ordinal" + }, + { + "op": "set_measurement_level", + "variable": "source_b", + "level": "ordinal" + }, + { + "op": "set_measurement_level", + "variable": "source_c", + "level": "scale" + }, + { + "op": "set_measurement_level", + "variable": "target", + "level": "scale" + }, + { + "op": "execute" + } + ] + }, + "expected_error": null, + "expected_plan_hash": "4b5e8d6c2e8043e25c1483b455dc497910cb66835916f6cdf7afa1eba8d334b8" + }, + { + "id": "negative-zero-canonical-positive-zero", + "plan": { + "contract": "openstatspec-transformation-plan-v0.2", + "input_alias": "parent", + "operations": [ + { + "op": "assign", + "target": "target", + "target_mode": "replace", + "value": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "0000000000000000" + } + } + }, + { + "op": "execute" + } + ] + }, + "expected_plan_hash": "0e881516aef24bbd5e3c6e2afe12a2f7f712d58f33fcba2bce2b74691bfe5a9c", + "expected_error": null + }, + { + "id": "reject-string-assign-value", + "plan": { + "contract": "openstatspec-transformation-plan-v0.2", + "input_alias": "parent", + "operations": [ + { + "op": "assign", + "target": "target", + "target_mode": "replace", + "value": { + "kind": "literal", + "value": { + "type": "string", + "value": "R" + } + } + } + ] + }, + "expected_plan_hash": null, + "expected_error": "expression_type_unsupported" + }, + { + "id": "reject-string-conditional-assign-value", + "plan": { + "contract": "openstatspec-transformation-plan-v0.2", + "input_alias": "parent", + "operations": [ + { + "op": "conditional_assign", + "condition": { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_a" + }, + "operator": "=", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + }, + "target": "target", + "value": { + "kind": "literal", + "value": { + "type": "string", + "value": "R" + } + } + } + ] + }, + "expected_plan_hash": null, + "expected_error": "expression_type_unsupported" + }, + { + "id": "string-measurement-level-target", + "plan": { + "contract": "openstatspec-transformation-plan-v0.2", + "input_alias": "parent", + "operations": [ + { + "op": "set_measurement_level", + "variable": "color", + "level": "nominal" + }, + { + "op": "execute" + } + ] + }, + "expected_plan_hash": "5693460d700531e6e9162a9cf112bd276b475f9d414bd58c283c6f2db8fdbf9d", + "expected_error": null + }, + { + "id": "reject-nested-same-operator-boolean", + "plan": { + "contract": "openstatspec-transformation-plan-v0.2", + "input_alias": "parent", + "operations": [ + { + "op": "conditional_assign", + "condition": { + "expression": "boolean", + "operator": "and", + "operands": [ + { + "expression": "boolean", + "operator": "and", + "operands": [ + { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_a" + }, + "operator": "=", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + }, + { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_b" + }, + "operator": "=", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + } + ] + }, + { + "expression": "comparison", + "left": { + "kind": "variable", + "variable": "source_c" + }, + "operator": "=", + "right": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + } + ] + }, + "target": "target", + "value": { + "kind": "literal", + "value": { + "type": "binary64", + "bits": "3ff0000000000000" + } + } + } + ] + }, + "expected_plan_hash": null, + "expected_error": "noncanonical_boolean_shape" + } + ] +} diff --git a/docs/spss-syntax-frontend-profile-0.2.md b/docs/spss-syntax-frontend-profile-0.2.md new file mode 100644 index 0000000..cab875c --- /dev/null +++ b/docs/spss-syntax-frontend-profile-0.2.md @@ -0,0 +1,132 @@ +# OpenStatSpec SPSS-like Syntax Frontend Profile 0.2 + +## Status and boundary + +Status: release candidate for the planned OpenStatSpec `v0.3.0` release. This profile is not a published stable specification until a `v0.3.0` tag targets its exact commit. + +This optional frontend maps a bounded SPSS-like command language to +Transformation Plan 0.1 or 0.2. Its request contract is +`openstatspec-spss-syntax-frontend-v0.2`. It is not full IBM SPSS Statistics +syntax and MUST reject every command, expression, or coercion not defined here. + +All lexical rules, command termination, exact source hashing, case-insensitive +name resolution, quote handling, and failure atomicity from Frontend 0.1 remain +in force. The input request conforms to +[`../transformation/spss-syntax-frontend-0.2.schema.json`](../transformation/spss-syntax-frontend-0.2.schema.json). + +The 0.1 commands `RECODE`, `VARIABLE LABELS`, and `VALUE LABELS` are +unchanged. A program containing only that subset MUST emit an exact Plan 0.1 +object. A program containing any command introduced below emits Plan 0.2. + +## Numeric expression grammar + +The bounded grammar is: + +```text +operand := numeric-variable | finite-number +comparison := operand ("=" | "<" | "<=" | ">" | ">=") operand +predicate := comparison | "(" predicate ")" + | predicate AND predicate + | predicate OR predicate +``` + +Comparison binds tighter than `AND`; `AND` binds tighter than `OR`. +A maximal chain of the same boolean operator MUST lower to one n-ary boolean +plan node whose operands remain in source order. Thus `a AND b AND c` lowers +to one `and` node with three operands, never left- or right-nested binary +`and` nodes. Parentheses override precedence across different operators; +parentheses around a same-operator subexpression do not prevent this required +flattening. Mixed operators remain nested according to precedence and explicit +parentheses. Keywords are ASCII case-insensitive. Unary operators, arithmetic, +functions, strings, `NOT`, `~=`, `<>`, `MISSING`, locale-dependent +numbers, and implicit numeric/string coercion are unsupported. + +The `finite-number` token has this exact ASCII grammar: + +```text +finite-number := "-"? ("0" | [1-9] [0-9]*) ("." [0-9]+)? + ([eE] [+-]? [0-9]+)? +``` + +Whitespace is not allowed inside the token. A leading plus, leading decimal +point, trailing decimal point, leading zero before another integer digit, +`NaN`, and `Infinity` are rejected. Thus `-1` and `1E2` are accepted, +while `.5` and `1.` are rejected. + +The frontend MUST interpret the token as an exact base-ten value, independent +of host locale, then correctly round it to the nearest IEEE 754 binary64 value +using ties-to-even. It MUST NOT first round through a host decimal or extended +binary floating-point type. A result whose magnitude overflows to infinity is +rejected with `spss_syntax_error`; a zero result, including signed or +underflowed zero, is canonicalized to positive-zero bits +`0000000000000000`. Emitted bits use exactly 16 lowercase hexadecimal digits. +NaN and infinity can never be emitted. + +A numeric variable is resolved at the point its command is lowered, including +variables created by preceding commands. All variables used as expression operands, assignment targets, or `FORMATS` +targets MUST be numeric. `VARIABLE LEVEL` targets MAY be numeric or string +variables; measurement level is metadata and does not coerce stored values. + +## Added commands + +`COMPUTE target = operand.` lowers to one ordered `assign`. If `target` +does not yet exist, it uses `target_mode=create`; otherwise it uses +`target_mode=replace`. No expression beyond the bounded operand form is +accepted in 0.2. + +`IF (predicate) target = operand.` lowers to one `conditional_assign`. +Parentheses around the complete predicate are required. The target MUST exist +at that point in command order. There is no ELSE form: a row is changed only +when the predicate is TRUE. FALSE and UNKNOWN retain the previous target value. + +`FORMATS variable (Fwidth.decimals) [variable +(Fwidth.decimals) ...].` lowers each pair in source order to `set_format`. +Only the numeric `F` family is supported. Family is canonicalized to uppercase; +width and decimals are canonical decimal integers. + +`VARIABLE LEVEL variable-list (NOMINAL|ORDINAL|SCALE) +[/ variable-list (level) ...].` lowers one `set_measurement_level` operation +per resolved variable, preserving group and variable order. Levels are +canonicalized to lowercase. + +`EXECUTE.` lowers to `{"op":"execute"}`. It marks source intent and ordered +identity but does not perform or authorize a database commit. + +For example: + +```spss +COMPUTE target = 0. +IF (source_a = 1 AND source_b = 1) target = 1. +VARIABLE LABELS target 'Example label'. +VALUE LABELS target 0 'No' 1 'Yes'. +FORMATS target (F1.0). +VARIABLE LEVEL target (NOMINAL). +EXECUTE. +``` + +lowers to sequential initialization, conditional replacement, label, value +labels, format, measurement level, and execution-boundary operations. No +command may be dropped. + +## System missing + +A variable operand may evaluate to numeric system missing. `COMPUTE` copies +that missing value. Comparisons involving missing are UNKNOWN, boolean +predicates use SQL three-valued truth, and `IF` writes only for TRUE. The +frontend MUST NOT silently turn missing into zero or FALSE. + +## Failure contract + +The frontend returns no partial plan on error. In addition to 0.1 diagnostics, +stable diagnostics include `expression_type_unsupported`, +`conditional_target_missing`, and `invalid_format`. Unsupported commands or +expression productions return `unsupported_spss_command` or +`spss_syntax_error` as appropriate. Diagnostics identify source spans without +including credentials or unrelated row values. + +The source hash remains SHA-256 of UTF-8 source after CRLF and CR are converted +to LF. The complete emitted plan is hashed independently under its declared +plan contract. + +Machine-readable golden cases are in +[`../conformance/spss-syntax-frontend-0.2.json`](../conformance/spss-syntax-frontend-0.2.json). diff --git a/docs/transformation-plan-profile-0.2.md b/docs/transformation-plan-profile-0.2.md new file mode 100644 index 0000000..f3380fc --- /dev/null +++ b/docs/transformation-plan-profile-0.2.md @@ -0,0 +1,127 @@ +# OpenStatSpec Transformation Plan Profile 0.2 + +## Status and scope + +Status: release candidate for the planned OpenStatSpec `v0.3.0` release. This profile is not a published stable specification until a `v0.3.0` tag targets its exact commit. + +This optional profile is the backward-compatible successor to Transformation +Plan 0.1. Its contract identifier is +`openstatspec-transformation-plan-v0.2`. The key words **MUST**, **MUST NOT**, +**SHOULD**, **SHOULD NOT**, and **MAY** are normative. + +Version 0.2 retains every 0.1 operation unchanged and adds bounded numeric +assignment, conditional assignment, display-format, measurement-level, and +execution-boundary operations. It remains a language-neutral plan rather than +an arbitrary expression language, SQL fragment, or execution program. A plan +MUST conform to +[`../transformation/plan-0.2.schema.json`](../transformation/plan-0.2.schema.json). + +## Canonical identity and backward compatibility + +A 0.2 plan contains exactly `contract`, `input_alias`, and a non-empty ordered +`operations` array. Canonical JSON and `plan_hash` use the same restricted RFC +8785 and SHA-256 rules as Plan 0.1. The hash covers the complete exact plan +object, including every expression node and metadata operation. Object member +order is immaterial; array order, expression shape, typed binary64 bits, and +exact unnormalized Unicode strings are material. + +The 0.1 schemas, contract identifiers, fixtures, canonical bytes, and hashes are +immutable. A consumer MUST validate a plan against the schema named by its +contract and MUST NOT silently reinterpret, normalize, or upgrade a 0.1 plan as +0.2. An implementation that migrates source text recompiles it explicitly, +produces a separately identified plan, and preserves the prior plan and audit. +A 0.2-capable SPSS frontend emits a 0.1 plan for a program containing only the +0.1 command subset; this preserves old plan hashes. + +Typed values, finite binary64 requirements, positive-zero canonicalization, +exact strings, reserved identifiers, and existing `recode`, +`set_variable_label`, and `replace_value_labels` semantics are unchanged +from Plan 0.1. + +## Ordered operations + +Operations execute strictly in array order. Each operation observes data, +schema, and metadata resulting from all preceding operations in the same plan. +This sequential rule is material: an `assign` may initialize a variable and a +following `conditional_assign` may selectively replace its initialized value. + +### Numeric operands and predicates + +Version 0.2 operands are either: + +- `{"kind":"variable","variable":"name"}`, resolved to one numeric variable; or +- `{"kind":"literal","value":typed-value}`, where the bounded assignment and + predicate subset requires a finite numeric binary64 value. + +String operands and string predicates are outside 0.2. They fail before +mutation with `expression_type_unsupported`. This avoids implicit SQL +collation or coercion semantics. + +A comparison node has `expression=comparison`, two operands, and one of +`=`, `<`, `<=`, `>`, or `>=`. A boolean node has +`expression=boolean`, `operator=and|or`, and at least two ordered predicate +operands. A maximal same-operator chain MUST be represented by one n-ary node +whose operands remain in source order; a boolean node MUST NOT have a direct +boolean child with the same operator. Parentheses preserve grouping across +different operators, but same-operator parentheses do not prevent the required +flattening. Implementations MUST NOT otherwise simplify, reorder, deduplicate, +distribute, or constant-fold nodes before canonical hashing. + +Predicates use SQL three-valued truth semantics. A comparison with numeric +system missing (SQL `NULL`) is UNKNOWN. `AND` and `OR` use their standard +three-valued truth tables. A conditional assignment writes only where its +predicate is TRUE; FALSE and UNKNOWN both retain the current target value. + +### assign + +`assign` contains `target`, `target_mode=create|replace`, and one numeric +operand. It evaluates that operand against the pre-operation row and writes the +result to every case. A variable operand that is system missing writes system +missing. `create` appends a fresh numeric variable; `replace` requires an +existing numeric target and retains identity, ordinal, and metadata. + +### conditional_assign + +`conditional_assign` contains a predicate, an existing numeric `target`, and +one numeric operand. It evaluates the predicate and value against the +pre-operation row, writes only when the predicate is TRUE, and otherwise leaves +the target unchanged. The target MUST already exist when this operation binds. +It has no implicit ELSE branch and never creates a variable. + +### metadata and execute + +`set_format` replaces only the numeric variable's print/write format with +family `F`, width 1 through 40, and decimals 0 through 16. Width and decimals +MUST form a valid SPSS F format; when decimals are nonzero, width MUST provide +space for a sign, decimal separator, and the requested decimals. Invalid forms +fail with `invalid_format`. + +`set_measurement_level` replaces only the variable measurement level with +`nominal`, `ordinal`, or `scale`. + +`execute` is an explicit ordered execution boundary retained in the canonical +plan. In one atomic public apply it does not commit, start a second apply, or +publish intermediate state. It performs no row or metadata mutation by itself. + +Newly created variables begin without labels, value labels, missing-value +metadata, or measurement metadata. Later ordered metadata operations establish +those properties. Existing targets retain metadata except for the exact +metadata component explicitly replaced by a later operation. + +## Binding, validation, and diagnostics + +Structural validation, name resolution, type checking, target-state checking, +predicate validation, and backend capability checks all precede mutation. +Stable 0.2 diagnostics add `expression_type_unsupported`, +`conditional_target_missing`, `invalid_format`, and +`noncanonical_boolean_shape`, and retain the 0.1 diagnostics. A canonical +boolean expression MUST NOT nest the same `and` or `or` operator; it is +flattened in source order instead. + +The in-place SQL binding is +[`transformation-plan-sql-binding-0.2.md`](transformation-plan-sql-binding-0.2.md). +A fixed plan, bound schema, implementation version, and target profile MUST +compile deterministically. Generated SQL is not canonical identity. + +Machine-readable independent golden cases are in +[`../conformance/transformation-plan-0.2.json`](../conformance/transformation-plan-0.2.json). diff --git a/docs/transformation-plan-sql-binding-0.2.md b/docs/transformation-plan-sql-binding-0.2.md new file mode 100644 index 0000000..b268c87 --- /dev/null +++ b/docs/transformation-plan-sql-binding-0.2.md @@ -0,0 +1,100 @@ +# In-Place Transformation Binding 0.2 + +## Status + +Status: release candidate for the planned OpenStatSpec `v0.3.0` release. This profile is not a published stable specification until a `v0.3.0` tag targets its exact commit. + +## Normative decision + +This binding extends In-Place Transformation Binding 0.1 for Transformation +Plan 0.2. It preserves one logical dataset, one physical wide table, dataset +identity, physical table identity, case order, and case count. It creates no +derived dataset, persistent copy, snapshot, OpenStatSpec rollback layer, or +automatic database commit. + +Existing-target `assign` compiles to a direct all-row `UPDATE`. +`conditional_assign` compiles to a direct `UPDATE ... WHERE predicate`; +only SQL TRUE rows change. Metadata operations update only their normative +catalog fields. `execute` compiles to no SQL statement. + +## Target creation by SQL profile + +SQLite and PostgreSQL MAY execute `target_mode=create` only where the adapter +has test evidence that schema, data, catalogs, and compact audit are one native +transaction. + +MySQL, MariaDB, and Dolt MUST reject `target_mode=create` before mutation with +`schema_change_not_atomic`. Version 0.2 does not claim atomic create-target +apply on these profiles and MUST NOT use a copied table or compensating +recovery artifact to simulate it. + +A caller that needs a new target on MySQL, MariaDB, or Dolt performs a separate, +explicit, versioned provisioning action before the transformation apply. That +action MUST create both: + +- one nullable numeric physical column in the existing wide table; and +- one matching normative catalog variable with the same physical binding, + unique variable identity, and next ordinal. + +Provisioning is not a transformation apply and MUST NOT be hidden inside one. +It must leave no uncataloged physical column or catalog-only variable. Before a +later apply, the target resolves as an existing numeric variable, so +`COMPUTE target = ...` lowers to `assign target_mode=replace`. The apply then +contains only DML, metadata changes, and compact audit. + +For Dolt, the provisioning action is committed separately by the caller. The +subsequent transformation apply begins from that clean committed HEAD and uses +that exact commit as `expected_head`. OpenStatSpec never calls `DOLT_COMMIT` +for either action. + +## Controlled Dolt context + +Every apply requires a non-empty actor. An absent or empty actor MUST fail +before mutation with the stable actor_required diagnostic. Dolt additionally requires +`expected_branch`, `expected_head`, and a clean working set. Branch, HEAD, +and cleanliness are checked before mutation. Branch and HEAD are checked again +before successful completion. Stable diagnostics are +`dolt_context_required`, `dolt_branch_mismatch`, `dolt_head_mismatch`, +`dolt_working_set_dirty`, and `dolt_context_changed`. + +The apply remains on the same branch and MUST NOT commit, switch, merge, reset, +tag, or create a persistent recovery object. A successful apply leaves one +inspectable Dolt working-set diff. The audit's before and after HEAD identities +are equal. + +## Atomicity and failure + +All plan operations, normative and compatibility metadata changes, and the one +compact audit row belong to one public apply boundary. For an existing target, +the adapter uses the engine's native transaction semantics. A failure leaves +data, metadata, and audit unchanged. + +Capability and target checks happen before mutation. A profile that cannot +provide the required boundary fails closed. Pre-existing unrelated state is +never treated as apply-owned cleanup state. + +The compact audit and forbidden-artifact requirements from Binding 0.1 are +unchanged. It records exact plan/source hashes and operation count, not row +values or copied state. + +## Audit schema compatibility + +The versioned `sql/transformation-plan-profile-schema.sql` DDL accepts both the +0.1 and 0.2 in-place binding contract identifiers in the same logical +`transformation_apply` table. Before the first 0.2 apply, an installation whose +0.1 table still has the single-contract check MUST migrate that check using its +database profile's transactional DDL procedure where available. The migration +preserves every existing compact audit row and logical audit-table identity. + +SQLite, whose `ALTER TABLE` cannot drop the existing table-level check, MUST +perform the standard create/copy/drop/rename table rebuild inside one native +transaction. The replacement table is strictly ephemeral migration state: it +must use the versioned DDL, receive only the existing compact audit rows, and be +renamed to `transformation_apply` before commit. After successful migration +there MUST be exactly one persistent audit table, with no dataset, data-table +copy, snapshot, rollback artifact, or recovery-version layer added. A failure +rolls back the entire rebuild. This schema migration is completed before an +apply and is not part of the transformation apply boundary. + +Machine-readable binding cases are in +[`../conformance/in-place-transformation-0.2.json`](../conformance/in-place-transformation-0.2.json). diff --git a/examples/spss-syntax-transformation-0.2.md b/examples/spss-syntax-transformation-0.2.md new file mode 100644 index 0000000..d12bd86 --- /dev/null +++ b/examples/spss-syntax-transformation-0.2.md @@ -0,0 +1,30 @@ +# Conditional SPSS-like transformation on an existing target + +Transformation Plan and SPSS Syntax Frontend 0.2 support bounded sequential +numeric assignment. Assume `target` already exists as a nullable numeric +physical column and normative catalog variable: + +```spss +COMPUTE target = 0. +IF (source_a = 1 AND source_b = 1) target = 1. +VARIABLE LABELS target 'Synthetic conjunction'. +VALUE LABELS target 0 'No' 1 'Yes'. +FORMATS target (F1.0). +VARIABLE LEVEL target (NOMINAL). +EXECUTE. +``` + +`COMPUTE` initializes every case. `IF` then changes only rows whose complete +predicate is TRUE. If either comparison is UNKNOWN because a source is numeric +system missing, the row retains zero. The remaining commands establish the +label, complete value-label map, F1.0 format, and nominal measurement level. + +On SQLite or PostgreSQL an implementation may create `target` in the apply +only when its declared profile proves one native transaction covers schema, +data, catalogs, and audit. MySQL, MariaDB, and Dolt reject create mode with +`schema_change_not_atomic`. + +For Dolt, provision the physical and catalog target together in a separate +caller-owned commit. Begin the transformation from that clean HEAD. The exact +syntax above then binds `COMPUTE` as replacement of the existing target, +leaves an inspectable working-set diff, and does not call `DOLT_COMMIT`. diff --git a/sql/transformation-plan-profile-schema.sql b/sql/transformation-plan-profile-schema.sql index ac7f317..bb761f1 100644 --- a/sql/transformation-plan-profile-schema.sql +++ b/sql/transformation-plan-profile-schema.sql @@ -1,4 +1,4 @@ --- OpenStatSpec In-Place Transformation Binding 0.1 compact operation audit. +-- OpenStatSpec In-Place Transformation Bindings 0.1 and 0.2 compact operation audit. -- This table is not a dataset-version catalog and stores no row data or copies. CREATE TABLE transformation_apply ( @@ -19,7 +19,10 @@ CREATE TABLE transformation_apply ( operation_count INTEGER NOT NULL, started_at TIMESTAMP NOT NULL, completed_at TIMESTAMP NOT NULL, - CHECK (contract_id = 'openstatspec-in-place-transformation-v0.1'), + CHECK (contract_id IN ( + 'openstatspec-in-place-transformation-v0.1', + 'openstatspec-in-place-transformation-v0.2' + )), CHECK (database_profile IN ('sqlite', 'postgresql', 'mysql', 'mariadb', 'dolt')), CHECK (status IN ('succeeded', 'failed')), CHECK (operation_count > 0), diff --git a/tools/validate_repository.py b/tools/validate_repository.py index 44f1345..2d0f0bc 100644 --- a/tools/validate_repository.py +++ b/tools/validate_repository.py @@ -924,9 +924,6 @@ def main() -> None: require(f"CREATE TABLE {table} (" in schema, f"Schema table is missing: {table}") validate_transformation_profile() - from validate_transformation_plan import validate_all as validate_transformation_plan - - validate_transformation_plan() validate_dialect_baseline() json.loads(MANIFEST.read_text(encoding="utf-8")) diff --git a/tools/validate_transformation_plan.py b/tools/validate_transformation_plan.py deleted file mode 100644 index 405306b..0000000 --- a/tools/validate_transformation_plan.py +++ /dev/null @@ -1,465 +0,0 @@ -"""Validate Transformation Plan 0.1 and its SPSS-like frontend fixtures.""" - -from __future__ import annotations - -import hashlib -import json -import struct -from pathlib import Path - -from validate_repository import canonical_hash, require, require_string - - -ROOT = Path(__file__).resolve().parents[1] -PLAN_MANIFEST = ROOT / "conformance/transformation-plan-0.1.json" -FRONTEND_MANIFEST = ROOT / "conformance/spss-syntax-frontend-0.1.json" -PLAN_SCHEMA = ROOT / "transformation/plan-0.1.schema.json" -FRONTEND_SCHEMA = ROOT / "transformation/spss-syntax-frontend-0.1.schema.json" -IN_PLACE_MANIFEST = ROOT / "conformance/in-place-transformation-0.1.json" - -IN_PLACE_SUCCESS_IDS = { - "mysql-recode-and-labels-preserve-dataset-and-table-identity", - "dolt-recode-and-labels-preserve-controlled-context", -} -IN_PLACE_EXPECTED_ERRORS = { - "reject-dolt-branch-mismatch": "dolt_branch_mismatch", - "reject-dolt-head-mismatch": "dolt_head_mismatch", - "reject-dolt-dirty-working-set": "dolt_working_set_dirty", - "reject-mysql-nontransactional-create-target": "schema_change_not_atomic", -} -IN_PLACE_FORBIDDEN_ARTIFACTS = { - "derived_dataset_row", - "persistent_output_table", - "full_table_copy", - "staging_dataset", - "staging_relation", - "snapshot_table", - "rollback_table", - "dataset_version_row", - "retirement_or_recovery_row", - "temporary_object_residue", -} -IN_PLACE_AUDIT_FIELDS = { - "apply_id", - "contract_id", - "database_profile", - "dataset_id", - "physical_table_schema", - "physical_table_name", - "plan_hash", - "source_hash", - "actor", - "status", - "dolt_branch", - "dolt_head_before", - "dolt_head_after", - "operation_count", - "started_at", - "completed_at", -} - - -def binary64_value(value: object, context: str) -> tuple[str, str]: - require(isinstance(value, dict), f"{context}: typed value must be an object.") - require(set(value) in ({"type", "bits"}, {"type", "value"}), f"{context}: typed value fields are invalid.") - kind = value.get("type") - if kind == "binary64": - bits = value.get("bits") - require( - isinstance(bits, str) - and len(bits) == 16 - and all(character in "0123456789abcdef" for character in bits), - f"{context}: binary64 bits must be 16 lowercase hex digits.", - ) - raw = int(bits, 16) - require((raw >> 52) & 0x7FF != 0x7FF, f"{context}: non-finite binary64 is forbidden.") - require(bits != "8000000000000000", f"{context}: negative zero must canonicalize to positive zero.") - return kind, bits - require(kind == "string" and isinstance(value.get("value"), str), f"{context}: typed value kind is invalid.") - return "string", value["value"] - - -def binary64_number(bits: str) -> float: - return struct.unpack(">d", bytes.fromhex(bits))[0] - - -def validate_result(result: object, context: str) -> set[str]: - require(isinstance(result, dict), f"{context}: result must be an object.") - kind = result.get("kind") - if kind == "literal": - require(set(result) == {"kind", "value"}, f"{context}: literal result fields are invalid.") - return {binary64_value(result["value"], context + ".value")[0]} - require(kind in {"system_missing", "copy"} and set(result) == {"kind"}, f"{context}: result is invalid.") - return {kind} - - -def validate_plan(plan: object, context: str) -> str | None: - require(isinstance(plan, dict) and set(plan) == {"contract", "input_alias", "operations"}, f"{context}: plan fields are invalid.") - require(plan["contract"] == "openstatspec-transformation-plan-v0.1", f"{context}: contract is invalid.") - require_string(plan["input_alias"], context + ".input_alias") - operations = plan["operations"] - require(isinstance(operations, list) and operations, f"{context}: operations are missing.") - semantic_error: str | None = None - for operation_index, operation in enumerate(operations): - op_context = f"{context}.operations[{operation_index}]" - require(isinstance(operation, dict), f"{op_context}: operation must be an object.") - op = operation.get("op") - if op == "recode": - require( - set(operation) == {"op", "source", "target", "target_mode", "rules", "unmatched"}, - f"{op_context}: recode fields are invalid.", - ) - source = require_string(operation["source"], op_context + ".source") - target = require_string(operation["target"], op_context + ".target") - require(not target.startswith("__"), f"{op_context}: reserved target name.") - mode = operation["target_mode"] - require(mode in {"create", "replace"}, f"{op_context}: target mode is invalid.") - require(mode != "replace" or source == target, f"{op_context}: replace target must equal source.") - rules = operation["rules"] - require(isinstance(rules, list) and rules, f"{op_context}: rules are missing.") - result_kinds: set[str] = set() - for rule_index, rule in enumerate(rules): - rule_context = f"{op_context}.rules[{rule_index}]" - require(isinstance(rule, dict) and set(rule) == {"match", "result"}, f"{rule_context}: fields are invalid.") - match = rule["match"] - require(isinstance(match, dict), f"{rule_context}.match must be an object.") - match_kind = match.get("kind") - if match_kind == "values": - require(set(match) == {"kind", "values"}, f"{rule_context}.match fields are invalid.") - require(isinstance(match["values"], list) and match["values"], f"{rule_context}: values are missing.") - for value_index, value in enumerate(match["values"]): - binary64_value(value, f"{rule_context}.match.values[{value_index}]") - elif match_kind == "range": - require(set(match) == {"kind", "lower", "upper"}, f"{rule_context}.match fields are invalid.") - lower_kind, lower_bits = binary64_value(match["lower"], rule_context + ".match.lower") - upper_kind, upper_bits = binary64_value(match["upper"], rule_context + ".match.upper") - require(lower_kind == upper_kind == "binary64", f"{rule_context}: range endpoints must be binary64.") - if binary64_number(lower_bits) > binary64_number(upper_bits): - semantic_error = semantic_error or "invalid_numeric_range" - else: - require(match_kind == "system_missing" and set(match) == {"kind"}, f"{rule_context}.match is invalid.") - result_kinds.update(validate_result(rule["result"], rule_context + ".result")) - result_kinds.update(validate_result(operation["unmatched"], op_context + ".unmatched")) - literal_kinds = result_kinds & {"binary64", "string"} - if len(literal_kinds) > 1: - semantic_error = semantic_error or "mixed_result_types" - elif op == "set_variable_label": - require(set(operation) == {"op", "variable", "label"}, f"{op_context}: variable-label fields are invalid.") - require_string(operation["variable"], op_context + ".variable") - require(isinstance(operation["label"], str), f"{op_context}.label must be a string.") - elif op == "replace_value_labels": - require(set(operation) == {"op", "variable", "labels"}, f"{op_context}: value-label fields are invalid.") - require_string(operation["variable"], op_context + ".variable") - labels = operation["labels"] - require(isinstance(labels, list) and labels, f"{op_context}: labels are missing.") - seen: set[tuple[str, str]] = set() - for label_index, label in enumerate(labels): - label_context = f"{op_context}.labels[{label_index}]" - require(isinstance(label, dict) and set(label) == {"value", "label"}, f"{label_context}: fields are invalid.") - key = binary64_value(label["value"], label_context + ".value") - if key in seen: - semantic_error = semantic_error or "duplicate_value_label" - seen.add(key) - require(isinstance(label["label"], str), f"{label_context}.label must be a string.") - else: - require(False, f"{op_context}: unknown operation.") - return semantic_error - - -def validate_plan_manifest() -> dict[str, dict[str, object]]: - manifest = json.loads(PLAN_MANIFEST.read_text(encoding="utf-8")) - require( - isinstance(manifest, dict) - and set(manifest) == {"manifest_version", "profile", "contract", "schema", "cases"}, - "Transformation Plan manifest fields are invalid.", - ) - require(manifest["manifest_version"] == "0.1", "Unexpected Transformation Plan manifest version.") - require(manifest["profile"] == "OpenStatSpec Transformation Plan 0.1", "Unexpected Transformation Plan profile.") - require(manifest["contract"] == "openstatspec-transformation-plan-v0.1", "Unexpected Transformation Plan contract.") - require(manifest["schema"] == "../transformation/plan-0.1.schema.json", "Unexpected Transformation Plan schema path.") - expected_cases = { - "numeric-recode-and-declared-labels": None, - "string-value-label-replacement": None, - "reject-descending-range": "invalid_numeric_range", - "reject-duplicate-value-label": "duplicate_value_label", - } - case_map: dict[str, dict[str, object]] = {} - for case in manifest["cases"]: - require(isinstance(case, dict) and set(case) == {"id", "plan", "expected_plan_hash", "expected_error"}, "Transformation Plan case fields are invalid.") - identifier = require_string(case["id"], "Transformation Plan case id") - require(identifier not in case_map, f"Duplicate Transformation Plan case: {identifier}") - case_map[identifier] = case - error = validate_plan(case["plan"], identifier) - require(error == case["expected_error"] == expected_cases.get(identifier), f"{identifier}: semantic result differs.") - if error is None: - require(case["expected_plan_hash"] == canonical_hash(case["plan"]), f"{identifier}: plan hash differs.") - else: - require(case["expected_plan_hash"] is None, f"{identifier}: invalid plan must not claim a hash.") - require(set(case_map) == set(expected_cases), "Transformation Plan conformance case set is incomplete.") - return case_map - - -def validate_frontend_manifest(plan_cases: dict[str, dict[str, object]]) -> int: - manifest = json.loads(FRONTEND_MANIFEST.read_text(encoding="utf-8")) - require( - isinstance(manifest, dict) - and set(manifest) - == {"manifest_version", "profile", "contract", "plan_contract", "request_schema", "plan_schema", "cases"}, - "SPSS frontend manifest fields are invalid.", - ) - require(manifest["manifest_version"] == "0.1", "Unexpected SPSS frontend manifest version.") - require(manifest["profile"] == "OpenStatSpec SPSS-like Syntax Frontend 0.1", "Unexpected SPSS frontend profile.") - require(manifest["contract"] == "openstatspec-spss-syntax-frontend-v0.1", "Unexpected SPSS frontend contract.") - require(manifest["plan_contract"] == "openstatspec-transformation-plan-v0.1", "Unexpected frontend plan contract.") - expected_errors = { - "recode-labels-and-value-labels": None, - "in-place-recode-default-copy": None, - "string-value-labels": None, - "recode-varlists-positionally": None, - "value-label-varlists-and-groups": None, - "reject-unknown-command": "unsupported_spss_command", - "reject-unknown-variable": "unknown_variable", - "reject-nonfinal-else": "else_not_last", - "reject-string-system-missing": "system_missing_for_string", - "reject-duplicate-value-label": "duplicate_value_label", - "reject-string-target-without-declaration": "string_target_requires_declaration", - "reject-comment-command": "unsupported_spss_command", - "reject-inline-comment": "spss_syntax_error", - } - identifiers: set[str] = set() - for case in manifest["cases"]: - require(isinstance(case, dict), "SPSS frontend case must be an object.") - identifier = require_string(case.get("id"), "SPSS frontend case id") - require(identifier not in identifiers, f"Duplicate SPSS frontend case: {identifier}") - identifiers.add(identifier) - request = case.get("request") - require(isinstance(request, dict) and set(request) == {"contract", "input_alias", "input_schema", "source_text"}, f"{identifier}: request fields are invalid.") - require(request["contract"] == manifest["contract"], f"{identifier}: request contract differs.") - source = require_string(request["source_text"], f"{identifier}.source_text") - require(source == source.replace("\r\n", "\n").replace("\r", "\n"), f"{identifier}: source is not LF-normalized.") - source_hash = hashlib.sha256(source.encode("utf-8")).hexdigest() - require(case.get("expected_source_hash") == source_hash, f"{identifier}: source hash differs.") - require(case.get("expected_error") == expected_errors.get(identifier), f"{identifier}: expected diagnostic differs.") - if case["expected_error"] is None: - if "expected_plan_case" in case: - allowed = {"id", "request", "expected_plan_case", "expected_source_hash", "expected_error"} - require(set(case) == allowed | ({"expected_output_metadata"} if "expected_output_metadata" in case else set()), f"{identifier}: fields are invalid.") - plan_case = plan_cases.get(case["expected_plan_case"]) - require(plan_case is not None and plan_case["expected_error"] is None, f"{identifier}: referenced plan case is invalid.") - else: - allowed = {"id", "request", "expected_plan", "expected_plan_hash", "expected_source_hash", "expected_error"} - require(set(case) == allowed | ({"expected_output_metadata"} if "expected_output_metadata" in case else set()), f"{identifier}: fields are invalid.") - require(validate_plan(case["expected_plan"], identifier + ".expected_plan") is None, f"{identifier}: expected plan is invalid.") - require(case["expected_plan_hash"] == canonical_hash(case["expected_plan"]), f"{identifier}: expected plan hash differs.") - if identifier == "in-place-recode-default-copy": - require(case.get("expected_output_metadata") == { - "q1": { - "variable_label": "Original score", - "value_labels": [{ - "value": {"type": "binary64", "bits": "3ff0000000000000"}, - "label": "Original one", - }], - }, - }, f"{identifier}: metadata preservation proof differs.") - else: - require(set(case) == {"id", "request", "expected_source_hash", "expected_error"}, f"{identifier}: failure fields are invalid.") - require(identifiers == set(expected_errors), "SPSS frontend conformance case set is incomplete.") - return len(identifiers) - - -def validate_in_place_failure_case(case: dict[str, object], identifier: str) -> None: - if identifier == "reject-mysql-nontransactional-create-target": - require( - set(case) - == {"id", "database_profile", "source_text", "expected_error", - "mutation_started"}, - f"{identifier}: failure fields are invalid.", - ) - require(case["database_profile"] == "mysql", f"{identifier}: profile differs.") - require(" INTO " in case["source_text"].upper(), f"{identifier}: create target is missing.") - require( - case["expected_error"] == IN_PLACE_EXPECTED_ERRORS[identifier], - f"{identifier}: diagnostic differs.", - ) - require(case["mutation_started"] is False, f"{identifier}: mutation starts.") - return - require( - set(case) - == {"id", "database_profile", "expected_context", "observed_context", - "expected_error", "mutation_started"}, - f"{identifier}: failure fields are invalid.", - ) - require(case["database_profile"] == "dolt", f"{identifier}: failure must use Dolt.") - expected, observed = case["expected_context"], case["observed_context"] - require( - isinstance(expected, dict) - and set(expected) == {"branch", "head"} - and isinstance(observed, dict) - and set(observed) == {"branch", "head", "working_set_clean"}, - f"{identifier}: controlled context is invalid.", - ) - require( - case["expected_error"] == IN_PLACE_EXPECTED_ERRORS[identifier], - f"{identifier}: diagnostic differs.", - ) - require(case["mutation_started"] is False, f"{identifier}: mutation starts.") - if identifier == "reject-dolt-branch-mismatch": - valid = ( - observed["branch"] != expected["branch"] - and observed["head"] == expected["head"] - and observed["working_set_clean"] is True - ) - elif identifier == "reject-dolt-head-mismatch": - valid = ( - observed["branch"] == expected["branch"] - and observed["head"] != expected["head"] - and observed["working_set_clean"] is True - ) - else: - valid = ( - observed["branch"] == expected["branch"] - and observed["head"] == expected["head"] - and observed["working_set_clean"] is False - ) - require(valid, f"{identifier}: fixture does not isolate its declared failure.") - - -def validate_in_place_success_case(case: dict[str, object], identifier: str) -> None: - require( - set(case) - == {"id", "database_profile", "source_text", "before", "after", - "forbidden_artifacts", "required_audit_fields"}, - f"{identifier}: success fields are invalid.", - ) - before, after = case["before"], case["after"] - require( - isinstance(before, dict) and isinstance(after, dict), - f"{identifier}: identity snapshots are invalid.", - ) - for field in ("dataset_id", "physical_table_schema", "physical_table_name"): - require(before[field] == after[field], f"{identifier}: {field} changes.") - require( - before["dataset_count"] == after["dataset_count"], - f"{identifier}: dataset count grows.", - ) - require( - before["persistent_data_table_count"] == after["persistent_data_table_count"], - f"{identifier}: persistent data-table count grows.", - ) - require( - set(case["forbidden_artifacts"]) == IN_PLACE_FORBIDDEN_ARTIFACTS, - f"{identifier}: forbidden-artifact proof is incomplete.", - ) - require( - set(case["required_audit_fields"]) == IN_PLACE_AUDIT_FIELDS, - f"{identifier}: compact audit fields are incomplete.", - ) - require( - " INTO " not in case["source_text"].upper(), - f"{identifier}: implicit-commit success creates a target.", - ) - require( - after.get("same_table_recode_target") == "score" - and "same_table_has_column" not in after, - f"{identifier}: success does not prove an existing-target recode.", - ) - if case["database_profile"] == "mysql": - require( - not any(key.startswith("dolt_") for key in before | after), - f"{identifier}: non-Dolt case contains Dolt context.", - ) - else: - require(case["database_profile"] == "dolt", f"{identifier}: unexpected profile.") - require( - before["dolt_branch"] == after["dolt_branch"] - and before["dolt_head"] == after["dolt_head"], - f"{identifier}: Dolt branch/HEAD changes.", - ) - require(after["dolt_commit_performed"] is False, f"{identifier}: Dolt commit.") - - -def validate_in_place_manifest() -> None: - manifest = json.loads(IN_PLACE_MANIFEST.read_text(encoding="utf-8")) - require( - isinstance(manifest, dict) - and set(manifest) == {"manifest_version", "profile", "contract", "cases"}, - "In-place manifest fields are invalid.", - ) - require(manifest["manifest_version"] == "0.1", "Unexpected in-place manifest version.") - require( - manifest["profile"] == "OpenStatSpec In-Place Transformation Binding 0.1", - "Unexpected in-place profile.", - ) - require( - manifest["contract"] == "openstatspec-in-place-transformation-v0.1", - "Unexpected in-place contract.", - ) - cases = manifest["cases"] - require(isinstance(cases, list), "In-place cases must be an array.") - case_map = {require_string(case.get("id"), "In-place case id"): case for case in cases} - require(len(case_map) == len(cases), "Duplicate in-place case.") - require( - set(case_map) == IN_PLACE_SUCCESS_IDS | set(IN_PLACE_EXPECTED_ERRORS), - "In-place conformance case set is incomplete.", - ) - for identifier in IN_PLACE_SUCCESS_IDS: - validate_in_place_success_case(case_map[identifier], identifier) - for identifier in IN_PLACE_EXPECTED_ERRORS: - validate_in_place_failure_case(case_map[identifier], identifier) - - -def validate_repository_links() -> None: - plan_schema = json.loads(PLAN_SCHEMA.read_text(encoding="utf-8")) - frontend_schema = json.loads(FRONTEND_SCHEMA.read_text(encoding="utf-8")) - require(plan_schema["$id"].endswith("transformation-plan-0.1.schema.json"), "Transformation Plan schema ID is invalid.") - require(frontend_schema["$id"].endswith("spss-syntax-frontend-0.1.schema.json"), "SPSS frontend schema ID is invalid.") - plan_sql = (ROOT / "sql/transformation-plan-profile-schema.sql").read_text(encoding="utf-8") - require("CREATE TABLE transformation_apply (" in plan_sql, "Compact apply audit table is missing.") - require( - "openstatspec-in-place-transformation-v0.1" in plan_sql - and "openstatspec-dolt-in-place-transformation-v0.1" not in plan_sql, - "Compact apply audit does not use the generic in-place contract.", - ) - for field in ("database_profile", "physical_table_schema", "physical_table_name"): - require(field in plan_sql, f"Compact apply audit field is missing: {field}") - for forbidden in ( - "transformation_plan_version", - "transformation_plan_compilation", - "transformation_plan_run_binding", - "transformation_run_artifact", - ): - require(forbidden not in plan_sql, f"Forbidden history/copy audit relation remains: {forbidden}") - binding = (ROOT / "docs/transformation-plan-sql-binding-0.1.md").read_text(encoding="utf-8") - binding_normalized = " ".join(binding.split()) - for phrase in ( - "dataset_id`, physical schema, and physical table name remain unchanged", - "MUST NOT publish a derived dataset", - "does not version successful data states", - "MUST NOT call `DOLT_COMMIT`", - "clean Dolt working set", - "For a non-Dolt profile every Dolt-specific field is NULL", - "MySQL-family implicit-commit DDL", - "schema_change_not_atomic", - ): - require(phrase in binding_normalized, f"In-place binding requirement is missing: {phrase}") - - validate_in_place_manifest() - - -def validate_all() -> tuple[int, int]: - plan_cases = validate_plan_manifest() - frontend_count = validate_frontend_manifest(plan_cases) - validate_repository_links() - return len(plan_cases), frontend_count - - -def main() -> None: - plan_count, frontend_count = validate_all() - print( - f"Validated {plan_count} Transformation Plan and " - f"{frontend_count} SPSS frontend conformance cases." - ) - - -if __name__ == "__main__": - main() diff --git a/transformation/plan-0.2.schema.json b/transformation/plan-0.2.schema.json new file mode 100644 index 0000000..7f2b185 --- /dev/null +++ b/transformation/plan-0.2.schema.json @@ -0,0 +1,139 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://openstatspec.org/schemas/transformation-plan-0.2.schema.json", + "title": "OpenStatSpec Transformation Plan 0.2", + "type": "object", + "additionalProperties": false, + "required": ["contract", "input_alias", "operations"], + "properties": { + "contract": {"const": "openstatspec-transformation-plan-v0.2"}, + "input_alias": {"type": "string", "minLength": 1}, + "operations": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/operation"}} + }, + "$defs": { + "binary64_value": { + "type": "object", "additionalProperties": false, "required": ["type", "bits"], + "properties": { + "type": {"const": "binary64"}, + "bits": {"type": "string", "pattern": "^(?!8000000000000000$)(?!(?:7ff|fff)[0-9a-f]{13}$)[0-9a-f]{16}$"} + } + }, + "string_value": { + "type": "object", "additionalProperties": false, "required": ["type", "value"], + "properties": {"type": {"const": "string"}, "value": {"type": "string"}} + }, + "typed_value": {"oneOf": [{"$ref": "#/$defs/binary64_value"}, {"$ref": "#/$defs/string_value"}]}, + "literal": { + "type": "object", "additionalProperties": false, "required": ["kind", "value"], + "properties": {"kind": {"const": "literal"}, "value": {"$ref": "#/$defs/typed_value"}} + }, + "system_missing": { + "type": "object", "additionalProperties": false, "required": ["kind"], + "properties": {"kind": {"const": "system_missing"}} + }, + "copy": { + "type": "object", "additionalProperties": false, "required": ["kind"], + "properties": {"kind": {"const": "copy"}} + }, + "result": {"oneOf": [{"$ref": "#/$defs/literal"}, {"$ref": "#/$defs/system_missing"}, {"$ref": "#/$defs/copy"}]}, + "variable_operand": { + "type": "object", "additionalProperties": false, "required": ["kind", "variable"], + "properties": {"kind": {"const": "variable"}, "variable": {"type": "string", "minLength": 1}} + }, + "operand": {"oneOf": [{"$ref": "#/$defs/variable_operand"}, {"$ref": "#/$defs/literal"}]}, + "comparison": { + "type": "object", "additionalProperties": false, + "required": ["expression", "left", "operator", "right"], + "properties": { + "expression": {"const": "comparison"}, + "left": {"$ref": "#/$defs/operand"}, + "operator": {"enum": ["=", "<", "<=", ">", ">="]}, + "right": {"$ref": "#/$defs/operand"} + } + }, + "boolean": { + "type": "object", "additionalProperties": false, + "required": ["expression", "operator", "operands"], + "properties": { + "expression": {"const": "boolean"}, + "operator": {"enum": ["and", "or"]}, + "operands": {"type": "array", "minItems": 2, "items": {"$ref": "#/$defs/predicate"}} + } + }, + "predicate": {"oneOf": [{"$ref": "#/$defs/comparison"}, {"$ref": "#/$defs/boolean"}]}, + "match": { + "oneOf": [ + {"type": "object", "additionalProperties": false, "required": ["kind", "values"], "properties": {"kind": {"const": "values"}, "values": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/typed_value"}}}}, + {"type": "object", "additionalProperties": false, "required": ["kind", "lower", "upper"], "properties": {"kind": {"const": "range"}, "lower": {"$ref": "#/$defs/binary64_value"}, "upper": {"$ref": "#/$defs/binary64_value"}}}, + {"$ref": "#/$defs/system_missing"} + ] + }, + "rule": { + "type": "object", "additionalProperties": false, "required": ["match", "result"], + "properties": {"match": {"$ref": "#/$defs/match"}, "result": {"$ref": "#/$defs/result"}} + }, + "recode": { + "type": "object", "additionalProperties": false, + "required": ["op", "source", "target", "target_mode", "rules", "unmatched"], + "properties": { + "op": {"const": "recode"}, "source": {"type": "string", "minLength": 1}, + "target": {"type": "string", "minLength": 1}, "target_mode": {"enum": ["create", "replace"]}, + "rules": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/rule"}}, + "unmatched": {"$ref": "#/$defs/result"} + } + }, + "assign": { + "type": "object", "additionalProperties": false, "required": ["op", "target", "target_mode", "value"], + "properties": { + "op": {"const": "assign"}, "target": {"type": "string", "minLength": 1}, + "target_mode": {"enum": ["create", "replace"]}, "value": {"$ref": "#/$defs/operand"} + } + }, + "conditional_assign": { + "type": "object", "additionalProperties": false, "required": ["op", "condition", "target", "value"], + "properties": { + "op": {"const": "conditional_assign"}, "condition": {"$ref": "#/$defs/predicate"}, + "target": {"type": "string", "minLength": 1}, "value": {"$ref": "#/$defs/operand"} + } + }, + "set_variable_label": { + "type": "object", "additionalProperties": false, "required": ["op", "variable", "label"], + "properties": {"op": {"const": "set_variable_label"}, "variable": {"type": "string", "minLength": 1}, "label": {"type": "string"}} + }, + "value_label": { + "type": "object", "additionalProperties": false, "required": ["value", "label"], + "properties": {"value": {"$ref": "#/$defs/typed_value"}, "label": {"type": "string"}} + }, + "replace_value_labels": { + "type": "object", "additionalProperties": false, "required": ["op", "variable", "labels"], + "properties": { + "op": {"const": "replace_value_labels"}, "variable": {"type": "string", "minLength": 1}, + "labels": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/value_label"}} + } + }, + "set_format": { + "type": "object", "additionalProperties": false, + "required": ["op", "variable", "family", "width", "decimals"], + "properties": { + "op": {"const": "set_format"}, "variable": {"type": "string", "minLength": 1}, + "family": {"const": "F"}, "width": {"type": "integer", "minimum": 1, "maximum": 40}, + "decimals": {"type": "integer", "minimum": 0, "maximum": 16} + } + }, + "set_measurement_level": { + "type": "object", "additionalProperties": false, "required": ["op", "variable", "level"], + "properties": {"op": {"const": "set_measurement_level"}, "variable": {"type": "string", "minLength": 1}, "level": {"enum": ["nominal", "ordinal", "scale"]}} + }, + "execute": { + "type": "object", "additionalProperties": false, "required": ["op"], + "properties": {"op": {"const": "execute"}} + }, + "operation": { + "oneOf": [ + {"$ref": "#/$defs/recode"}, {"$ref": "#/$defs/assign"}, {"$ref": "#/$defs/conditional_assign"}, + {"$ref": "#/$defs/set_variable_label"}, {"$ref": "#/$defs/replace_value_labels"}, + {"$ref": "#/$defs/set_format"}, {"$ref": "#/$defs/set_measurement_level"}, {"$ref": "#/$defs/execute"} + ] + } + } +} diff --git a/transformation/spss-syntax-frontend-0.2.schema.json b/transformation/spss-syntax-frontend-0.2.schema.json new file mode 100644 index 0000000..3e6d3f3 --- /dev/null +++ b/transformation/spss-syntax-frontend-0.2.schema.json @@ -0,0 +1,57 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://openstatspec.org/schemas/spss-syntax-frontend-0.2.schema.json", + "title": "OpenStatSpec SPSS-like Syntax Frontend Request 0.2", + "type": "object", + "additionalProperties": false, + "required": ["contract", "input_alias", "input_schema", "source_text"], + "properties": { + "contract": {"const": "openstatspec-spss-syntax-frontend-v0.2"}, + "input_alias": {"type": "string", "minLength": 1}, + "input_schema": { + "type": "object", "additionalProperties": false, "required": ["variables"], + "properties": { + "variables": { + "type": "array", "minItems": 1, + "items": { + "type": "object", "additionalProperties": false, "required": ["name", "storage_kind"], + "properties": { + "name": {"type": "string", "minLength": 1}, + "storage_kind": {"enum": ["numeric", "string"]}, + "variable_label": {"type": ["string", "null"]}, + "value_labels": {"type": "array", "items": {"$ref": "#/$defs/value_label"}}, + "format_family": {"type": ["string", "null"]}, + "width": {"type": ["integer", "null"], "minimum": 1}, + "decimals": {"type": ["integer", "null"], "minimum": 0}, + "measurement_level": {"enum": ["nominal", "ordinal", "scale", null]} + } + } + } + } + }, + "source_text": {"type": "string", "minLength": 1} + }, + "$defs": { + "binary64_value": { + "type": "object", "additionalProperties": false, "required": ["type", "bits"], + "properties": { + "type": {"const": "binary64"}, + "bits": {"type": "string", "pattern": "^(?!8000000000000000$)(?!(?:7ff|fff)[0-9a-f]{13}$)[0-9a-f]{16}$"} + } + }, + "string_value": { + "type": "object", "additionalProperties": false, "required": ["type", "value"], + "properties": {"type": {"const": "string"}, "value": {"type": "string"}} + }, + "typed_value": {"oneOf": [{"$ref": "#/$defs/binary64_value"}, {"$ref": "#/$defs/string_value"}]}, + "value_label": { + "type": "object", + "additionalProperties": false, + "required": ["value", "label"], + "properties": { + "value": {"$ref": "#/$defs/typed_value"}, + "label": {"type": "string"} + } + } + } +}