Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ jobs:
uses: actions/checkout@v7
with: &specification-checkout
repository: OpenStatSpec/specification
ref: 79339ec3d8f8aa81789b7e85f6b8afa6f1374e50
ref: e49252c00890aed76dcaabc5d1ab5121b45929db
path: openstatspec-specification
- name: Checkout required SPSS engine
uses: actions/checkout@v7
Expand Down
34 changes: 34 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,40 @@

All notable changes to this reference implementation are documented here.

## Unreleased

Planned adapter release: `0.5.0`, after lifecycle integration and final specification conformance.

### Added

- Added bounded typed expressions with variable references, numeric literals,
parentheses, `=`, `<`, `<=`, `>`, `>=`, `AND`, and `OR`. String comparison
and v0.2 string assignment fail closed pending exact portable semantics.
- Added sequential SPSS-like `COMPUTE` and `IF` assignment operations plus
`FORMATS`, `VARIABLE LEVEL`, and `EXECUTE`.
- Added atomic numeric target creation on SQLite and PostgreSQL. MySQL,
MariaDB, and Dolt fail closed on `target_mode=create`; their targets must be
provisioned physically and in the catalog by a separate versioned stage
before this executor applies assignment and metadata operations.
- Added synthetic exact-program, catalog, failure-boundary, and pre-existing
target regression coverage.

### Changed

- Bumped the canonical transformation-plan and SPSS frontend contracts to
`v0.2`; canonical JSON and hashes include every sequential operation.
- In-place apply now records variable label, value labels, `F` print/write
format, and measurement level in both normative and compatibility catalogs.
- Dolt still requires an exact branch, exact HEAD, and clean working set;
successful apply leaves an inspectable diff and never calls `DOLT_COMMIT`.

### Specification basis

- Release validation is pinned to the untagged OpenStatSpec specification
release candidate at immutable commit
`e49252c00890aed76dcaabc5d1ab5121b45929db`. Its
`specification_release` remains null until that commit receives a stable tag.

## 0.4.0 — 2026-07-31

### Added
Expand Down
19 changes: 13 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,12 +67,19 @@ implemented capability boundary.

## SPSS-like transformation frontend

The SPSS-like frontend lowers supported `RECODE`, `VARIABLE LABELS`, and
`VALUE LABELS` syntax into a language-neutral canonical plan. The in-place
path applies it to the same logical dataset, physical wide table, and metadata
catalog without a derived dataset, copied table, snapshot, or separate
rollback/history layer. Dolt remains the sole versioning layer for Dolt-backed
edits, and the transformer never calls `DOLT_COMMIT`.
The bounded SPSS-like frontend lowers `RECODE`, sequential `COMPUTE` and `IF`,
`VARIABLE LABELS`, `VALUE LABELS`, numeric `FORMATS`, `VARIABLE LEVEL`, and
`EXECUTE` into a language-neutral typed canonical plan. Conditions support
parentheses, variable and numeric literal operands, comparisons, `AND`, and
`OR`. String comparison and v0.2 string assignment currently fail closed.
The in-place path applies the plan to the same logical dataset, physical wide
table, and normative/compatibility metadata without a derived dataset, copied
table, snapshot, or hidden history layer. Numeric targets may be created
atomically on SQLite and PostgreSQL. MySQL, MariaDB, and Dolt fail closed on
`target_mode=create` and require a separately provisioned physical and catalog
target before assignment. Dolt requires the caller's exact clean branch/HEAD,
leaves success as an inspectable working-set diff, and never calls
`DOLT_COMMIT`.

See the [dataset transformations manual](docs/transformations.md) for schema
installation, Python and CLI surfaces, database invariants, audit provenance,
Expand Down
29 changes: 23 additions & 6 deletions docs/release-readiness.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# 0.4.0 release readiness
# 0.5.0 release readiness

This page records the expected release contract, not a publication event.
Creating a version tag remains a separate maintainer action.
Expand Down Expand Up @@ -67,6 +67,17 @@ The gate must prove that:

- a TransformationPlan object and its strict JSON mapping produce the same
plan hash and in-place result;
- the exact bounded `COMPUTE`/`IF` program compiles to all seven ordered
operations without dropping `FORMATS`, `VARIABLE LEVEL`, or `EXECUTE`;
- boolean data results match the equivalent expression and the target's label,
0/1 value labels, `F1.0` print/write format, and nominal level exist in both
normative and compatibility catalogs;
- injected schema, data, catalog, and audit failures leave no partial apply;
- compensation tracks only newly created targets and never drops or rewrites a
pre-existing target;
- MySQL, MariaDB, and Dolt reject create-target plans before mutation; their
service evidence covers assignment to a separately provisioned physical and
catalog target without schema DDL;
- top-level SPSS compiler imports and legacy openstatspec.transform re-exports
still load from an installed wheel;
- install-in-place-schema, apply-plan, and apply-spss execute their documented
Expand All @@ -78,8 +89,11 @@ The gate must prove that:
source/plan hashes and frontend contract, and contain no copied data;
- no OpenStatSpec rollback, snapshot, staging, copy, derived-dataset, or
parallel history artifacts are created; and
- Dolt checks expected branch, HEAD, and a clean working set without committing
or changing HEAD; other supported SQL connections remain allowed.
- Dolt checks expected branch, HEAD, and a clean working set; success changes
neither HEAD nor branch and never commits or resets; state is rechecked after
the dataset lock and success must leave an inspectable working-set diff;
other supported SQL connections remain allowed; and
- string comparisons and v0.2 string assignments fail closed until exact

The built wheel must contain the generic openstatspec.transform modules and the
implemented openstatspec.frontends.spss package. Stata and SAS remain empty
Expand All @@ -98,9 +112,12 @@ capability claim, or implied support.
4. Build with `python -m build` and install the generated wheel in a clean
environment.
5. Confirm `openstatspec capabilities` reflects the intended support boundary.
6. Confirm the release tag matches the package version and that CI, release
fixtures, and capabilities use OpenStatSpec specification release `v0.2.0`
at exact commit `79339ec3d8f8aa81789b7e85f6b8afa6f1374e50`.
6. Confirm CI, release fixtures, and capabilities use the untagged OpenStatSpec
specification release candidate at exact commit
`e49252c00890aed76dcaabc5d1ab5121b45929db`, publish
`specification_status=release_candidate`, and leave
`specification_release` null. If that exact commit receives a stable tag
before this package is tagged, update the identity and re-run every gate.
7. Review this document, the README, and CHANGELOG for accurate scope.

The tag-triggered release workflow repeats the non-service test suite, builds
Expand Down
52 changes: 33 additions & 19 deletions docs/transformations.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,13 @@ OpenStatSpec separates transformation syntax, canonical meaning, and database
mutation. This lets multiple language frontends produce the same plan without
coupling the executor to any one language.

The implemented frontend accepts a small SPSS-like subset: `RECODE`,
`VARIABLE LABELS`, and `VALUE LABELS`. Stata and SAS are not implemented.
The implemented bounded SPSS-like frontend accepts `RECODE`, sequential
`COMPUTE` and `IF`, `VARIABLE LABELS`, `VALUE LABELS`, numeric `FORMATS`,
`VARIABLE LEVEL`, and `EXECUTE`. Predicates support typed variable/literal
operands, parentheses, numeric comparisons, `AND`, and `OR`. String comparison
and v0.2 string assignment fail closed until exact profile-independent
collation and explicit-width semantics are available; arbitrary SPSS, Python,
and SQL expressions are rejected. Stata and SAS are not implemented.

## Architecture

Expand Down Expand Up @@ -106,9 +111,13 @@ result = openstatspec.apply_spss_in_place(
dataset_id="responses",
actor="agent@example.org",
source_text="""
RECODE age (18 THRU 34 = 1) (35 THRU 64 = 2).
VARIABLE LABELS age 'Age group'.
VALUE LABELS age 1 '18-34' 2 '35-64'.
COMPUTE target = 0.
IF (source_a = 1 AND source_b = 1) target = 1.
VARIABLE LABELS target 'Example label'.
VALUE LABELS target 0 'No' 1 'Yes'.
FORMATS target (F1.0).
VARIABLE LEVEL target (NOMINAL).
EXECUTE.
""",
)
```
Expand All @@ -130,20 +139,25 @@ On Dolt, also pass `--expected-branch` and `--expected-head`.
Every successful apply preserves the logical `dataset_id` and physical
schema/table identity. It creates no derived dataset, output table, full-table
copy, staging table, snapshot, rollback artifact, or recovery/history layer.
Existing-target recodes use direct `UPDATE`; label operations mutate existing
catalog rows.

SQLite and PostgreSQL may add a numeric target where native transactions make
the complete operation atomic. MySQL, MariaDB, and Dolt reject target-creating
plans before the first mutation because implicit-commit DDL could leave a
partial apply. Their target column and metadata must already exist.

Dolt is the sole history, diff, branch, and rollback layer for Dolt-backed
datasets. Before mutation, the executor verifies the expected branch and
`HEAD` and requires clean `dolt_status`. Success changes the same working set
without changing `HEAD`. OpenStatSpec does not call `DOLT_COMMIT`, switch
branches, merge, reset, tag, or create a hidden recovery commit. The caller
reviews `dolt diff` and separately decides whether to commit or restore.
Assignments and recodes use ordered `UPDATE` statements; later operations see
earlier results. Label, value-label, format, and measurement-level operations
update both the normative and compatibility catalogs.

A numeric create target is supported atomically on SQLite and PostgreSQL.
MySQL, MariaDB, and Dolt reject `target_mode=create` before mutation. On those
profiles a separate versioned stage must first provision the nullable numeric
physical column and both catalog representations; the transformation executor
then sees a pre-existing target and performs no schema DDL.
The public operation reports success only after physical data, both metadata
representations, and the compact audit row are mutually complete.

Before Dolt mutation, the executor verifies the expected branch and `HEAD` and
requires clean `dolt_status`. Success changes the same working set without
changing `HEAD`; OpenStatSpec does not call `DOLT_COMMIT`, `DOLT_RESET`, switch
branches, merge, tag, or create a hidden recovery commit. It rechecks branch,
HEAD, and a clean working set after locking the dataset and immediately before
mutation. The caller reviews a successful `dolt diff` and
separately decides whether to commit or restore.

## Audit and provenance

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "hatchling.build"

[project]
name = "openstatspec"
version = "0.4.0"
version = "0.5.0"
description = "Reference adapter for the OpenStatSpec relational contract"
readme = "README.md"
requires-python = ">=3.11"
Expand Down
11 changes: 9 additions & 2 deletions src/openstatspec/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,17 +14,24 @@
from .sql.workflow import TransformationError
from .frontends.spss import SpssFrontendCompilation, compile_spss_syntax
from .transform import (
AssignOperation, BooleanExpression, ComparisonExpression,
ConditionalAssignOperation, ExecuteOperation, Operand, PredicateExpression,
RecodeMatch, RecodeOperation, RecodeResult, RecodeRule,
ReplaceValueLabelsOperation, SetVariableLabelOperation,
ReplaceValueLabelsOperation, SetFormatOperation,
SetMeasurementLevelOperation, SetVariableLabelOperation,
TransformationFrontendError, TransformationPlan, TypedValue, ValueLabel,
VariableDefinition, VariableSchema, transformation_plan_from_dict,
)

__all__ = [
"AssignOperation", "BooleanExpression", "ComparisonExpression",
"ConditionalAssignOperation", "ExecuteOperation", "Operand",
"PredicateExpression",
"CapabilityDeclaration", "LossReport", "SpssFrontendCompilation",
"TransformationError", "TransformationFrontendError",
"RecodeMatch", "RecodeOperation", "RecodeResult", "RecodeRule",
"ReplaceValueLabelsOperation", "SetVariableLabelOperation",
"ReplaceValueLabelsOperation", "SetFormatOperation",
"SetMeasurementLevelOperation", "SetVariableLabelOperation",
"TransformationPlan", "TypedValue", "ValueLabel",
"VariableDefinition", "VariableSchema", "transformation_plan_from_dict",
"UnsupportedOperationError", "capabilities", "capability_matrix",
Expand Down
10 changes: 7 additions & 3 deletions src/openstatspec/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@
)
from .transform import TransformationPlan
from .sql.capabilities import (
SPECIFICATION_COMMIT, SPECIFICATION_RELEASE, active_connection, catalog_binding,
SPECIFICATION_COMMIT, SPECIFICATION_RELEASE, SPECIFICATION_STATUS,
active_connection, catalog_binding,
)


Expand All @@ -44,7 +45,7 @@ def capability_matrix(database_url: str | None = None) -> Mapping[str, Any]:
"""
declaration = {
"specification": "OpenStatSpec",
"specification_status": "released",
"specification_status": SPECIFICATION_STATUS,
"specification_release": SPECIFICATION_RELEASE,
"specification_commit": SPECIFICATION_COMMIT,
"profile": "SPSS SAV/ZSAV 1.0",
Expand Down Expand Up @@ -145,7 +146,10 @@ def apply_spss_in_place(
actor: str, expected_branch: str | None = None,
expected_head: str | None = None,
) -> Mapping[str, Any]:
"""Apply supported SPSS-like syntax to the same SQL dataset/table."""
"""Apply bounded sequential SPSS syntax to one SQL dataset/table.

Supports typed COMPUTE/IF predicates and dictionary metadata operations.
"""
return result(_apply_spss_in_place(
database_url=str(database_url),
dataset_id=dataset_id,
Expand Down
2 changes: 1 addition & 1 deletion src/openstatspec/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ def main(argv: Sequence[str] | None = None) -> int:

apply_spss = commands.add_parser(
"apply-spss",
help="compile supported SPSS syntax and apply it in-place",
help="compile bounded sequential SPSS syntax and apply it in-place",
)
apply_spss.add_argument("--database-url", required=True)
apply_spss.add_argument("--dataset-id", required=True)
Expand Down
2 changes: 1 addition & 1 deletion src/openstatspec/frontends/spss/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
)


SPSS_FRONTEND_CONTRACT = "openstatspec-spss-syntax-frontend-v0.1"
SPSS_FRONTEND_CONTRACT = "openstatspec-spss-syntax-frontend-v0.2"

__all__ = [
"SPSS_FRONTEND_CONTRACT",
Expand Down
Loading