Serializer v8: declare IO specs as literal tuples instead of pickling them

[fumitoh]

Summary

Adds serializer version 8, whose writer declares every BaseIOSpec and its parameters as literal tuples in a central text file (_data/iospecs.py) instead of pickling spec objects into _data/iospecs.pickle. The only pickle left in a saved model is _data/data.pickle (user data values), which keeps the v7 persistent-id shapes (("DataValue", key) / ("BaseIOSpec", key)) resolved against the specs restored from the literal file.

Motivation: iospecs.pickle was the one binary blob in an otherwise text-first, diffable format, and pickled spec metadata has repeatedly broken (pandas 3 removing pc.load_reduce, two class renames, load-bearing import paths). Spec metadata is a handful of literal-friendly fields — pickle was pure downside for it.

The format

# modelx: iospecs
# (key, class, version, path, io_args, spec_args)
(1, 'PandasData', 1, 'files/data.csv', {'file_type': 'csv'}, {'sheet': None, 'is_series': False, 'name': None, 'index_nlevels': 1, 'columns_nlevels': 1})

• Keys are the writer-assigned assign_id(spec) ints — the same ids the data.pickle persistent-id stubs use, so spec identity is explicit and two refs sharing one spec trivially resolve to the same object.
• Ref sites emit ("IOSpec", key) (the v7 value_id element is dropped; values no longer enter data.pickle through the ref path) plus a regenerated readability comment:
  pdref = ("IOSpec", 2) # PandasData path='files/data.csv' file_type='csv' sheet='SheetA'
• Class names resolve through the modelx.io.SPEC_CLASSES catalog, never by import path.

Design

• Spec classes own their literal format through hooks declared on BaseIOSpec (format_version, _on_serialize_args, _on_unserialize_args, _on_comment_args). The serializer owns only the container format. Each entry carries the class's format_version, so spec payloads can evolve without a new serializer version: older versions go through the class's compat handling; newer versions fail that entry per line ("written by a newer version of modelx") before its IO is created.
• PandasData persists shape metadata (sheet, is_series, name, index_nlevels, columns_nlevels) derived from the live value and recomputes read_args on load the way _init_spec does, instead of persisting the raw pandas-API dict. numpy-scalar Series names are coerced to plain Python scalars. ExcelRange persists range/sheet/keyids; ModuleData only its path.
• Unwritable models fail before any output: ModelWriter.validate_model runs ahead of the backup rotation, so a non-literal parameter or an out-of-catalog spec class raises a clean TypeError with the previous save untouched.
• Per-line tolerance with distinct failure classes (unparsable line / invalid-or-duplicate key / future format version / restore failure): the reader warns, records the key as lost (ref sites and data.pickle stubs degrade to None), deletes orphan IOs, and keeps loading. An unreadable iospecs.py loses the specs but not the model, matching the v4-7 behavior for iospecs.pickle.
• The v4-era is_hidden flag is dropped: v8 files carry no such field and a formerly hidden spec reloads as an ordinary one.
• The v8 parser tolerates trailing comments on REFDEFS assignment lines (the shared tokenizer keeps them in the statement token list).
• _get_serializer now fails with a clean ValueError on unknown (newer) serializer versions instead of a ModuleNotFoundError.

Compatibility

• Serializers 1-7 are untouched; all existing fixtures (v4-v7) keep loading. The only core change is adding 8 to the Interface.__reduce__ serializer-version gate.
• Journal/rollback error-path behavior carries over unchanged (verified by the parametrized error-cleanup suites).
• Dependent packages: _on_serialize (modelx-cython's exporter) and _get_attrdict/_on_pickle (spyder-modelx) are untouched.

Determinism

A no-op save → load → save round trip is byte-identical, including the ref-site comments, across processes with different hash seeds. The determinism suite is parametrized to v8 alongside 6/7.

Testing

Full suite: 1218 passed, 6 skipped. New coverage includes round trips for all three spec types, shared-spec identity (two refs; ref + cells input via the DataValue path), hidden-spec migration, v7→v8 resave, missing IO file (dir + zip), malformed/duplicate/future-version literal lines, unknown spec classes, missing/unreadable iospecs.py, pre-write validation, ref-comment byte identity, a clean-error test for unknown future versions, and a frozen model_v8 fixture in the serializer_compat pattern.

The implementation went through four adversarial multi-agent review rounds; all confirmed findings were fixed (per-line tolerance escapes, duplicate-key clobbering, write-path validation ordering, numpy-scalar rejection, a too-strict class-identity write gate).

The design brief with the settled decisions is included as devnotes/IOSpecLiteralTask.md, following the DependentPackages.md precedent.

🤖 Generated with Claude Code