refactor!: Remove bulk of IO and serialization logic

[Oliver Borchert (borchero)]

Motivation

Implements #367.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (482d77c) to head (2d90ccf).

Additional details and impacted files

@@                 Coverage Diff                 @@
##           remove-to-polars      #379    +/-   ##
===================================================
  Coverage            100.00%   100.00%            
===================================================
  Files                    56        46    -10     
  Lines                  3413      2604   -809     
===================================================
- Hits                   3413      2604   -809     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR implements the direction of #367 by removing most of dataframely’s built-in I/O and schema/collection serialization logic (including metadata roundtrips and delta support), and simplifying persistence to lightweight parquet helpers for Collection and FailureInfo without implicit validation or schema metadata.

Changes:

• Removed schema/collection serialization and metadata-driven read/scan behavior; removed related exceptions and helpers.
• Removed delta / fsspec-based storage backends and associated optional dependencies and tests (including S3/moto fixtures).
• Reworked Collection parquet persistence and FailureInfo parquet persistence to store only minimal metadata (rule-columns) and require explicit validation by users.

Reviewed changes

Copilot reviewed 47 out of 48 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/test_serialization.py	Removed tests for schema JSON decoding/serialization.
tests/test_deprecation.py	Removed deprecation tests tied to deprecated I/O methods slated for removal.
tests/storage/test_fsspec.py	Removed fsspec-specific storage helper tests.
tests/storage/test_delta.py	Removed delta-backend storage tests.
tests/schema/test_storage.py	Removed schema storage/metadata/implicit-validation tests.
tests/schema/test_serialization.py	Removed schema serialization roundtrip and deserialization failure tests.
tests/schema/test_read_write_parquet.py	Removed schema parquet metadata tests and mkdir behavior tests tied to schema I/O.
tests/failure_info/test_storage.py	Updated FailureInfo persistence tests to parquet-only and new metadata expectations.
tests/conftest.py	Removed S3/moto fixtures used for storage backends.
tests/column_types/test_enum.py	Simplified enum test to assert persisted configuration flags directly (no dict roundtrip).
tests/collection/test_storage.py	Updated collection storage tests to parquet-only directory layout and no implicit validation behavior.
tests/collection/test_serialization.py	Removed collection serialization/deserialization tests.
skills/SKILL.md	Removed I/O conventions section that is no longer applicable after I/O removal.
pyproject.toml	Dropped fsspec/s3/deltalake dependencies and removed pytest S3 marker/warning filters.
pixi.toml	Dropped fsspec/s3/moto/boto3/deltalake from pixi environments and optionals.
docs/guides/features/serialization.md	Rewritten to “Persisting data” with explicit “no schema metadata / no implicit validation” guidance.
docs/api/schema/io.rst	Removed schema I/O API docs.
docs/api/schema/index.rst	Removed schema I/O section from schema docs index.
docs/api/filter_result/failure_info.rst	Removed delta I/O entries for FailureInfo.
docs/api/errors/index.rst	Removed ValidationRequiredError from documented errors.
docs/api/collection/io.rst	Removed delta I/O and collection serialization metadata API entries.
dataframely/testing/storage.py	Removed test-only storage abstraction layer for parquet/delta/fsspec backends.
dataframely/schema.py	Removed schema serialization + schema I/O APIs and metadata-driven validation logic; FailureInfo no longer carries schema.
dataframely/filter_result.py	Simplified FailureInfo (no schema), removed delta support, and implemented parquet-only persistence with rule_columns metadata.
dataframely/exc.py	Removed I/O-related exceptions (ValidationRequiredError, DeserializationError).
dataframely/columns/struct.py	Removed dict (de)serialization hooks now that schema serialization is removed.
dataframely/columns/list.py	Removed dict (de)serialization hooks now that schema serialization is removed.
dataframely/columns/enum.py	Updated comment to reflect removal of dict roundtripping concerns.
dataframely/columns/array.py	Removed dict (de)serialization hooks now that schema serialization is removed.
dataframely/columns/_registry.py	Removed column_from_dict deserialization factory.
dataframely/columns/_base.py	Removed as_dict / from_dict and check-expression conversion utilities used for serialization.
dataframely/columns/init.py	Stopped exporting removed column_from_dict.
dataframely/collection/filter_result.py	Updated FailureInfo construction to match new signature (no schema).
dataframely/collection/collection.py	Removed collection serialization and metadata-driven I/O; implemented direct parquet write/read/scan per member file.
dataframely/collection/init.py	Removed exports for removed collection (de)serialization and metadata readers.
dataframely/_typing.py	Removed Validation type alias now that validation-on-read is removed.
dataframely/_storage/parquet.py	Removed parquet storage backend implementation.
dataframely/_storage/delta.py	Removed delta storage backend implementation.
dataframely/_storage/constants.py	Removed metadata-key constants used by storage backends.
dataframely/_storage/_fsspec.py	Removed fsspec file-prefix helper.
dataframely/_storage/_exc.py	Removed storage metadata assertion helper.
dataframely/_storage/_base.py	Removed storage-backend abstraction interface.
dataframely/_storage/init.py	Removed storage package exports.
dataframely/_serialization.py	Removed schema/collection JSON encoder/decoder utilities.
dataframely/_rule.py	Removed rule dict (de)serialization factory utilities.
dataframely/_compat.py	Removed deltalake + partition-sink compatibility shims now that delta I/O is removed.
dataframely/init.py	Removed exports for removed (de)serialization and metadata readers, and removed Validation/DeserializationError exports.

Comments suppressed due to low confidence (1)

dataframely/filter_result.py:208

• Same metadata type-assumption issue as in write_parquet: {**metadata, ...} will raise a TypeError for non-dict metadata values (e.g. None). Validating metadata avoids a confusing failure mode.

        metadata = kwargs.pop("metadata", {})
        self._lf.sink_parquet(
            file,
            metadata={**metadata, "rule_columns": json.dumps(self._rule_columns)},
            **kwargs,