feat(etl): add agent-native observable pipelines

[keli-wen]

Summary

Add an independent, standard-library-only quantmind.etl package for agent-native ETL authoring. The new scaffolds give a coding agent a Run ID and absolute local run.json path before execution, then maintain atomic snapshots and sparse events so the agent can observe stage, real progress, batch position, PID hint, dry-run mode, and terminal state without a CLI or external service.

Expose two explicit parallel execution shapes:

• ETLPipeline executes one whole-run extract → transform → load delivery and returns the load result.
• BatchETLPipeline lazily executes strict serial extract → transform → load cycles for meaningful business batches and returns a bounded BatchRunSummary.

Use composition rather than an ABC or framework hierarchy, keep existing quantmind.flows unchanged, and enforce a two-way architecture boundary through import-linter. Both run shapes require explicit dry_run, execute all three stages, preserve original failures/cancellation, and persist only allowlisted observation data.

Staging is intentionally not a fourth framework stage. Authored extract or transform code may perform idempotent intermediate writes, while load remains the formal delivery boundary; dry-run must suppress both staging and delivery mutations.

The agent-facing launch contract is intentionally machine-readable rather than a free-form For Agent Observe: ... prefix:

run = pipeline.create_run(source, dry_run=False, config_summary={"batch_size": 100})
print(run.receipt(), flush=True)
result = await run.execute()

{"event":"etl_run_created","run_id":"qmr_...","dry_run":false,"status_file":"/absolute/path/.quant-mind/etl-pipeline-runs/qmr_.../run.json"}

A batch receipt uses event="etl_batch_run_created". The caller prints the receipt before execute(), and the next agent parses status_file and reads that local JSON directly.

The change also includes focused network-free whole-run and micro-batch local artifact examples, public documentation, design context, package exports, mirrored contributor guidance, JSON persistence hardening, and adversarial tests for failure, cancellation, invalid Unicode, journal failures, progress scopes, and non-cooperative extractor cleanup.

Related Issue

Closes #147.

Verification

• uv run pytest tests/etl tests/test_contexts.py --no-cov — 67 passed.
• bash scripts/verify.sh — passed: ruff format/check, basedpyright, 9 import-linter contracts, 485 tests, and 86.45% coverage.
• The pre-push hook reran the deterministic verification harness and passed.
• No live-network smoke test applies because the new ETL scaffolds, tests, and examples are local and network-free.

Checklist

• The title uses English Conventional Commit format: type(scope): summary.
• The related issue or design discussion is linked when applicable.
• bash scripts/verify.sh passes.
• Every applicable live-network component smoke test passes, or this PR states why none applies.
• Public behavior has focused tests, an example, and documentation where applicable.
• The PR is complete, small, and contains no unrelated changes.

[Copilot]

Pull request overview

This PR adds a new, standard-library-only quantmind.etl leaf package that provides agent-native, locally observable ETL scaffolding with atomic run.json snapshots and sparse events.jsonl journals, supporting both whole-run and strict serial micro-batch execution shapes while preserving existing quantmind.flows behavior and architecture boundaries.

Changes:

• Introduces ETLPipeline (whole-run) and BatchETLPipeline (micro-batch) with run handles that emit machine-readable receipts and persist local, JSON-safe observation state.
• Adds comprehensive tests (success, failure, cancellation, journaling failures, surrogate/unicode hardening, progress semantics), plus runnable local examples.
• Updates docs, design context, contributor guidance, .gitignore, and import-linter contracts to codify quantmind.etl as an independent leaf.

Reviewed changes

Copilot reviewed 21 out of 23 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
quantmind/etl/_record.py	Core snapshot/event persistence, progress scoping, JSON-safety, and error summarization for ETL runs.
quantmind/etl/_pipeline.py	Whole-run extract → transform → load pipeline scaffold with one-shot run handle and receipt.
quantmind/etl/_batch.py	Strict serial micro-batch runner with bounded summary, batch snapshot fields, and count aggregation.
quantmind/etl/__init__.py	Public exports for the new ETL scaffolds and contracts.
tests/etl/test_pipeline.py	Whole-run pipeline behavioral and hardening tests (progress rules, lifecycle, failures/cancellation).
tests/etl/test_batch.py	Micro-batch pipeline behavioral and adversarial tests (serial semantics, totals assertions, cleanup/cancel edges).
tests/etl/test_examples.py	Executes example scripts in dry-run/normal modes and validates artifacts + recorded snapshots.
tests/etl/__init__.py	Test package marker for ETL tests.
examples/etl/local_artifact.py	Runnable example showing whole-run ETL that writes an idempotent local artifact (and plans in dry-run).
examples/etl/batch_local_artifacts.py	Runnable example showing micro-batch ETL delivering per-batch local artifacts (and planning in dry-run).
docs/etl.md	Public guide describing ETL APIs, dry-run semantics, local observation contract, and progress/event behavior.
docs/README.md	Adds ETL rows to the public operation catalog with examples and guide links.
contexts/design/operations/etl.md	Canonical design contract for ETL execution shapes, dry-run semantics, and observation behavior.
contexts/design/README.md	Adds ETL design doc to the design index.
contexts/usage/README.md	Updates public import guidance to include quantmind.etl as the ETL entry point.
contexts/CONTEXT_MAP.md	Adds ETL design doc to the global context navigation map.
AGENTS.md	Updates module map and architecture constraints to include ETL scaffolds and their separation from flows.
pyproject.toml	Adds import-linter contract enforcing quantmind.etl as an independent leaf; updates other boundaries accordingly.
.gitignore	Ignores .quant-mind/ run artifacts and .coverage.* files.
.claude/skills/quantmind-dev/SKILL.md	Updates contributor skill metadata to include etl in the module list.
.claude/skills/quantmind-dev/references/develop-components.md	Documents ETL’s import boundary (“imports nothing”) and ETL component development guidance.
.agents/skills/quantmind-dev/SKILL.md	Mirrors the skill metadata update for the .agents copy.
.agents/skills/quantmind-dev/references/develop-components.md	Mirrors the ETL boundary + guidance update for the .agents copy.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.