Repository navigation
v0.1.0
First public release. Published to PyPI as microtrade-fwf; the import path
and CLI command stay microtrade.
Added
- Streaming FWF ingest pipeline (
microtrade ingest) that reads each
<trade_type>_<YYYYMM>.zipdirectly from its archive, slices columns
against a versioned spec, and writesyear=YYYY/month=MM/part-0.parquet
atomically under a per-type dataset root. Per-partition JSONL manifest
under<output>/_manifests/<trade_type>/<run_id>.jsonl. - Excel → YAML spec converter (
microtrade import-spec) for the real
upstream workbook layout: positional sheet→trade_type mapping,
autodetectedPosition | Description | Length | Typeheader, skipped
Blankfiller rows,Char/Numdtype tokens. - Per-sheet preamble sanity check that refuses to accept a mis-ordered
workbook (catches swapped imports/exports sheets before they land in
YAML). Datedtype (pyarrowdate32) withyyyymmdd_to_date/
yyyymm_to_dateparsers, resolved per column at stream start.- Row-level quality-issue log: parse failures (bad numeric, blank
non-nullable, bad date) go to
<output>/_quality_issues/<trade_type>/<run_id>.jsonland the row is
skipped; the surrounding partition still writes. Capped per partition
(--max-quality-issues, default 10,000) and guarded by a skip-rate
abort (--max-skip-rate, default 0.5) so pathological inputs can't
balloon the log or silently land a mostly-broken partition. - Canonical
_dataset_schema.jsonrefreshed at
<output>/<trade_type>/_dataset_schema.jsonafter every run — union
of all committed specs for that trade type, latest dtype wins,
nullability widens. microtrade validate-specs— lints everyv*.yamlunder a spec dir,
verifies filename versions matcheffective_from, prints a
per-trade-typediff_specschangelog, and checkscanonical_columns
to surface cross-version dtype conflicts. Summary reports scan scope
(OK (N trade types, M specs)).microtrade inspect— dumps the resolved spec and first rows of a raw
zip (or plain FWF via--type/--period), with per-column
[start..end] dtype 'value'annotation;--rawprints full lines.- Reference YAML specs generated from
examples/microdata-layout.xls
ship undersrc/microtrade/specs/, somicrotrade ingestworks out
of the box against any real data that matches the 2020-01 layout.
Notes
- Consumers that hive-scan the dataset should use a
**/*.parquetglob
because_dataset_schema.jsonlives at the dataset root (see the
README's "Output layout" section). - All four CLI subcommands (
ingest,import-spec,validate-specs,
inspect) ship implemented; the package is fully typed (py.typed
marker included in the wheel).