cms-mrf-extractor

Extract CMS Transparency-in-Coverage machine-readable files (MRFs) to parquet.

Payer MRFs are single JSON objects that reach tens of gigabytes decompressed. This package reads one in a single pass, splits its arrays into byte blobs at exact item boundaries, and parses those straight into Arrow buffers — around 160–200 MB/s of decompressed JSON on one machine, against 55 MB/s for the obvious ijson + json.loads implementation.

Local paths and s3:// URIs work the same for both input and output.

Install

pip install cms-mrf-extractor

Linux and macOS. The parallel paths use fork(); on Windows the extractor says so and runs sequentially in one process.

Use it from the shell

mrf-extract /data/aetna                        # -> ./output-aetna/{pr,nr,nrpr}/
mrf-extract /data/aetna -o /data/out-aetna
mrf-extract s3://bucket/payer/ -o s3://bucket/parquet/
mrf-extract /data/aetna --dry-run --json       # which sections each file holds

mrf-extract --help lists every flag. The ones that matter most:

Flag	What it does
--rows-per-file N	rows per parquet file (default 5000)
--id-type, --npi-type	force a column type instead of detecting it
--workers N	processes when there are more files than cores
--blob-workers N	decode workers inside a single file
--no-arrow-json	decode with json.loads (reference path, ~3x slower)
--json	run summary as JSON on stdout; logs stay on stderr

Use it from Python

from cms_mrf_extractor import run

summary = run('/data/aetna', output='/data/out-aetna', rows_per_file=50_000)
# {'aetna_in-network': {'nr': (1_284_530, 257), 'pr': (2_411, 1)}}

configure() takes the same keywords and sets them without extracting, and discover() reports what a directory holds:

from cms_mrf_extractor import configure, discover, run

configure('/data/payer', id_type='string', progress=False)
for path, base_name, kinds, pr_key in discover():
    print(base_name, kinds, pr_key)   # e.g. payer_01 ['nr'] provider_references
run()

One configured extractor per process — settings are module state that the worker processes inherit through fork().

What it writes

Three section types, each into its own subdirectory of the output, as {source_name}_{kind}_{NNN}.parquet with zstd compression:

Directory	Section	Shape
pr/	provider_references	provider_group_id, provider_groups[] (npi list + tin)
nr/	in_network referencing provider ids	rates carry provider_references[]
nrpr/	in_network with groups inline	rates carry provider_groups[]

Which one a file produces is discovered from the file, not from its name. The nested list/struct shape of the source is preserved; nothing is exploded or joined.

Some payers ship the provider definitions under provider_group_reference with business_name beside tin rather than inside it. That layout is detected and reshaped to the same schema. Items carrying only a location URL are fetched over HTTP and inlined.

Column types

provider_group_id / provider_reference and npi are not the same JSON type in every payer's files — some quote npi, some ship ids too large for int64. The types are read out of the files during discovery and widened one way (int64 → decimal128 → string for ids, int64 → string for npi), then the widest answer across the directory is used, because one run writes one schema.

Detection reads forward from byte 0 under a bounded event budget. A payer that puts its provider section after a multi-gigabyte in_network array hides it behind more events than that budget allows; the run warns and names those files. Pass --npi-type string if the warning applies to you.

Correctness ladder

Three tiers, each strictly more general than the last, per file:

1. Arrow-native — pyarrow.json parses blobs into Arrow buffers directly.
2. Per-blob Python — json.JSONDecoder.raw_decode, one blob at a time.
3. ijson — structure-agnostic streaming, for layouts the byte splitter cannot prove boundaries for.

A failure at any tier discards that file's output and retries at the next one, so a layout the fast path cannot handle costs speed rather than a run. raw_decode consumes exactly one JSON value and reports where it ended, so a bad split surfaces as a JSONDecodeError and a retry — never as corrupted parquet. Truncated files are reported and listed in bad_files.log in the output directory.

Environment variables

Equivalent to the CLI flags, useful for containers:

Variable	Effect
MRF_SOURCE_DIR, MRF_OUTPUT_DIR	source and output when none is passed
MRF_ID_TYPE, MRF_NPI_TYPE	force a column type
MRF_TYPE_DETECT=0	skip detection, keep the int64 defaults
MRF_ARROW_JSON=0	decode with json.loads
MRF_STREAM_JSON=0	decode blob by blob rather than streaming a section

The MRF_ARROW_JSON / MRF_STREAM_JSON switches exist so a suspected regression can be bisected across the three tiers without editing code.

Why it is fast

PERF_NOTES.md in the source distribution carries the measurements. The short version:

• ijson.items() never stops early, so pulling an 80 MB array out of a 34 GB file read all 34 GB — 399 s instead of 2 s. Structure discovery still uses ijson, where incremental parsing is the right tool; extraction does not.
• Items are located by bytes.find() on the first item's first key (~3.2 GB/s), so a section becomes independent byte blobs without parsing anything.
• Building Python dicts and walking them with pa.Table.from_pylist ran at 81 MB/s per core and scaled at 35% efficiency across 12 processes — a 10 MB blob becomes 50–100 MB of Python objects and the workers end up bound by the allocator. pyarrow.json on the same bytes creates no Python objects: 161–343 MB/s per core.
• One pass covers every section of a file, in the order the sections appear, so a file with provider_references last costs the same as one with it first.

License

MIT — see LICENSE.