Repository navigation
Releases
Releases · digitalashes/hpt-parser-rs
v0.1.4
Changed
- Faster, with the same Parquet bytes: the records are made on a thread of their own, the
column values are made and encoded on two to six threads, a column that reads only the
charge is made once for all its payers, and JSON is read by a reader of its own. Against
0.1.3: a 1 GB tall CSV 4.3 -> 2.7 seconds, the 2.5 GB wide CSV 4.4 -> 2.3, a 1.2 GB CSV
with 1000 columns 9.6 -> 3.3, a 229 MB JSON 1.4 -> 0.7; with theexamples/payer-names
config the tall CSV 3.5 -> 2.7 and the JSON 1.3 -> 0.7. CPU time is up to 36% higher on
CSV files. - The release binaries are built with thin LTO and one codegen unit: 22 MB instead of 27.
- A JSON charge item's records stay in one plugin call: a call ends before an item that
does not fit in 512, and an item with more records than that is a call of its own. - JSON syntax errors say what was expected and what was found, at the start of the token:
line 1, column 6: expected ':', found '1'for{"a" 1}(wasUnexpected NUMBER (1) in state COLON).
Fixed
- A JSON file with a syntax error gives every record before the error: the ones read in
the same 64 KiB as the error were lost.
v0.1.3
Two CSV header shapes found in real 2.0.0 files are read now:
- Wide files that name a payer without a plan,
standard_charge | Aetna | negotiated_dollar
orestimated_amount | Aetna, lost those values without a word. They are read as a
payer/plan block with no plan (planis null). In one sampled file this was 14 of 25
payers, in another 16 of 18. In a tall file such columns are still not read. - A tall file with a template column left in, such as
estimated_amount | [payer_name] | [plan_name], looked both tall and wide and failed. A[payer_name]or[plan_name]
placeholder is no payer now, so the file reads as tall.
Files without these shapes give the same records and the same Parquet bytes as 0.1.2.
v0.1.2
- About 30% faster with plugins: the filters, the plugin calls and the rows of a batch run
on a thread of their own while the next batch is read, as without plugins, with the same
Parquet bytes as before. On a 2019 laptop, a 1 GB tall CSV with theexamples/rates
config 7.7 -> 5.2 seconds, the 2.5 GB wide CSV 5.8 -> 4.0, a 229 MB JSON 2.1 -> 1.5. - With plugins, a batch now ends early where the file's metadata changes, so each record
has the metadata it was read with, and a parse error still has the records before it
written.
v0.1.1
- About twice as fast without plugins: records are read, made into rows and encoded on
three threads, with the same Parquet bytes as before. On a 2019 laptop: a 1 GB tall CSV
8.2 -> 4.3 seconds, the 2.5 GB wide CSV 7.4 -> 4.4, a 1.2 GB CSV with 1000 columns
16.8 -> 9.7, a 229 MB JSON 2.8 -> 1.3. - The command reads a
.gz, or a.zipwith one.csvor.jsonin it, unpacked on the
fly (a Mac's__MACOSX/entries are left out). The default output name drops.gzor
.zip. - The binary has a default feature of its own,
cli(withflate2andzip), so
parquetalone no longer pulls them in or builds it. - An input that is neither
.csvnor.jsonnow exits with 1 (the input failed), not 2. cargo binstall hpt-parserfetches the release binary.