Skip to content

Releases

v0.1.4

Choose a tag to compare

@github-actions github-actions released this 24 Oct 23:34
v0.1.4
8192eff

Changed

  • Faster, with the same Parquet bytes: the records are made on a thread of their own, the
    column values are made and encoded on two to six threads, a column that reads only the
    charge is made once for all its payers, and JSON is read by a reader of its own. Against
    0.1.3: a 1 GB tall CSV 4.3 -> 2.7 seconds, the 2.5 GB wide CSV 4.4 -> 2.3, a 1.2 GB CSV
    with 1000 columns 9.6 -> 3.3, a 229 MB JSON 1.4 -> 0.7; with the examples/payer-names
    config the tall CSV 3.5 -> 2.7 and the JSON 1.3 -> 0.7. CPU time is up to 36% higher on
    CSV files.
  • The release binaries are built with thin LTO and one codegen unit: 22 MB instead of 27.
  • A JSON charge item's records stay in one plugin call: a call ends before an item that
    does not fit in 512, and an item with more records than that is a call of its own.
  • JSON syntax errors say what was expected and what was found, at the start of the token:
    line 1, column 6: expected ':', found '1' for {"a" 1} (was Unexpected NUMBER (1) in state COLON).

Fixed

  • A JSON file with a syntax error gives every record before the error: the ones read in
    the same 64 KiB as the error were lost.

v0.1.3

Choose a tag to compare

@github-actions github-actions released this 23 Oct 12:11
v0.1.3
9efceee

Two CSV header shapes found in real 2.0.0 files are read now:

  • Wide files that name a payer without a plan, standard_charge | Aetna | negotiated_dollar
    or estimated_amount | Aetna, lost those values without a word. They are read as a
    payer/plan block with no plan (plan is null). In one sampled file this was 14 of 25
    payers, in another 16 of 18. In a tall file such columns are still not read.
  • A tall file with a template column left in, such as estimated_amount | [payer_name] | [plan_name], looked both tall and wide and failed. A [payer_name] or [plan_name]
    placeholder is no payer now, so the file reads as tall.

Files without these shapes give the same records and the same Parquet bytes as 0.1.2.

v0.1.2

Choose a tag to compare

@github-actions github-actions released this 23 Oct 00:02
v0.1.2
4874494
  • About 30% faster with plugins: the filters, the plugin calls and the rows of a batch run
    on a thread of their own while the next batch is read, as without plugins, with the same
    Parquet bytes as before. On a 2019 laptop, a 1 GB tall CSV with the examples/rates
    config 7.7 -> 5.2 seconds, the 2.5 GB wide CSV 5.8 -> 4.0, a 229 MB JSON 2.1 -> 1.5.
  • With plugins, a batch now ends early where the file's metadata changes, so each record
    has the metadata it was read with, and a parse error still has the records before it
    written.

v0.1.1

Choose a tag to compare

@github-actions github-actions released this 22 Oct 22:54
v0.1.1
73a356a
  • About twice as fast without plugins: records are read, made into rows and encoded on
    three threads, with the same Parquet bytes as before. On a 2019 laptop: a 1 GB tall CSV
    8.2 -> 4.3 seconds, the 2.5 GB wide CSV 7.4 -> 4.4, a 1.2 GB CSV with 1000 columns
    16.8 -> 9.7, a 229 MB JSON 2.8 -> 1.3.
  • The command reads a .gz, or a .zip with one .csv or .json in it, unpacked on the
    fly (a Mac's __MACOSX/ entries are left out). The default output name drops .gz or
    .zip.
  • The binary has a default feature of its own, cli (with flate2 and zip), so
    parquet alone no longer pulls them in or builds it.
  • An input that is neither .csv nor .json now exits with 1 (the input failed), not 2.
  • cargo binstall hpt-parser fetches the release binary.

v0.1.0

Choose a tag to compare

@github-actions github-actions released this 22 Oct 19:18
v0.1.0
551fc3a
v0.1.0