qvd2parquet v2.2.0

Everything a run reports about itself gets more honest. Two of the release's
threads come from the same 213-column SAP extract that shaped 2.1.0: a profile
is only useful if it renders a value in the type the column is written as, and
a composite primary key gets nothing from the dictionary encoding the writer
defaults to. --encoding auto answers the second by measuring rather than
guessing, since whether delta_byte_array pays depends on the row order and
nothing in the symbol table reveals it.

The other thread is --log, which was accepted and silently ignored on every
single-file conversion, and which could be pointed at a file the run itself
writes. That destroyed the run's own output or its log and still exited 0.

No flag changed meaning and no conversion produces different data, so this is
minor rather than major: --encoding and --encoding auto change nothing
without being asked for, and the compatibility promise from 2.0.0 holds.

Added

• An --exclude pattern that matches no field is reported instead of passing
  silently. --columns already fails on a name that matches nothing, while
  --exclude accepted anything, and both ways of writing a pattern wrongly
  look exactly like success: % is not the wildcard %* and matches only a
  field named %, and the name --field-regex produces is never what
  --exclude sees, since exclusion is decided first. It stays a note rather
  than an error, because one command line is often pointed at a folder of
  tables that do not all carry the same fields. The patterns appear in
  --inspect, in the run log, and in a --log record as excludeNoMatch.
• --field-regex says which fields it left alone. A field the expression does
  not match keeps its original name, which is what lets a rule target a subset,
  but on a 213 column SAP extract the two fields that stayed behind are
  invisible among the renamed ones. Runs now close the schema notes with
  2 of 3 field(s) renamed, 1 unchanged: PlainField, naming at most five.
  --schema-report carries the full list under fieldRegex, and a --log
  record carries fieldsRenamed and fieldsUnchanged as counts, since a
  record there is one line per file.
• --inspect and --schema-report show each column's value range, rendered in
  the type the column is written as. A QVD stores a date as a serial day
  number, so a profile reported WADAT as 38365..411241 and a goods-issue date
  in the year 3025 read as an ordinary integer among plausible neighbours. The
  same range now reads 2005-01-13 .. 3025-12-08, next to a BUDAT of
  2019-12-27 .. 2026-08-31. Timestamps follow the run's timezone rules, so
  the rendering matches what would be written rather than assuming UTC.
• Decimal columns whose widest value already fills most of the type's range are
  named. A decimal's precision is inferred from its values, so it fits them
  exactly by construction and the question is never whether the data fits; it
  is what a later load has left. On a 213-column SAP extract exactly one column
  qualified, VV120 at 81% of a decimal(12,2), which one larger value would
  have taken out of range. --schema-report carries the limit and
  usedFraction per column, and the --log record carries
  decimalsNearLimit, the names alone, since a record there is one line per
  file and stays that way.
• --encoding pins a column to a Parquet encoding, as
  --encoding '%*_PKEY=delta_byte_array'. A column whose values are nearly all
  distinct gets nothing from the default dictionary: the dictionary page
  overflows, the writer falls back to PLAIN, and the column is stored as raw
  bytes with only the compressor working on it. A Qlik composite primary key is
  that column, one distinct value per row. Patterns are wildcards over both the
  output name and the original QVD name, so one rule covers a folder of SAP
  tables whose keys are named per table, and a later rule wins over an earlier
  one. An encoding the column's type cannot carry is refused before the
  conversion starts, naming the ones that fit, and --inspect shows what a run
  would pin. Nothing changes without the flag.
• --encoding auto measures the choice instead of guessing at it. Whether
  delta_byte_array pays depends on the order the rows arrive in, and nothing
  in the symbol table reveals that order, so sampled rows are written through
  the real writer twice, once as the run would today and once with each
  candidate, and the compressed column chunks are compared. Three windows of
  100,000 consecutive rows, at head, middle and tail, land within about a point
  of the whole file, and the estimate converges from above, so it understates a
  win rather than overselling one; a file of 300,000 rows or fewer is measured
  in full. On a 3M row key that arrives in document order the
  sample measured 31% and the conversion wrote 1.8 MiB against the default's
  6.2 MiB; on the same values shuffled nothing is adopted and the run says so.
  --inspect --encoding auto reports the measurement without converting, and
  is the only thing that makes inspect read records at all. Over --out-dir
  each file is measured on its own, since no pattern can know what a given
  table's key looks like. A column an explicit rule names is not measured at
  all, so the tool never recommends against a decision already taken, and
  --schema-report records the encoding each column is actually written with,
  measured or pinned.
• Progress lines say how far along a phase is and roughly how long is left:
  converted 5000000/20589661 rows (24%) in 3m21s (24875 rows/s, about 10m26s left). The row total is read from the QVD header before a record is
  decoded, so both come from numbers the run already holds. The throughput
  shown is still the average since the phase started, while the estimate
  follows the recent rate, because a run carries its startup cost in the
  average long after it has found its speed. The quality gate projects
  separately, having its own total and its own speed.

Fixed

• --log now writes a file record and summary for single-file conversions. It
  was accepted but silently ignored outside --out-dir mode.
• --log is refused when it names a file the run itself writes, in batch mode
  as well as for a single conversion. The log is created with O_TRUNC, so a
  collision destroyed whichever of the two was written second while the run
  still reported writing both. A batch was the worse case because none of the
  paths at risk are typed on the command line: the inputs come from expanding
  directories, and every output and per-file report is derived from an input
  under --out-dir. Pointing --log at an input truncated a 17 KiB QVD to a
  few hundred bytes of JSON Lines reporting that the file it had just destroyed
  was not a QVD, and pointing it at a generated output exited 0 having written
  the Parquet and no log at all. An input the run could not examine counts as
  an input here too: it is reported as a failed file and written to the log, so
  a log allowed to take its path named the file as missing and created it in
  the same breath.
• A batch run reports each file's progress. --out-dir passed a nil logger to
  the converter, so it discarded the schema notes, the row counts and the
  quality gate's progress: a folder holding one large table printed a line on
  starting and nothing again until it finished, which on a twenty-million-row
  table is a quarter of an hour of silence. Converting several files at once,
  each line names the file it belongs to, and the writer is serialized so two
  files cannot interleave mid-line.

Changed

• --decimal-strict reports up to three offending values per column instead of
  stopping at the first, with the total so the reader knows how much was not
  shown. One example rarely settles whether a column holds genuine extra
  decimals or float64 representation error, which is the question a reader
  actually has, and finding the rest previously meant leaving the tool.
• A value in those messages is shown as the double actually holds it, not in
  scientific notation. A value whose shortest form reads 8115022364.865 is
  stored as 8115022364.864999771, and seeing both is what separates
  representation error from a third decimal that is really in the source. The
  message names the step it is not a multiple of, 0.001, rather than a power
  of ten, and no longer quotes an empty display string for a pure numeric
  symbol that never had one.
• Each phase reports its own duration. The conversion's timing used to arrive
  only as the last --progress line, so --progress 0 left it as the one
  phase that never accounted for itself while the quality gate always did, and
  a slow run could not be attributed without re-running it. converted N/M rows is now the running count --progress governs, and conversion finished in T: N rows is printed whatever it is set to, matching the gate's line.
• The final line says overall. It reports the whole run -- conversion, gate,
  and the rest -- but sat next to a verb about writing, so its figure read as
  the time taken to write the file. On a wide file the gate alone can be the
  larger half of it.

Install

Download the archive for your platform below, unpack it, and put qvd2parquet
on your PATH. Verify the download against SHA256SUMS:

shasum -a 256 -c SHA256SUMS --ignore-missing

Binaries are pure Go and statically linked, so they need no runtime
dependencies: Linux (amd64, arm64), Windows (amd64, arm64) and macOS (amd64,
arm64).