qvd2parquet v2.0.0

Defaults re-chosen for wide files. A 213-column, 20.6M-row SAP extract was the
case that prompted it: it converted at 23k rows/s while using 16% of a 16-core
Xeon and 36 GB of resident memory. None of the changes below alter the data a
QVD converts to. Two of them are why this is a major release rather than a
minor one:

• --batch-rows no longer sets the Parquet row group size, so it has changed
  meaning. If you passed it to control row groups, pass --row-group-rows
  instead; if you passed it to control memory, it still does that and now
  defaults to sizing itself.
• --quality-gate defaults to full, so a conversion whose output does not
  match its input now exits 6 where it previously exited 0. The output was
  already wrong in those cases; the gate is what is new. It also makes a
  default run several times slower -- roughly 4.7x on a wide file -- so a
  scheduled job should either budget for it or name a cheaper mode.

Changed

• --workers=0 now resolves to one decode worker per two CPUs, with a floor of
  two, instead of one per CPU. Decoding itself scales close to linearly, but it
  is only half the pipeline -- the Parquet writer is a single goroutine -- and
  every worker costs its share of in-flight Arrow memory, which on a wide file
  is roughly workers * batch-rows * columns * 16 bytes and dominates resident
  size. A 213-column file on a 16-core machine held around 9.8 GB at one worker
  per CPU against 7.3 GB at four. Half the CPUs keeps about two thirds of the
  decode throughput at half the batches in flight; on a hyper-threaded machine,
  where runtime.NumCPU() counts threads, it works out to roughly one worker
  per physical core. This changes only how much of the machine a conversion
  uses, not what any file converts to, so it stays inside the compatibility
  promise above. Pass --workers explicitly to override it in either
  direction.

• --file-workers divides the same automatic budget, so batch mode and
  single-file mode now agree on how much of the machine to use.

• --quality-gate now defaults to full instead of none. A conversion
  nobody checked is not a conversion anybody can trust, and full is the only
  mode that fingerprints values, so it is the only one that catches a value
  which survived the type policy but not the round trip. Two consequences to
  plan for. It is not free, and it costs in two places: the gate reads the whole
  output back and digests every cell, and, because full is the only mode that
  fingerprints values, each decode worker also digests every value inside the
  conversion itself -- so the reported rows/s drops as well as the total wall
  clock. Together that is roughly 4.7x the conversion on a 213-column fixture,
  and it grows with width; on a 213-column, 20.6M-row SAP extract on a 16-core
  Xeon, conversion alone went from 26.7k rows/s to 22.4k before the read-back
  began. Name basic, numeric or none when throughput matters -- neither
  carries the inline cost. And a conversion that previously exited 0 can now exit 6, because
  the file is checked where it previously was not -- a run that starts failing
  under this default was already producing that output, the gate is only now
  reporting it. Validation still reads the temporary file before the final
  rename, so a failed gate never leaves a final-looking output behind.

• --batch-rows and the Parquet row group size are no longer the same number.
  The two were one setting, which made them impossible to tune apart: a batch
  is held per worker and again in the queue to the writer, so it costs about
  rows * columns * 16 bytes and wants to shrink on a wide file, while the row
  group is what a reader scans and a dictionary is built over, and shrinking it
  inflates the output. Lowering --batch-rows to save memory tripled the file
  on a 213-column fixture, to 486 MiB. Row group size moves to a new
  --row-group-rows, still 65536, so row groups hold the same number of rows
  as before.

  What a row group holds is a separate matter, and is not changed by this: a
  row group has been filled from whichever chunks finish while it is open ever
  since decoding became parallel, so on a sorted input its statistics already
  covered a wider range than its rows, by a factor that varies from run to run.
  Measured on a 500k-row fixture keyed by an ascending integer, the spans
  covered by the row groups summed to 3.8x and 3.1x the row count on two runs
  under the old coupling, and 2.7x under the new default -- against 1.0x at
  --workers=1 in both. This is now stated in the README rather than left
  implied by "row order is not preserved".

• --batch-rows now defaults to 0, meaning a row count sized from the file's
  width to hold about 2M cells, between 4096 and 65536 rows. A narrow file
  still batches 65536 rows; a 213-column file batches ~9.4k, so in-flight
  memory stays put instead of growing with width. On that 213-column, 1M-row
  fixture, peak resident size fell from 7.3 GB to 1.8 GB at four workers. The
  throughput effect is the larger one: at a fixed 65536-row batch, going from
  four workers to sixteen bought 10% (52.2k to 57.7k rows/s), because each
  worker carried a batch of 65536 * 213 cells and the machine spent its time
  moving memory; sized by cells the same step goes 54.7k to 95.1k rows/s. The
  batch size was what stopped the workers from scaling. An explicit
  --batch-rows is honoured unchanged.

• Row order is now preserved. Decoding still runs in parallel, but records
  reach the Parquet writer in chunk order instead of completion order, so the
  output holds the QVD's rows in their original order whatever the worker
  count. This replaces the previous "row order is not preserved" caveat.

  The reason to want it is row groups. A row group's per-column min/max only
  bound the rows it holds if those rows are contiguous in the source, and under
  completion order they were not: on a 500k-row fixture keyed by an ascending
  integer, the spans covered by the row groups summed to 3.8x and 3.1x the row
  count on two runs, so an engine skipping row groups over a sorted key could
  rule out far fewer than the data allowed. It is now 1.0x, and stable across
  runs.

  A chunk that finishes ahead of its predecessors waits in a reorder buffer,
  bounded by a feeder window of two chunks per worker: without that window a
  slow first chunk would let the other workers pile every remaining chunk into
  memory behind it, and a writer that simply waited for the missing chunk
  instead would fill the results channel and deadlock the slow chunk when it
  finally handed its record over. Measured on a 213-column fixture, ordering
  costs no throughput and slightly less memory, the window being tighter than
  what was previously in flight.

Fixed

• BenchmarkDecode measured four workers in every case above four. The
  fixture is four chunks at the default batch size and WorkerCount clamps to
  the chunk count, so the worker-scaling table in the README was reporting a
  plateau that was the clamp rather than the code. The benchmark now uses a
  batch small enough to keep every worker fed, names the automatic case
  workers=default instead of workers=numcpu, measures one per CPU
  separately, and reports the worker count each case actually ran.

• Decode workers no longer serialize their reads on Windows. Every worker read
  its chunks through the one *os.File opened for the input, and Windows -
  unlike Unix pread, which needs no lock - implements ReadAt by taking that
  descriptor's read and write locks and moving its shared file pointer, so all
  workers queued on a single mutex for every chunk. Each worker now opens its
  own handle, which is a separate kernel file object with its own pointer; the
  handle is reopened from the path rather than duplicated, since a duplicated
  handle shares the very pointer that lock exists to protect. A worker falls
  back to the shared handle if the path cannot be reopened or no longer names
  the same file, so a replaced input is never decoded unvalidated. Unix
  behaviour is unchanged.

Install

Download the archive for your platform below, unpack it, and put qvd2parquet
on your PATH. Verify the download against SHA256SUMS:

shasum -a 256 -c SHA256SUMS --ignore-missing

Binaries are pure Go and statically linked, so they need no runtime
dependencies: Linux (amd64, arm64), Windows (amd64, arm64) and macOS (amd64,
arm64).