v2.1.0

What's Changed

• Added progress logging.
• Improved performance of the slicing of small input slices.

Progress logs (PR #29)

Slicer logs progress with exponentially increasing intervals.

Improved performance for small input slices (PR #27)

The slicer loads the next input slice into the memory ahead, so opening file delay has been removed.

❗ A side effect is a slight increase in memory consumption, the default soft memory limit was increased from 256MB to 512MB.

See example bellow (200 input slices, each has 5MB).

Before:

After:

Skipping small input tables (PR #30)

Slicing and compression of small tables may not make any sense.

• Job can be slower than if we skipped slicing completely.
• Therefore, it is possible to set a size threshold .
• The threshold (50MB by default) must be exceeded by at least one slice to start slicing.
• If the table is not sliced, the size of the entire table must exceed the threshold.
• Otherwise, the table is copied to the output without modification.

Example:

Skipping table "mytable": maximum size of slice "36B" is smaller than the threshold "100KB".
Table "mytable" has been copied to the output without modification.

Summary

New flags with default values:

• --log-interval-initial duration
  • Initial log interval. (default 10s)
• --log-interval-maximum duration
  • Maximum log interval. (default 15m0s)
• --log-interval-multiplier float
  • Log interval multiplier. (default 1.5)
• --ahead-block-size int
  • Size of a one read ahead input block. (default 1MB)
• --ahead-blocks int
  • Number of blocks read ahead from an input slice. (default 16)
  • 0 disables read-ahead.
• --ahead-slices int
  • Number of input slices opened ahead. (default 1)
• --input-size-threshold string
  • At least one slice must exceed the threshold, otherwise the table is copied without modification. (default 50MB)