Increase the default segment size to 1 GiB

[mavam]

📔 Description

The 4x increase in segment size decreases the range map fragmentation in the archive. It also will reduce startup time for larger archives. There should not be a latency problem with larger file sizes since we mmap the segments.

📝 Checklist

• All user-facing changes have changelog entries.
• The changes are reflected on docs.tenzir.com/vast, if necessary.
• The PR description contains instructions for the reviewer, if necessary.

🎯 Review Instructions

All at once.

[dominiklohmann]

Startup and export speed seems unchanged; import speed is increased with larger segments. Setting it to 2 GiB causes segfaults on my machine.

128 MiB

❯ rm -rf vast.db; repeat 10000; do gunzip -c integration/data/zeek/conn.log.gz; done | time vast --max-segment-size=128 -vquiet -N import --batch-encoding=msgpack zeek
user=1887.21s system=250.88s cpu=394% total=9:01.61

❯ hyperfine --warmup=3 --shell=zsh --min-runs=50 -- "vast -vquiet -N export -n 1 ascii '\"zeek\" in #type'"
Benchmark #1: vast -vquiet -N export -n 1 ascii '"zeek" in #type'
  Time (mean ± σ):      1.076 s ±  0.139 s    [User: 511.8 ms, System: 297.1 ms]
  Range (min … max):    1.030 s …  1.800 s    50 runs

1 GiB

❯ rm -rf vast.db; repeat 10000; do gunzip -c integration/data/zeek/conn.log.gz; done | time vast --max-segment-size=1024 -vquiet -N import --batch-encoding=msgpack zeek
user=1766.27s system=233.89s cpu=394% total=8:27.49

❯ hyperfine --warmup=3 --shell=zsh --min-runs=50 -- "vast -vquiet -N export -n 1 ascii '\"zeek\" in #type'"
Benchmark #1: vast -vquiet -N export -n 1 ascii '"zeek" in #type'
  Time (mean ± σ):      1.069 s ±  0.010 s    [User: 591.1 ms, System: 330.0 ms]
  Range (min … max):    1.053 s …  1.094 s    50 runs

2 GiB

❯ rm -rf vast.db; repeat 10000; do gunzip -c integration/data/zeek/conn.log.gz; done | time vast --max-segment-size=2048 -vquiet -N import --batch-encoding=msgpack zeek
user=507.01s system=48.39s cpu=463% total=1:59.77
zsh: segmentation fault  repeat 10000; do; gunzip -c integration/data/zeek/conn.log.gz; done

[mavam]

Yeah, writing larger strides intuitively also increases throughput. So an additional import speedup makes sense.

A backtrace for the segfault would help a lot.

[dominiklohmann]

A backtrace for the segfault would help a lot.

The maximum size of a FlatBuffers builder is 2 GiB.