Skip to content

perf(recordengine): intern repeated values in a byte column - #153

Closed
tdakkota wants to merge 1 commit into
mainfrom
perf/head-memory
Closed

perf(recordengine): intern repeated values in a byte column#153
tdakkota wants to merge 1 commit into
mainfrom
perf/head-memory

Conversation

@tdakkota

@tdakkota tdakkota commented Jul 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Closes #152. Partially addresses #140.

Replaces this PR's first approach (a leading-run form), which real-data measurement showed captured
almost nothing — see below.

What

A head buffer is sorted (stream, ts), and columns like a stream's serialized resource attributes or
a severity text take a handful of distinct values over hundreds of thousands of rows. Every row stored
a full copy regardless, until the flush encoder's CodecDict deduped it. byteCol now interns by
default: the blob holds the distinct values, and a 4-byte id per row indexes them.

A near-unique column — a span id, a trace id, a verbose body — would pay a hash per row for a
dictionary that saves nothing, so interning is abandoned for good once the distinct bytes exceed
half the logical bytes. Half, not equal, because the id index costs 4 B a row. The test re-runs every
512 rows, because a column can start repetitive and diverge.

The flush buffer opts out entirely (noIntern): it feeds an encoder that wants a flat blob, so
interning there pays for the dictionary and the expansion. byteSize reports the logical bytes in
either form, so in-flight accounting and flush sizing are unchanged.

Also from #140: the ts sort permutes an interned column's id index alone, and otherwise through one
scratch column shared across a whole flush instead of a fresh destination per column per stream.

Why this and not run-length

Measured over 4 real log parts (558k rows, (stream, ts)-sorted — the order a head accumulates in):

column runs/row flat by run by distinct value
resource 0.24 149 MB 23 MB (6.4×) 108 KB (1379×)
severity_text 0.054 1.5 MB 128 KB (11.9×) 134 B (11363×)
attrs 0.16 70 MB 35 MB (2.0×) 26 MB (3×)
body 0.63 175 MB 144 MB (1.2×) 142 MB (1×)
trace_id 0.017 176 KB 149 KB (1.2×) 119 KB (1×)
span_id 0.020 88 KB 88 KB (1.0×) 88 KB (1×)

Two things this settles:

  1. A leading-run form is worthless here. Per part it covered 82, 103, 54 and 1 rows out of
    105k–228k. With Reduce log-stream cardinality: classify fields (identity / high-card indexed / low-card pivot) instead of hashing all resource attributes #104 a stream deliberately holds several distinct resources (that is the point —
    service.instance.id no longer splits streams), so rows from different instances interleave by
    timestamp and resource averages a run of ~8 rows.
  2. Interning beats run-length by ~200× on the column that matters, because those few distinct
    blobs recur non-adjacently. Runs only catch adjacency.

The last three rows are the bail-out cases, and the threshold picks them off exactly: body 0.81,
trace_id 0.68 and span_id 1.00 distinct/logical are all above ½, while resource (~0),
severity_text (~0) and attrs (0.37) stay interned.

Numbers

BenchmarkWriteLogs/32svc_2000recs, B/op (100x, count=3, mean):

tree ingest +flush
main 73.7 MB 93.0 MB
main + this PR 50.0 MB 77.5 MB
main + #104 (adds the resource column) 81.2 MB 107.0 MB
main + #104 + this PR 51.0 MB 84.7 MB

BenchmarkByteColAppend, 300 B cells: repeated 42 ns/op, 12 B/op; unique 248 ns/op, 1595 B/op —
the unique case matching the flat path it falls back to.

What is left of #140

writePart takes a whole column, so buildFlushColumns still materializes a second full copy of
every byte column; that is why the flush buffer has to opt out rather than carry its dictionary
through to the encoder. Removing it needs an incremental (or dictionary-aware) column API in block
— deliberately not attempted here. Pooling flushColumns was already in place (Engine.flushBuf).

Testing

Table tests over both forms, bail-out tests for unique-from-the-start and diverges-later, an
equivalence test that expanding changes contents not at all, cross-form appendRange/permute/
keep/gather coverage, a fuzzer (FuzzByteColForms, 4M execs clean) asserting a column reads back
exactly the cells appended wherever it is expanded, and an append benchmark. Full suite green,
golangci-lint clean.

🤖 Generated with Claude Code

@tdakkota tdakkota mentioned this pull request Jul 23, 2026
Draft
@github-actions

github-actions Bot commented Jul 23, 2026 •
edited
Loading

Copy link
Copy Markdown

📊 Golden benchmarks

Warning

3 benchmarks regressed and 1 improved across the golden set.
Largest regression: logs/write_flush latency +20.2%.

ℹ️ Some deltas are large but not statistically significant — likely shared-runner noise. Trust repeated, consistent runs.

benchstat base → head · linux/amd64 · AMD Ryzen 5 3600 6-Core Processor · n=6/benchmark · ~ = not significant (p≥0.05)

⏱️ Time — sec/op (lower is better)

Benchmark base head Δ
write/head 13.46ms 11.08ms −17.7% ⚪
write/flush 28.12ms 27.90ms −0.8% ⚪
write/concurrent 22.24ms 22.26ms +0.1% ⚪
read/fetch_all 2.82ms 2.85ms +1.1% ⚪
read/fetch_all_release 2.30ms 2.30ms +0.1% ⚪
read/fetch_recent 1.83ms 1.86ms +1.4% ⚪
query/promql_count_cpu_cores 686.49µs 683.24µs −0.5% ⚪
query/promql_full_scan_count 633.90µs 637.92µs +0.6% 🔴
query/promql_cpu_usage_range 2.17ms 2.18ms +0.3% ⚪
density 47.13ms 47.55ms +0.9% ⚪
logs/write_flush 1.67ms 2.01ms +20.2% 🔴
logs/merge 16.78ms 19.27ms +14.8% 🔴
geomean +1.34%

🚀 Throughput — B/s (higher is better)

Benchmark base head Δ
write/head 113.4MiB/s 137.7MiB/s +21.5% ⚪
write/flush 54.3MiB/s 54.7MiB/s +0.8% ⚪
write/concurrent 68.6MiB/s 68.6MiB/s −0.1% ⚪
read/fetch_all 541.4MiB/s 535.6MiB/s −1.1% ⚪
read/fetch_all_release 663.7MiB/s 663.1MiB/s −0.1% ⚪
logs/write_flush 84.7MiB/s 70.5MiB/s −16.8% 🔴
logs/merge 67.5MiB/s 58.8MiB/s −12.9% 🔴
geomean -1.88%

🚀 Throughput — Mpoints/s (higher is better)

Benchmark base head Δ
write/head 9.25 9.33 +0.9% ⚪
write/flush 3.56 3.58 +0.8% ⚪
write/concurrent 9.65 9.34 −3.2% ⚪
geomean -0.51%

📦 Bytes per op — B/op (lower is better)

Benchmark base head Δ
write/head 10.4MiB 6.5MiB −37.2% ⚪
write/flush 47.5MiB 46.8MiB −1.4% ⚪
write/concurrent 69.2MiB 67.0MiB −3.2% ⚪
read/fetch_all 2.3MiB 2.3MiB ≈0% ⚪
read/fetch_all_release 652.9KiB 653.0KiB ≈0% ⚪
read/fetch_recent 795.1KiB 795.1KiB ≈0% ⚪
query/promql_count_cpu_cores 416.7KiB 416.9KiB ≈0% ⚪
query/promql_full_scan_count 408.4KiB 408.4KiB ≈0% ⚪
query/promql_cpu_usage_range 495.1KiB 495.0KiB ≈0% ⚪
density 76.6MiB 76.6MiB ≈0% ⚪
logs/write_flush 1.5MiB 1.4MiB −7.2% 🟢
logs/merge 42.0MiB 40.1MiB −4.5% ⚪
geomean -5.13%

♻️ Allocations — allocs/op (lower is better)

Benchmark base head Δ
write/head 5.9k 5.7k −3.6% ⚪
write/flush 30.8k 30.6k −0.4% ⚪
write/concurrent 9.6k 9.3k −2.6% ⚪
read/fetch_all 4.5k 4.5k ≈0% ⚪
read/fetch_all_release 2.5k 2.5k ≈0% ⚪
read/fetch_recent 4.5k 4.5k ≈0% ⚪
query/promql_count_cpu_cores 1.3k 1.3k ≈0% ⚪
query/promql_full_scan_count 1.3k 1.3k ≈0% ⚪
query/promql_cpu_usage_range 1.7k 1.7k ≈0% ⚪
density 50.1k 50.1k ≈0% ⚪
logs/write_flush 13.0k 14.7k +12.7% 🔴
logs/merge 33.8k 35.7k +5.8% 🔴
geomean +0.91%

🔢 Rows per op — rows/op (informational)

Benchmark base head Δ
read/fetch_all 100.0k 100.0k ≈0% ⚪
read/fetch_all_release 100.0k 100.0k ≈0% ⚪
read/fetch_recent 10.0k 10.0k ≈0% ⚪
geomean +0.00%

🗜️ Density — B/point (lower is better)

Benchmark base head Δ
density 1.498 1.498 ≈0% ⚪
geomean +0.00%
Raw benchstat table 
goos: linux
goarch: amd64
pkg: github.com/oteldb/storage
cpu: AMD Ryzen 5 3600 6-Core Processor              
                                       │   base.txt   │              head.txt               │
                                       │    sec/op    │    sec/op     vs base               │
Golden/write/head-12                     13.46m ± 20%   11.08m ± 21%        ~ (p=0.240 n=6)
Golden/write/flush-12                    28.12m ±  4%   27.90m ±  2%        ~ (p=0.818 n=6)
Golden/write/concurrent-12               22.24m ± 68%   22.26m ±  9%        ~ (p=0.589 n=6)
Golden/read/fetch_all-12                 2.818m ±  1%   2.849m ±  5%        ~ (p=0.394 n=6)
Golden/read/fetch_all_release-12         2.299m ±  2%   2.301m ±  1%        ~ (p=0.818 n=6)
Golden/read/fetch_recent-12              1.831m ±  3%   1.857m ±  2%        ~ (p=0.310 n=6)
Golden/query/promql_count_cpu_cores-12   686.5µ ±  2%   683.2µ ±  2%        ~ (p=1.000 n=6)
Golden/query/promql_full_scan_count-12   633.9µ ±  2%   637.9µ ±  1%   +0.63% (p=0.041 n=6)
Golden/query/promql_cpu_usage_range-12   2.170m ±  1%   2.177m ±  2%        ~ (p=0.818 n=6)
Golden/density-12                        47.13m ±  3%   47.55m ±  2%        ~ (p=0.589 n=6)
Golden/logs/write_flush-12               1.673m ±  3%   2.011m ±  2%  +20.22% (p=0.002 n=6)
Golden/logs/merge-12                     16.78m ±  2%   19.27m ±  7%  +14.79% (p=0.002 n=6)
geomean                                  4.728m         4.792m         +1.34%

                                 │    base.txt    │               head.txt               │
                                 │      B/s       │      B/s       vs base               │
Golden/write/head-12               113.4Mi ±  25%   137.7Mi ± 17%        ~ (p=0.240 n=6)
Golden/write/flush-12              54.26Mi ±   4%   54.68Mi ±  2%        ~ (p=0.818 n=6)
Golden/write/concurrent-12         68.64Mi ± 214%   68.55Mi ±  8%        ~ (p=0.589 n=6)
Golden/read/fetch_all-12           541.4Mi ±   1%   535.6Mi ±  5%        ~ (p=0.394 n=6)
Golden/read/fetch_all_release-12   663.7Mi ±   2%   663.1Mi ±  1%        ~ (p=0.818 n=6)
Golden/logs/write_flush-12         84.73Mi ±   3%   70.48Mi ±  2%  -16.82% (p=0.002 n=6)
Golden/logs/merge-12               67.55Mi ±   2%   58.85Mi ±  7%  -12.88% (p=0.002 n=6)
geomean                            136.2Mi          133.6Mi         -1.88%

                           │  base.txt   │             head.txt             │
                           │  Mpoints/s  │ Mpoints/s   vs base              │
Golden/write/head-12         9.248 ±  3%   9.334 ± 2%       ~ (p=0.240 n=6)
Golden/write/flush-12        3.556 ±  4%   3.583 ± 2%       ~ (p=0.818 n=6)
Golden/write/concurrent-12   9.646 ± 47%   9.341 ± 5%       ~ (p=0.310 n=6)
geomean                      6.820         6.785       -0.51%

                                       │    base.txt    │              head.txt               │
                                       │      B/op      │     B/op       vs base              │
Golden/write/head-12                     10.417Mi ± 46%   6.541Mi ± 59%       ~ (p=0.485 n=6)
Golden/write/flush-12                     47.51Mi ±  6%   46.84Mi ±  1%       ~ (p=0.974 n=6)
Golden/write/concurrent-12                69.22Mi ± 77%   66.98Mi ±  3%       ~ (p=0.589 n=6)
Golden/read/fetch_all-12                  2.338Mi ±  0%   2.338Mi ±  0%       ~ (p=0.058 n=6)
Golden/read/fetch_all_release-12          652.9Ki ±  0%   653.0Ki ±  0%       ~ (p=0.429 n=6)
Golden/read/fetch_recent-12               795.1Ki ±  0%   795.1Ki ±  0%       ~ (p=0.593 n=6)
Golden/query/promql_count_cpu_cores-12    416.7Ki ±  0%   416.9Ki ±  0%       ~ (p=0.394 n=6)
Golden/query/promql_full_scan_count-12    408.4Ki ±  0%   408.4Ki ±  0%       ~ (p=0.818 n=6)
Golden/query/promql_cpu_usage_range-12    495.1Ki ±  0%   495.0Ki ±  0%       ~ (p=1.000 n=6)
Golden/density-12                         76.57Mi ±  0%   76.57Mi ±  0%       ~ (p=0.310 n=6)
Golden/logs/write_flush-12                1.472Mi ±  0%   1.365Mi ±  0%  -7.22% (p=0.002 n=6)
Golden/logs/merge-12                      41.98Mi ±  4%   40.11Mi ±  7%       ~ (p=0.310 n=6)
geomean                                   3.956Mi         3.753Mi        -5.13%

                                       │   base.txt   │               head.txt                │
                                       │  allocs/op   │  allocs/op    vs base                 │
Golden/write/head-12                     5.907k ± 26%   5.692k ± 24%        ~ (p=0.810 n=6)
Golden/write/flush-12                    30.76k ±  0%   30.64k ±  0%        ~ (p=0.177 n=6)
Golden/write/concurrent-12               9.552k ± 59%   9.300k ±  3%        ~ (p=1.000 n=6)
Golden/read/fetch_all-12                 4.451k ±  0%   4.451k ±  0%        ~ (p=1.000 n=6) ¹
Golden/read/fetch_all_release-12         2.464k ±  0%   2.464k ±  0%        ~ (p=1.000 n=6)
Golden/read/fetch_recent-12              4.451k ±  0%   4.451k ±  0%        ~ (p=1.000 n=6) ¹
Golden/query/promql_count_cpu_cores-12   1.285k ±  0%   1.285k ±  0%        ~ (p=1.000 n=6)
Golden/query/promql_full_scan_count-12   1.308k ±  0%   1.308k ±  0%        ~ (p=1.000 n=6) ¹
Golden/query/promql_cpu_usage_range-12   1.665k ±  0%   1.666k ±  0%        ~ (p=1.000 n=6)
Golden/density-12                        50.12k ±  0%   50.12k ±  0%        ~ (p=1.000 n=6)
Golden/logs/write_flush-12               13.01k ±  0%   14.66k ±  0%  +12.67% (p=0.002 n=6)
Golden/logs/merge-12                     33.77k ±  0%   35.73k ±  0%   +5.80% (p=0.002 n=6)
geomean                                  6.454k         6.512k         +0.91%
¹ all samples are equal

                                 │  base.txt   │              head.txt               │
                                 │   rows/op   │   rows/op    vs base                │
Golden/read/fetch_all-12           100.0k ± 0%   100.0k ± 0%       ~ (p=1.000 n=6) ¹
Golden/read/fetch_all_release-12   100.0k ± 0%   100.0k ± 0%       ~ (p=1.000 n=6) ¹
Golden/read/fetch_recent-12        10.00k ± 0%   10.00k ± 0%       ~ (p=1.000 n=6) ¹
geomean                            46.42k        46.42k       +0.00%
¹ all samples are equal

                  │  base.txt  │           head.txt            │
                  │  B/point   │  B/point    vs base           │
Golden/density-12   1.498 ± 0%   1.498 ± 0%  ~ (p=1.000 n=6) ¹
¹ all samples are equal

@codecov

codecov Bot commented Jul 23, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.14286% with 20 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
recordengine/bytecol.go 84.37% 14 Missing and 1 partial ⚠️
recordengine/flush.go 37.50% 4 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@tdakkota tdakkota changed the title perf(recordengine): hold a constant byte column as one value, not one per row perf(recordengine): store a run of equal cells once, not per row Jul 23, 2026
@tdakkota
tdakkota marked this pull request as draft July 23, 2026 15:41
@tdakkota @claude
perf(recordengine): intern repeated values in a byte column
00a50f5 
A head buffer is sorted (stream, ts) and columns like a stream's serialized resource attributes or a
severity text take a handful of distinct values over hundreds of thousands of rows, but every row
stored a full copy until the flush encoder's CodecDict deduped it. byteCol now interns by default:
the blob holds the distinct values and a 4-byte id per row indexes them.

A near-unique column — a span id, a trace id, a verbose body — would pay a hash per row for nothing,
so interning is abandoned for good once the distinct bytes exceed half the logical bytes, re-tested
every 512 rows because a column can start repetitive and diverge. The flush buffer opts out entirely:
it feeds an encoder that wants a flat blob, so interning there would pay for the dictionary and the
expansion both. byteSize reports the logical bytes in either form, so accounting is unchanged.

The ts sort permutes an interned column's id index alone, and otherwise through one scratch column
shared across a whole flush rather than a fresh destination per column per stream.

BenchmarkWriteLogs/32svc_2000recs B/op: ingest 73.7 -> 50.0 MB, flush 93.0 -> 77.5 MB. Appending a
repeated 300 B cell is 42 ns/12 B; a unique one 248 ns/1595 B, matching the flat path it falls back
to. Measured on real log parts, interning holds resource in 108 KB where a flat layout needs 149 MB,
while body/trace_id/span_id bail out immediately.

Closes #152
Refs #140

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@tdakkota tdakkota changed the title perf(recordengine): store a run of equal cells once, not per row perf(recordengine): intern repeated values in a byte column Jul 23, 2026
@tdakkota

tdakkota commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Closing: we're choosing CPU over memory here, and memory is the cheaper resource.

Interning is a CPU-for-memory trade — it cuts ingest bytes ~30% but adds ~12% ingest CPU and ~34%
flush CPU (the per-cell hash, and expanding a run back to flat for the encoder). A real-backfill CPU
profile (see #155) shows flush (22%) and merge (29%) already dominate ingest CPU on a box running
~1.7 of 32 cores, so interning adds cost to exactly the subsystems that are the bottleneck, while RAM
sits cheap and plentiful. Wrong trade for this deployment.

#152 is closed alongside as wontfix: storing a repeated byte-column value per row in the head is the
accepted state.

The permuteBytes scratch reuse bundled in this PR is a separate, pure win (from #140) and will be
salvaged independently; the branch perf/head-memory stays for that.

@tdakkota tdakkota closed this Jul 23, 2026
This was referenced Jul 23, 2026
Closed
Merged
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

perf(recordengine): the head stores repeated byte-column values once per row

1 participant

@tdakkota