Skip to content

perf(server-ng): cap sealed-segment read handles with per-partition LRU - #3806

Merged
spetz merged 2 commits into
masterfrom
feat/server-ng-sealed-read-lru
Aug 4, 2026
Merged

perf(server-ng): cap sealed-segment read handles with per-partition LRU#3806
spetz merged 2 commits into
masterfrom
feat/server-ng-sealed-read-lru

Conversation

@hubcio

@hubcio hubcio commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

A lone consumer crossing a sealed segment degraded to full-segment
scans: rotation drops the writer and the resident index, so every poll
re-opened the file and scanned from byte 0. Sealed segments now share
a per-partition read-state handle caching the read fd and the sparse
index reloaded from the .index file, capped by an LRU so descriptors
stay bounded.

Shard allocation defaults back to numa:auto now that multi-shard
server-ng is stable; the previous single-shard default collapsed all
work onto one core.

This is half of the single-node latency parity work; the produce-side
single-pass batch checksum change lives in #3746.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 3, 2026
@hubcio
hubcio force-pushed the feat/server-ng-sealed-read-lru branch from b9b41fb to cff1ba3 Compare August 3, 2026 09:35
A lone consumer crossing a sealed segment degraded to full-segment
scans: rotation drops the writer and the resident index, so every poll
re-opened the file and scanned from byte 0. Sealed segments now share
a per-partition read-state handle caching the read fd and the sparse
index reloaded from the .index file, capped by an LRU so descriptors
stay bounded. Shard allocation defaults back to numa:auto now that
multi-shard server-ng is stable; the previous single-shard default
collapsed all work onto one core.

A reloaded index materializes only under a byte cap; a denser one (an
aggressive flush cadence tracks every message) is binary-searched on
file, so a poll never reads or pins an unbounded index. Purge wipes the
shared read state in place, so a poll suspended across it fails closed
on the recreated files instead of serving purged data via a stale fd.
@hubcio
hubcio force-pushed the feat/server-ng-sealed-read-lru branch from cff1ba3 to 0a1759e Compare August 3, 2026 09:36
@codecov

codecov Bot commented Aug 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.54297% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 32.84%. Comparing base (b452d15) to head (d7b9b3d).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
core/partitions/src/poll_plan.rs 78.48% 14 Missing and 3 partials ⚠️
core/partitions/src/iggy_partition.rs 96.22% 7 Missing and 6 partials ⚠️
core/partitions/src/log.rs 97.36% 4 Missing and 1 partial ⚠️
core/partitions/src/iggy_index_reader.rs 96.61% 0 Missing and 4 partials ⚠️
core/partitions/src/iggy_partitions.rs 50.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3806       +/-   ##
=============================================
- Coverage     76.01%   32.84%   -43.17%     
  Complexity      969      969               
=============================================
  Files          1325     1323        -2     
  Lines        162380   141977    -20403     
  Branches     135260   114935    -20325     
=============================================
- Hits         123427    46637    -76790     
- Misses        35291    94127    +58836     
+ Partials       3662     1213     -2449     
Components Coverage Δ
Rust Core 21.93% <94.54%> (-54.01%) ⬇️
Java SDK 62.71% <ø> (ø)
C# SDK 71.13% <ø> (-1.14%) ⬇️
Python SDK 89.45% <ø> (ø)
PHP SDK 84.52% <ø> (ø)
Node SDK 96.24% <ø> (+0.08%) ⬆️
Go SDK 43.08% <ø> (ø)
Files with missing lines Coverage Δ
core/server-ng/src/dispatch.rs 40.00% <ø> (ø)
core/partitions/src/iggy_partitions.rs 70.02% <50.00%> (-0.34%) ⬇️
core/partitions/src/iggy_index_reader.rs 87.19% <96.61%> (+87.19%) ⬆️
core/partitions/src/log.rs 68.55% <97.36%> (+37.00%) ⬆️
core/partitions/src/iggy_partition.rs 64.27% <96.22%> (+8.09%) ⬆️
core/partitions/src/poll_plan.rs 73.07% <78.48%> (+8.62%) ⬆️

... and 733 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@spetz
spetz merged commit c51b754 into master Aug 4, 2026
100 checks passed
@spetz
spetz deleted the feat/server-ng-sealed-read-lru branch August 4, 2026 09:50
@github-actions github-actions Bot removed the S-waiting-on-review PR is waiting on a reviewer label Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants