JetStream file-store ~65x slower than memory-store on KV mirror (83% of seq space is deleted)

[cyqsign]

Environment

• nats-server: v2.14.2 (both hub and leaf)
• Leaf disk: Intel enterprise SATA SSD (SSDSC2KB960G7) — /data is on sda5
• Leaf RAM: 503 GB total, ~479 GB free during test
• Topology: 3-node hub cluster + single leaf node connected via leafnodes (no gateway)
• Test consumer: Go 1.22 + nats.go v1.39.1 (small standalone tool, code shown below)

Stream layout

The hub owns the authoritative KV bucket KV_DNS (file storage). We create two mirrors on the same leaf machine, differing only in storage:

• KV_DNS — storage: memory
• KV_DNS_FILE — storage: file

Both mirrors share the same source config:

Retention:           Limits
MaxMsgsPerSubject:   1
AllowDirect:         true
MirrorDirect:        true
AllowRollup:         true
Replicas:            1
Mirror.External:     $JS.hub.API  →  KV_DNS

Both end up with identical stream state after sync:

Messages:          2,019,878
Bytes:             5.9 GiB
First Sequence:    1
Last Sequence:     11,920,111
Deleted Messages:  9,900,233     <-- 83% of the seq space is holes
Subjects:          2,019,878

The bucket is a KV, so every Put on an existing key erases the previous seq and creates a new one (see filestore.go:4966-4989 — with MaxMsgsPer=1, the old sequence for a subject is removed via removeMsgViaLimits and a hole is left in the seq space). Over time this produces a large sparse seq space (~12 M seqs spanned, of which ~9.9 M are deleted holes and only ~2 M are live).

Observation 1: consumer throughput differs ~65x

Same machine, same client, same consumer code — only the mirror's storage differs. A plain pull consumer:

// Effective options passed to nats.go
consumer, _ := stream.CreateOrUpdateConsumer(ctx, jetstream.ConsumerConfig{
    DeliverPolicy:     jetstream.DeliverLastPerSubjectPolicy,
    AckPolicy:         jetstream.AckNonePolicy,
    FilterSubject:     "$KV.DNS.>",
    InactiveThreshold: time.Hour,
    Replicas:          1,
})

iter, _ := consumer.Messages(
    jetstream.PullMaxBytes(4 * 1024 * 1024),        // 4 MB per batch
    jetstream.PullExpiry(5 * time.Second),
    jetstream.PullThresholdMessages(500),
)

for {
    msg, err := iter.Next()
    if err != nil { break }
    total.Add(1)
    totalSize.Add(uint64(len(msg.Data())))
}

Memory-store mirror:

[   2.0s] total=696868   cur=348430 msg/s   113.05 MB/s
[   4.0s] total=979053   cur=141093 msg/s   789.42 MB/s
[   6.0s] total=1760092  cur=390519 msg/s   617.37 MB/s

Initial snapshot complete: 2,019,878 msgs in 7.57 s
Throughput: 266,840 msg/s (765 MB/s)

File-store mirror (everything else identical):

[   2.0s] total=11489  cur=5744 msg/s   1.60 MB/s
[   4.0s] total=18049  cur=3280 msg/s   0.85 MB/s
[   6.0s] total=21179  cur=1565 msg/s   0.60 MB/s
[   8.0s] total=37436  cur=8129 msg/s   0.58 MB/s
[  10.0s] total=44481  cur=3522 msg/s   0.50 MB/s
[  12.0s] total=52425  cur=3972 msg/s   0.44 MB/s
[  14.0s] total=63275  cur=5425 msg/s   0.41 MB/s
[  16.0s] total=66280  cur=1502 msg/s   0.36 MB/s
[  18.0s] total=75967  cur=4844 msg/s   0.34 MB/s

Sustained ~4 k msg/s, ~0.5 MB/s. The per-window rate is very lumpy (1.5 k → 8 k → 3.5 k …), which we suspect corresponds to blocks of varying seq density.

Observation 2: mirror sync from hub is also slow on file-store

When we first create the file-store mirror, initial sync from hub reports (excerpt):

mirror sync: msgs=855,017  lag=1,164,860  elapsed=412.5 s

That's roughly 2 k live-msg/s reaching the leaf (~12 k seq/s counting deletes). The memory-store mirror finishes syncing the same 2 M messages in seconds.

What we ruled out

• Disk I/O: during file-store consumption, iostat -xdm 1 on sda (SSD carrying /data) shows %util 0.00 – 0.30, r/s 0 – 24, rMB/s ≈ 0. The disk is idle.
• Page cache: pre-warming with cat msgs/*.blk > /dev/null (~6 GB into ~479 GB free RAM) does not change the consumer rate.
• CPU saturation (single-thread): top -H shows 2–3 nats-server threads active at any moment (73 % / 20 % / 13 %, next snapshot 87 % / 13 %, next 56 % / 25 % / 12 %). Aggregate ~100–120 % of one core; overall system is 98 % idle across 80 CPUs. Not a single stuck goroutine, but also not close to saturating a core.
• DeliverLastPerSubject cost: with MaxMsgsPerSubject=1, the server takes the fast path at consumer.go:6220-6227, setting o.sseq = state.FirstSeq. So this is effectively a sequential scan of the seq space, not a per-subject lookup.

Where we think the bottleneck may be

Reading the file-store path, this is what we see (best guess, not verified with a profile yet):

1. fileStore.LoadNextMsg (filestore.go:9155) walks blocks starting at the containing one, calling mb.firstMatching per block.
2. mb.firstMatching (filestore.go:2965) acquires an exclusive mb.mu.Lock() per call, then does:

   for seq := fseq; seq <= lseq; seq++ {
       if mb.dmap.Exists(seq) { continue }   // AVL Exists per deleted seq
       fsm, err := mb.cacheLookup(seq, sm)
       ...
   }

3. compactWithFloor (filestore.go:6162-6169) only prunes dmap entries at each block's head/tail edges. Interior deleted-seq entries appear to remain forever.

So even though physical compact packs the bytes well on the leaf side (on-disk 6.0 GiB vs live state.Bytes 5.89 GiB → ~98% of the disk is live bytes; see Attachment 3 for hub vs leaf comparison), each LoadNextMsg still has to spin through the interior seq holes under an exclusive block lock. With 83 % of the seq space being holes, that adds up.

We don't fully understand why the total gap is 65× — we'd expect the AVL + lock overhead to hurt but not by that much. Something else in the delivery path may be contributing that we haven't traced yet.

Questions

1. Is the firstMatching / dmap.Exists path actually the dominant cost here, or should we be looking elsewhere?
2. Is there a reason interior dmap entries can't be cleaned up when their surrounding seqs are physically gone? Are there invariants (replication, consumer state, Nats-Expected-Last-Sequence, etc.) that require keeping them?
3. Are there any config knobs today (sync_interval, compression, block size, cache expiry) that would materially close this gap on file-store?
4. Or is this simply the expected cost for a KV file-store with a heavily-recycled key space, and the intended pattern is what we're already using on the mem-mirror side (authoritative file-store on the hub, hot reads served from a mem-mirror on the leaf)?

Happy to gather more data — a CPU profile, a minimal reproducer, or run whatever experiments would help.

Attachment 1 — top -H during file-store consumption (3 snapshots, 2 s apart)

--- snapshot 1 ---
top - 17:42:12 up 17 days, load average: 0.59, 0.16, 0.08
Threads:  87 total,   1 running,  86 sleeping
%Cpu(s):  1.3 us,  0.2 sy,  0.2 ni, 98.2 id,  0.0 wa

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
17183 root      20   0   21.9g   9.4g  14808 S 73.3  1.9   2:34.81 nats-server
17330 root      20   0   21.9g   9.4g  14808 R 20.0  1.9   0:55.08 nats-server
 8574 root      20   0   21.9g   9.4g  14808 S 13.3  1.9   1:05.90 nats-server

--- snapshot 2 ---
%Cpu(s):  1.3 us,  0.0 sy,  0.1 ni, 98.6 id,  0.0 wa

17330 root      20   0   21.9g   9.4g  14808 S 86.7  1.9   0:55.92 nats-server
17183 root      20   0   21.9g   9.4g  14808 R 13.3  1.9   2:35.16 nats-server

--- snapshot 3 ---
%Cpu(s):  1.3 us,  0.1 sy,  0.1 ni, 98.5 id,  0.0 wa

17330 root      20   0   21.9g   9.4g  14808 R 56.2  1.9   0:56.44 nats-server
17183 root      20   0   21.9g   9.4g  14808 S 25.0  1.9   2:35.94 nats-server
17398 root      20   0   21.9g   9.4g  14808 S 12.5  1.9   2:10.03 nats-server

Attachment 2 — iostat -xdm 1 5 during file-store consumption

Device:  rrqm/s  wrqm/s   r/s   w/s   rMB/s   wMB/s   avgqu-sz   %util
sda        0.00    0.00  0.00 10.00    0.00    0.04       0.00    0.00
sda        0.00   28.00  0.00 24.00    0.00    0.21       0.00    0.30
sda        0.00    0.00  2.00 10.00    0.03    0.04       0.00    0.00
sda        0.00    0.00  0.00 12.00    0.00    0.05       0.00    0.20

sda5 is where /data/jetstream-leaf/... lives. Disk is essentially idle.

Attachment 3 — on-disk block layout: hub source vs leaf file mirror

For the same ~2 M live messages (identical state.Bytes ≈ 5.89 GiB), the physical layout on the two sides differs significantly.

Hub (authoritative KV_DNS, file storage):

$ ls -alh /data/jetstream/jetstream/$G/streams/KV_DNS/msgs/ | wc -l
5041                                    # ≈ 5037 .blk files + index.db + entries

$ du -sh /data/jetstream/jetstream/$G/streams/KV_DNS/msgs/
7.3G   /data/jetstream/jetstream/$G/streams/KV_DNS/msgs/

Ratio: live 5.89 GiB / on-disk 7.3 GiB ≈ 81 % — the source has accumulated many small, sparsely-populated blocks over time (many <1 MB block files left after in-place compact).

Leaf (file mirror KV_DNS_FILE, same source):

$ ls -alh /data/jetstream-leaf/jetstream/$G/streams/KV_DNS_FILE/msgs/ | wc -l
1600                                    # ≈ 1597 .blk files + index.db + entries

$ du -sh /data/jetstream-leaf/jetstream/$G/streams/KV_DNS_FILE/msgs/
6.0G   /data/jetstream-leaf/jetstream/$G/streams/KV_DNS_FILE/msgs/

Ratio: live 5.89 GiB / on-disk 6.0 GiB ≈ 98 % — the mirror re-packs the same data into ~3× fewer blocks with almost no dead bytes, because it replays messages sequentially rather than in-place.

Comparison:

	Hub source	Leaf file mirror
Block count (approx)	~5037	~1597
On-disk total	7.3 G	6.0 G
Live-bytes / on-disk	~81 %	~98 %

Consumer throughput is ~4 k msg/s on both. So neither the number of blocks nor the on-disk fragmentation appears to explain the slowness on its own.

[cyqsign]

@derekcollison @kozlovic Hi both, would love to hear your thoughts on this.

[derekcollison]

@neilalexander or @MauriceVanVeen could supply some guidance here.

[cyqsign]

Thanks @derekcollison, appreciate the quick redirect. Looking forward to hearing from @neilalexander or @MauriceVanVeen when they have a moment.

[MauriceVanVeen]

I'm working on an improvement in the filestore for a situation like this where the number of deletes are greater than the number of unique subjects.

In the meantime, could you test with the consumer's FilterSubject removed? It seems that you're not actually intending to filter and instead want to consume everything as fast as possible. If that is significantly faster/on par with the memstore, then we'll know what's going on.

[cyqsign]

Thanks so much for the detailed explanation, Maurice — this really cleared things up for me.

then the server always takes the (for your use case) cheaper linear scan path.

Thanks for the guidance. From what I understand now, the linear scan processes each file block (*.blk) by iterating through sequences in order within that block.

Let me confirm my understanding with a concrete example: suppose 1 file block holds 1,000 sequences. If 1.blk covers seq 1~999, and seqs 1000 ~ 1999 are all holes (deleted), then what would be the starting seq of 2.blk?

• If 2.blk starts at 2000, then the hole-skipping granularity is effectively at the file-block level — the scan jumps over the entire 1000~1999 hole range by skipping to the next block.

• If 2.blk starts at 1000, then the sequential scan would have to walk through a large stretch of holes, and as the seq space grows, full-read performance would degrade noticeably.

Which of these is closer to how the block layout actually works?

A script in Go that sets everything up and observes the difference between filestore and memstore would be fine

I haven't had a chance to rebuild a minimal reproduction yet. I'll open a separate issue when I'm able to put together a clean reproducer.

event.foo.> would delete messages with subject event.foo.a, event.foo.b, event.foo.c, etc.

Following your suggestion, I used "event.foo.>" for the rollup, and it successfully deleted the data under event.foo.a, event.foo.b, etc. However, without the "Nats-TTL": {"5"} header, the "event.foo.>" subject itself remains as a persistent record on the stream — which looks a bit odd since its only purpose was to trigger the deletion.

Here's how I'm doing it:

resp = batchCommit(nc, "b2", 2, "event.foo.>", "ROLLUP-ALL-FOO", nats.Header{
    "Nats-Rollup": {"sub"},
    "Nats-TTL":    {"5"},
})

And the helper:

func batchCommit(nc *nats.Conn, batchID string, seq int, subj, data string, extra nats.Header) string {
    m := nats.NewMsg(subj)
    m.Data = []byte(data)
    m.Header.Set("Nats-Batch-Id", batchID)
    m.Header.Set("Nats-Batch-Sequence", strconv.Itoa(seq))
    m.Header.Set("Nats-Batch-Commit", "1")
    for k, vals := range extra {
        for _, v := range vals {
            m.Header.Add(k, v)
        }
    }
    resp, err := nc.RequestMsg(m, 10*time.Second)
    check("batchCommit", err)
    return string(resp.Data)
}

Is there a more elegant way to tell nats-server "this is purely a delete operation, don't persist it on the stream," or is using TTL the recommended approach for these rollup marker messages?

[MauriceVanVeen]

If 1.blk covers seq 1~999, and seqs 1000 ~ 1999 are all holes (deleted), then what would be the starting seq of 2.blk?

2.blk would start at sequence 2000. 1.blk would start at sequence 1, and have deleted entries stored in its mb.dmap from 1000-1999. When that block is compacted it removes deletes at the tail. A block never starts with deleted messages, it can only have interior deletes between messages, or deletes at the tail that can be compacted. That is the range of the linear scan.

I used "event.foo.>" for the rollup ... "this is purely a delete operation, don't persist it on the stream,"

I wouldn't recommend publishing a message with a rollup for this, since you don't want that message to be persisted. Using an atomic batch that immediately commits also seems like more overhead than just publishing it directly?

What you'd want instead is js.PurgeStream, you can pass it a purge subject of event.foo.>. Purges the stream the same way, and doesn't store a separate message to do it.

[cyqsign]

It's not just about cleanup — it's an atomic dataset swap. The stream holds two "views" of the data (e.g. event.zone.> and event.foo.>), and a task needs to atomically replace one with the other: publish event.foo.a, event.foo.b, and delete event.zone.> all in one shot. If I do PurgeStream separately from the publishes, there's a window where either both datasets coexist (confusing consumers) or neither exists (data loss). The batch with rollup guarantees consumers see the switch as a single atomic transition — foo appears and zone disappears at the same commit point. That's the property I need, and PurgeStream as a separate API call can't provide it. I don't know if I'm saying this correctly.

[MauriceVanVeen]

Yeah, there is no way to swap distinct subjects like that. You can do an atomic swap with a rollup, but only for replacing that same subject. Might be worth to check if your subjects can be redesigned in a way where rollup on the subject works. Say N writes to subject foo, and then you collapse them down/snapshot them into one message that rolls up the previous messages on that same subject.

Otherwise you'll indeed need something like publishing to the subject purely to do the rollup and then let it age out via TTL. I'd recommend a different subject (hierarchy) design over that approach though.

[cyqsign]

@MauriceVanVeen Thank you for the thorough and clear explanation — this is extremely helpful. I'll close this discussion now since the root cause has been identified and the fix is clear. Thanks again to Maurice for the guidance!