[Bug] BanyanDB 0.10.1 trace merge: "offset must be equal to bytesRead" panic in part_iter, crashes the process

[Felix-wave]

Search before asking

• I had searched in the issues and found no similar issues. (Different from the related [Bug] BanyanDB trace block_writer panics on out-of-order timestamps within same traceID #13860 — that one is in the trace write path and is recovered by the gRPC interceptor; this one is in the trace merge/read path and crashes the process.)

Apache SkyWalking Component

BanyanDB

What happened

After upgrading from apache/skywalking-banyandb:0.9.0 to 0.10.1 (with OAP 10.4.0), BanyanDB crashes the process every ~7-8 minutes with:

panic: offset 1400877 must be equal to bytesRead 1400490

Unlike the timestamp-ordering panic in #13860 (which is recovered by grpc-middleware), this one fires from a background mergeLoop goroutine that is not wrapped by recovery, so the process exits and the pod restarts.

Full stack

goroutine 3900 [running]:
github.com/apache/skywalking-banyandb/pkg/logger.Panicf(...)
github.com/apache/skywalking-banyandb/banyand/trace.(*partMergeIter).mustReadRaw(0xc001ac4000, 0xc002d716b8, 0xc001ac4118)
    /mnt/d/skywalking-banyandb/banyand/trace/part_iter.go:359 +0xf5
github.com/apache/skywalking-banyandb/banyand/trace.(*blockReader).mustReadRaw(...)
    /mnt/d/skywalking-banyandb/banyand/trace/block_reader.go:263
github.com/apache/skywalking-banyandb/banyand/trace.mergeBlocks(...)
    /mnt/d/skywalking-banyandb/banyand/trace/merger.go:421 +0x79e
github.com/apache/skywalking-banyandb/banyand/trace.(*tsTable).mergeParts(...)
    /mnt/d/skywalking-banyandb/banyand/trace/merger.go:344 +0x42a
github.com/apache/skywalking-banyandb/banyand/trace.(*tsTable).mergePartsThenSendIntroduction(...)
    /mnt/d/skywalking-banyandb/banyand/trace/merger.go:118 +0x145
github.com/apache/skywalking-banyandb/banyand/trace.(*tsTable).mergeSnapshot(...)
    /mnt/d/skywalking-banyandb/banyand/trace/merger.go:104 +0x125
github.com/apache/skywalking-banyandb/banyand/trace.(*tsTable).mergeLoop.func1(...)
    /mnt/d/skywalking-banyandb/banyand/trace/merger.go:78 +0x1f9
github.com/apache/skywalking-banyandb/banyand/trace.(*tsTable).mergeLoop(...)
    /mnt/d/skywalking-banyandb/banyand/trace/merger.go:90 +0x271
created by github.com/apache/skywalking-banyandb/banyand/trace.(*tsTable).startLoop in goroutine 157
    /mnt/d/skywalking-banyandb/banyand/trace/tstable.go:130 +0x246

Source location (apache/skywalking-banyandb v0.10.1)

banyand/trace/part_iter.go:354-365:

func (pmi *partMergeIter) mustReadRaw(r *rawBlock, bm *blockMetadata) {
    r.bm = bm
    // spans
    if bm.spans != nil && bm.spans.size > 0 {
        // Validate the reader is aligned to the expected offset
        if bm.spans.offset != pmi.seqReaders.spans.bytesRead {
            logger.Panicf("offset %d must be equal to bytesRead %d", bm.spans.offset, pmi.seqReaders.spans.bytesRead)
        }
        ...
    }
    ...
}

So the merger sequentially reads spans from seqReaders.spans, and a per-block bm.spans.offset is expected to match how far the seqReader has advanced (bytesRead). When they diverge — by 387 bytes in our sample — the merger panics. The same pattern (offset must be equal to bytesRead) appears at:

• banyand/trace/block.go:196 (tag metadata)
• banyand/trace/block.go:329 (span data)
• banyand/internal/sidx/block.go
• banyand/measure/block.go
• banyand/stream/block.go

So the invariant is repeated across the new (0.10) trace storage engine.

Cadence and impact

In our cluster, BanyanDB pod restarted 126 times in 17 hours = roughly once every 8 minutes. Every time, OAP loses connection to BanyanDB and hot-loops crash too (~148 OAP restarts in the same window). Net effect: rolling availability — every ~8 minutes there is a 1-2 minute window where ingestion and queries fail.

For comparison, on 0.9.0 the only panic we saw fired ~once every 28 minutes. 0.10.1 is significantly less stable on our workload, primarily because of this new panic in the merger.

What you expected to happen

The merger should not panic on what is clearly a corrupted or out-of-sync block metadata. Reasonable options (maintainers know best):

1. Skip the offending block with a warning instead of Panicf — at minimum, contain the blast radius to one block instead of restarting the whole DB.
2. Restart the seqReader to the offset declared in bm.spans.offset (or vice versa) when divergence is detected — assumes the metadata is the source of truth.
3. Fail the merge of the affected part but keep the process running and let retention/cleanup eventually drop the corrupted part.

How to reproduce

Steady-state SkyWalking deployment, OAP forwarding traces to standalone BanyanDB. We see this on:

• BanyanDB: apache/skywalking-banyandb:0.10.1
• SkyWalking OAP: apache/skywalking-oap-server:10.4.0
• ~30+ Java services, apache-skywalking-java-agent 9.5.0, JDK 21
• Standalone BanyanDB on Kubernetes (Aliyun ACK), --trace-root-path=/data/trace
• 51 GB cumulative on disk after 17h of ingest (stream 38.5G + trace 12.9G + measure 26M)

The very first occurrence happens within ~30 minutes of starting fresh (after fully wiping /data and letting OAP recreate schemas). After that, panic cadence stabilizes at ~8 minutes.

Anything else

This bug is in the new trace storage engine introduced by #713 in 0.10.0; we did not see this panic on 0.9.0 (which uses the older trace path).

We have already reported the related — but distinct — timestamp-ordering panic in the write path as #13860 (recoverable, not crashing the process). Filing this one separately because the failure mode (background merge goroutine, no recovery, full process exit) is different and arguably more disruptive.

Happy to gather more samples (full stack traces over time, sample part dumps if a tool exists, sysrq dumps, anything) on request.

Are you willing to submit a pull request to fix on your own

• Yes, I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[hanahmily]

According to versions.yml in v0.10.1, the required file version is 1.4.0. This should be incompatible with version 1.3.0 (used in 0.9.0).

The database is expected to check the file version during bootstrap and stop if it is unsupported. If you managed to start 0.10.1 using 0.9.0 data files, it suggests the version check process is not functioning correctly.

[Felix-wave]

Thanks for the quick reply, @hanahmily. To verify the version-mismatch hypothesis, we ran a controlled experiment with a fully clean state on both of our environments (KL Prod and SG Dev):

1. Scaled OAP and BanyanDB to 0.
2. Deleted the PVC entirely, which (with reclaimPolicy: Delete and Aliyun ESSD CSI) cascaded to PV deletion and physical disk deletion. Confirmed both Aliyun disks were gone:
   • KL: d-8ps1hxrabeggd4foci5p → deleted
   • SG: d-t4niiqzwz9f8wke16x97 → deleted
3. Created a fresh PVC of the same size. Storage class alicloud-disk-essd provisioned brand-new disks (d-8ps458950s8goggjqiso for KL, d-t4ngw4m8qhuusgrtvnod for SG).
4. Scaled BanyanDB 0.10.1 + OAP 10.4.0 back up. Verified /data was empty on first boot and OAP recreated all groups + schemas from scratch.

So at this point there is no possibility of 0.9.x file residue — the disk itself is new, never touched by 0.9.

Initial observation (looked promising)

For roughly the first 2 hours, the merger panic was absent on both clusters:

KL Prod  (fresh disk, 2h6m):  trace 25.5G / stream 33G   #13861 = 0
SG Dev   (fresh disk, 1h):    trace 166M  / stream 5.8G  #13861 = 0

Notably, KL had already accumulated 25.5 GB of trace data — about 2× the 12.9 GB it had on the old disk when this panic was occurring every ~8 minutes — yet zero offset must be equal to bytesRead panics during this window.

After more time, the panic returned

I let both clusters run overnight. ~17 hours after starting fresh:

KL Prod  (fresh disk, 17h):
  banyandb restart=41
  most recent panic:  "offset 0 must be equal to bytesRead 63954"
  pod state: CrashLoopBackOff
  disk used: 528 GB

SG Dev   (fresh disk, 16h):
  banyandb restart=70
  most recent panic:  "offset 5263206 must be equal to bytesRead 5262950"
  disk used: 62 GB

So the panic does eventually fire on a brand-new disk; it just takes longer than ~2 hours to reach whatever condition triggers it. Restart cadence on the fresh disk in steady state for KL is roughly once every 25 minutes (vs ~8 min on the old disk — possibly a function of accumulated state, but still not stable).

What this rules out

The "0.9.x file residue + missing version check" hypothesis does not explain the failure: the failure reproduces on disks that were never written to by 0.9.x.

What it suggests

The panic appears tied to runtime state that 0.10.1 itself produces (merger output? compaction? specific tag layouts?), not to legacy file format. Happy to dig further — e.g., capture a part dump just before the next panic if there's a tool for it, or attach the full panic context with a few hundred surrounding log lines.

Should I file this as a separate finding, or treat it as a continuation of this issue? Also, is the version-check process itself worth a separate ticket, since you mentioned it might not be functioning correctly?

[wu-sheng]

Is that test data? could you share the data file(folder) with us to verify?

[Felix-wave]

Hi @wu-sheng — this is real production traffic, not test data. Trace payloads contain customer data (full SQL bodies, HTTP request URLs, internal service/endpoint names, infrastructure identifiers), so I'm afraid we can't share the raw /data directory or BanyanDB part folders directly.

Happy to collaborate in any other way that helps you reproduce. Some options I can offer, ordered roughly by what should be easiest:

1. Workload profile, in detail. I can describe the trace ingestion shape (services, agents, average trace depth, tag distributions, peak trace/sec, segment size distribution) so you can drive a synthetic generator to a comparable load. We're in the millions of segments/day range across ~30 Java services on agent 9.5.0.

2. Redacted / anonymized dumps. If there's an existing tool in BanyanDB (or one you can point me to / send me a snippet for) that dumps only structural metadata of a part (offsets, sizes, tag-family headers, block boundaries — no tag values, no span bytes), I can run it and share the output. The bug seems to be in offset/bytesRead alignment, so structural metadata may already be sufficient to diagnose.

3. Run an instrumented build for you. If you publish a debug image/binary with extra logging around block_writer/mergeBlocks/partMergeIter.mustReadRaw — e.g., logging the offsets, bytesRead, traceID, and the blockMetadata being written / read just before each panic — I can run it on KL Prod (high traffic) until we capture a panic and then share the redacted log snippet. We're already seeing this every ~25 min on a fresh disk, so iteration is fast.

4. Live triage session. I can pull arbitrary state from the cluster on demand: pod logs, banyandb HTTP API responses, du / find output of /data, schema dumps, OAP startup logs, etc. Tell me what you'd like to see and I'll capture it.

5. Synthetic reproducer. If we can pin down what's special in our traffic (e.g., particular tag shape, specific segment patterns), I can build a small load generator that drives the same pattern against a sandbox BanyanDB and share that.

Which of these would be the most useful starting point? Option 3 (an instrumented build) feels highest-signal given the failure mode is clearly inside the trace storage engine's offset bookkeeping, but I'll defer to whatever you think is most efficient.

[hanahmily]

Thanks for the clean fresh-disk experiment, @Felix-wave — that pinned down "this is not 0.9 file residue."

Update from this side: I have a unit test that reproduces the exact panic message verbatim by truncating spans.bin at a block boundary while leaving metadata.json/primary.bin intact:

panic: offset 5300 must be equal to bytesRead 5247

Same shape as your offset 1400877 must be equal to bytesRead 1400490. The on-disk state required to trigger it is what you'd see after a hard kill (SIGKILL/OOM/eviction) mid-merge: BanyanDB's merge write path doesn't fsync data files before writing metadata.json, so a kill between writeback and metadata commit leaves a torn part that survives across restarts and re-trips the panic on every wake-up of the merge loop. That matches the ~8-minute crash cadence in your cluster — once you have one torn part, the loop perpetuates.

Audit findings + proposed fix: #13862.

The one piece I haven't been able to pin down from your report is what triggers the first tear on a fresh disk — your "first occurrence ~30 min after wipe, then ~25 min cadence" doesn't square with a self-perpetuating cycle alone, since the cycle needs an initial torn part to start. Two quick questions that would discriminate:

1. Pod exit history before the first panic. Could you run, on one of the crashing pods:

   kubectl get events -n <ns> --sort-by='.lastTimestamp' | grep banyandb
   kubectl describe pod <banyandb-pod> | grep -A5 "Last State"
   

   What I'm looking for: any OOMKilling, Evicted, Killing, or BackOff events before the first panic-driven exit, and the Reason on the previous termination. If we see OOMKilled or Evicted, that's the first-cause torn write.

2. Memory limits + utilization. What's the BanyanDB pod's memory request/limit, and do you have a memory-utilization graph for the first 30-60 minutes of a fresh-disk run? OAP backfill into a fresh BanyanDB hits the merge loop hard right after schema creation; if the pod is brushing the limit there, an OOMKill is the most plausible first trigger.

Independent of what turns out to be the first cause, the durability + read-side fixes proposed in #13862 will stop the perpetuation: once they land, even a torn write from an OOMKill won't take down the merger — the affected part is quarantined and the loop continues.

Happy to share the reproducer test (it's a single self-contained Go test against tst.mergeParts) if you'd like to run it locally to convince yourself the panic shape matches.

[Felix-wave]

Brilliant root-cause work, @hanahmily — the torn-spans.bin reproducer matches our panic shape character-for-character, and the audit in #13862 lines up with the symptoms (perpetuates across restarts, hits all four storage engines that share the same merge writer). The two-phase commit + quarantine plan looks right.

Here are the diagnostics you asked for. Spoiler: I don't think OOMKill or eviction is the first-cause tear in our setup, but I have an alternative hypothesis below worth considering.

1. Pod exit history

Both clusters' BanyanDB pods exit cleanly via panic → process exit, not SIGKILL or eviction:

KL Prod:
  restartCount: 67  (over ~21h on the fresh disk)
  current state: waiting (CrashLoopBackOff)
  lastTerminated: reason=Error  exitCode=2  (Go runtime panic, NOT 137/OOMKilled)
  pod podStartTime: 2026-05-06T09:35:47Z (uninterrupted; same pod object the whole run)

SG Dev:
  restartCount: 70  (over ~20h)
  lastTerminated: reason=Error  exitCode=2
  pod podStartTime: 2026-05-06T10:41:07Z (uninterrupted)

Both pods are still the original ones from the fresh-disk start — same metadata.creationTimestamp, same UID. Just the container has been restarting in place.

kubectl get events is no help for the first panic (Kubernetes evicts events after ~1h by default and we're 21h in), but the recent windows show only Unhealthy (readiness probe vs. a DB that's mid-restart) and BackOff — no OOMKilling, no Evicted, no Killing. Pod-level restartCount only counts container restarts, and on every previous termination the reason has been Error (Go panic), not OOMKilled.

2. Memory limits + utilization

KL Prod  banyandb container:  requests cpu=2 memory=16Gi   limits cpu=8 memory=32Gi
SG Dev   banyandb container:  requests cpu=1 memory=8Gi    limits cpu=4 memory=16Gi

Live kubectl top snapshot:

SG Dev   banyandb:   2464m CPU, 4760Mi RSS  (≈ 30% of the 16Gi limit, ≈ 60% of the 8Gi request)
KL Prod  banyandb:   metrics-server returns "Metrics not available, age: 20h36m" — pod has been
                     restarting too aggressively for metrics-server to settle on a sample.
                     I'll capture this on the next clean restart and update.

We don't have a Prometheus retention long enough to cover the first 30-60 min after the wipe; if it would be useful, I can wipe one cluster again and run kubectl top --containers on a 5s loop for the first hour and share the trace.

3. An alternative hypothesis for the first tear

Given that exit code 2 is the only termination reason we ever see and there's no OOMKill/eviction signal, I don't think the first torn part comes from an external kill. My guess:

The timestamp panic from #13860 fires inside the merge goroutine, gets recovered by an inner gRPC recovery interceptor (not the merger's own goroutine), and leaves the merge mid-flight — torn spans.bin with metadata.json not yet written.

The reasoning:

• The very first cannot contain timestamp smaller than panic on a fresh KL disk fires within a few minutes of OAP starting to ingest, well before any of the offset panics.
• Cadence on a fresh KL disk: ~7 timestamp panics in the first 31 min, ~31 in the first 2h, all caught by grpc-middleware/v2 recovery (no process exit). Per-environment frequency tracks ingest volume (KL ≈ 4 min/panic at high traffic, SG ≈ 5 min/panic at dev traffic).
• The first offset panic only shows up after the timestamp panic has fired many times. After that it self-perpetuates exactly as [Bug] BanyanDB trace/measure/stream/sidx merge: missing fsync makes torn parts perpetuate CrashLoopBackOff (root cause for #13861) #13862 describes.

If the timestamp panic ever propagates from a write goroutine into a merge-write code path (or shares any I/O state with the merge writer's seqWriters), the torn-write window in the audit is open. This would also explain why the latency to the first offset panic correlates loosely with traffic: more timestamp panics → more chances to land one mid-merge.

I haven't proven this — it's a hypothesis from observing the panic ordering. Two ways to discriminate:

a. Add a panic-counter and timestamp to the merge goroutine. If timestamp panics are observed inside mergeParts / mustWriteBlock (i.e., the merge writer's own code path, not the gRPC write handler), this hypothesis is confirmed.
b. Land #13860's check as a recoverable error in the merge writer too, not just a Panicf. Independent of #13862, that closes a class of "merge goroutine raised a panic from inside an MustWriteBlock call and damaged its own output."

4. Help we can offer going forward

• I can run an instrumented build on KL Prod whenever you have one ready (e.g., one that logs (traceID, offsets, bytesRead, file path) at every seqWriter open/close and at every panic site). Iteration time is ~25 min between panics on a fresh disk in KL.
• I can run the standalone reproducer test you described (Test_merger_tornWriteRepro in banyand/trace/merger_repro_test.go) locally to confirm we reproduce the exact same panic shape — please link the branch when convenient.
• I can take controlled clean wipes on either KL or SG to characterize the first-tear timing as needed.

In the meantime, given that the issue is now well-understood and a fix is in flight, we're going to roll back to BanyanDB 0.9.0 + OAP 10.3.0 to restore monitoring availability. We'll be ready to pick up an 0.10.x release that includes #13862 whenever it's available, and to verify the fix on the same workload that exposed it.

[Felix-wave]

Quick update with strong evidence that PR #1114 (the trace-write timestamp panic fix) was the missing piece for stability on our workload.

Setup

I built a custom image that takes v0.10.x (which is already at v0.10.2) and cherry-picks the e810640 patch (PR #1114). Then I wiped both /data directories completely (as recommended by your torn-write analysis) and brought up KL Prod on this image.

banyandb image: skywalking-banyandb v0.10.x + cherry-pick e810640  (PR #1114)
OAP image:      apache/skywalking-oap-server:10.4.0  (unchanged)
disk:           wiped clean before start

Result: 12 hours, zero panics, zero restarts

KL Prod  banyandb (12h after fresh start):
  pod restartCount  = 0
  #13860 timestamp panic                  = 0
  #13861 offset/bytesRead panic           = 0
  total panic events                      = 0
  errors                                  = 0  (excluding the 'watcher channel full' info noise)
  data ingested                           = stream 46.6 GB / trace 98.1 GB / measure 55.5 MB

For comparison on the same cluster / workload over similar windows:

Build	Run length	Pod restarts	#13860 timestamp panic	#13861 offset panic
0.9.0 + OAP 10.3.0	17 h	17 h / ~28 min ≈ 36 (we measured ~406 over 7 days)	many (process exit)	n/a (different storage engine)
0.10.1 + OAP 10.4.0 (fresh disk)	17 h	41	many	41 (process exit)
0.10.2 + OAP 10.4.0 (fresh disk)	80 min before SG hit a backgroud-goroutine timestamp panic that wasn't caught by gRPC recovery	1	several	0 (good — race fix in 0.10.2 worked)
v0.10.x + e810640 (this build) + fresh disk	12 h and counting	0	0	0

So in our environment PR #1114 was the load-bearing fix: 0.10.2's race-shutdown fix on its own was enough to suppress #13861 (the merger-loop panic) on a clean disk, but #13860 was still firing from background goroutines (introducer/persister/merger of bluge index, measure flush/merge) where there is no gRPC recovery interceptor to catch it, so a single timestamp panic in the wrong goroutine still took the process down — that's exactly what we saw on SG.

With #1114 landed, those background paths stop generating the panic in the first place, and the cluster runs cleanly.

Operational confirmation

Functional probe via the OAP GraphQL endpoint while running on this image:

• service registry visible: 23 services including auto-creator-backend-prod, data-api-hub-*, spider-manager-service-*
• service_resp_time (avg) returns sensible numbers (~36-68 ms range, in line with what we observed on previous storage backends)
• service_cpm returns expected per-minute call counts (111-302 / min)

Cherry-pick to v0.10.x?

Given the impact, would it be worth cherry-picking #1114 onto v0.10.x and cutting a 0.10.3? We're happy to keep running our own build until upstream catches up, but I expect anyone else hitting this on a multi-agent SkyWalking deployment will see the same instability — the panic was firing every ~4 minutes for us before the fix.

Thanks again to you and @wu-sheng for the fast turnaround on this — really appreciate the in-depth root-cause work.

[wu-sheng]

0.10.3 is planned, should be in next week with cherry pick.