Research Review: Pushsync Silent Chunk Loss

[gacevicljubisa]

PR: #5390
Branch: fix/pushsync-out-of-depth-chunk-loss

Summary

Chunks can silently disappear from the network after a successful upload. Three bugs in the pushsync/pusher pipeline allow a chunk to be "delivered" (from the origin's perspective) while no node in the correct neighborhood actually retains it. The proposed fixes change protocol-level forwarding and receipt handling behavior.

Note: These issues are reproducible in smaller clusters such as the public testnet, where limited neighborhood sizes make out-of-depth forwarding and shallow receipts more likely.

---

Bug 1: Out-of-Depth Storing

Problem: When a node has no closer peer (ErrWantSelf) but the chunk is outside its AOR, it stores the chunk anyway. The chunk lands in a low-proximity bin, unreserve() evicts it immediately, but the origin already has a valid receipt. Chunk is gone.

Fix: Guard with proximity(chunk, self) < rad before storing — return ErrOutOfDepthStoring instead, origin retries with next peer. (pkg/pushsync/pushsync.go:301-309)

Doubling concern (@martinconic): The check uses StorageRadius() which already accounts for doubling (CommittedDepth - doublings). Using CommittedDepth instead would incorrectly reject chunks in sister neighborhoods. Appears correct, but needs research validation for edge cases during radius transitions.

---

Bug 2: Shallow Receipt Short-Circuits Parallel Pushes

Problem: On any shallow receipt, pushToClosest returns immediately with ErrShallowReceipt, discarding results from other inflight multiplex pushes (up to 32). Only 1 attempt of the error budget is used. Valid receipts from parallel pushes are thrown away.

Fix: Treat shallow receipt like any other peer failure — cache it, burn the budget, wait for inflight pushes. Only return ErrShallowReceipt after the full budget is exhausted or when falling back to ErrWantSelf. (pkg/pushsync/pushsync.go:369-530)

---

Bug 3: False ChunkSynced After Retry Exhaustion

Problem: After 6 pusher retries, chunks are marked ChunkSynced even though no node in the correct neighborhood stored them. The upload appears complete but the chunk is lost.

Fix: Report ChunkCouldNotSync instead of ChunkSynced. (pkg/pusher/pusher.go:283-289)

Gap identified (@martinconic): upload.Report() tag switch (pkg/storer/internal/upload/uploadstore.go:609-617) doesn't handle ChunkCouldNotSync — chunk gets cleaned up from the upload store but tag counters are never updated. Uploads with undeliverable chunks will show permanently incomplete sync counts.

Possible approaches:

• A. Add ChunkCouldNotSync case with a new Failed counter in TagItem + surface via API
• B. Keep chunk in push queue for later retry (risk: infinite loops for undeliverable chunks)
• C. Surface via metrics/API only, accept the tag gap

---

Protocol Impact

Scenario	Before	After
Node outside AOR, no closer peer	Stores + sends receipt (evicted shortly after)	Returns error, origin retries
Shallow receipt during multiplex	Immediate abort, 1 of 32 attempts used	Full budget exhausted before giving up
Pusher exhausts 6 retries	Marked ChunkSynced (false positive)	Marked ChunkCouldNotSync (correct, but tag gap)

[zelig]

when there are options, which one is implemented?

[gacevicljubisa]

when there are options, which one is implemented?

This is the PR with proposed fix #5390

[zsgsp]

APGergő: assign to Lat.

[lat-murmeldjur]

In connection with this. My experience when pushing to testnet from weeb-3 was that out of 10k chunks ~350 didn't become available after getting back receipt. My solution was — hammering pushing chunks when a retrieve-check signaled that they did not become available right away.
After pushing chunk, I try to retrieve it, and if it fails, chunk is added back to push queue with clean slate for retries. This made uploading reliable on first try.

[lat-murmeldjur]

Some further comment:
https://github.com/ethersphere/bee/blob/master/pkg/pushsync/pushsync.go#L295
Here it looks like the first node who considers itself a storer issues a receipt and stops pushing towards the closest node. It would make it easier to reason about this if the node would continue to push towards the actual closest node. Without that here we would need to reason about all the cases when a node could misjudge the actual storage depth which involves lots of other modules and makes reasoning harder. And also, this way the closest node doesn't get the chunk immediately and we also have to reason about the time and probability of pullsync repairing this.
This keep-pushing could happen in multiple meaningful ways. First is "issue storage receipt, but push towards closest node in separate goroutine". The benefit of this is that sending back the storage receipt is immediate, however the closest node still gets the chunk. The cost of this is that the originator could get a shallow receipt and still retry even though the receipt with higher proximity from the closest node would have been acceptable .
Another meaningful way is to "keep pushing towards closest peer, but if it doesnt gives back a receipt in 10 seconds (or gives back an error instead), as a potential storer node preemptively send back self-issued storage receipt". This is slightly more complex, the in-AOR node starts a shorter timeout for the closest node to hand back a receipt, but the originator more likely receives a receipt from the highest possible proximity to the chunk.

[lat-murmeldjur]

Another issue here could be stated as:
https://github.com/ethersphere/bee/blob/master/pkg/pushsync/pushsync.go#L418
iiuc, when closer peers exist, but they are in overdraft for the moment, a reachable storer node that includes itself in peer selection can get back ErrWantSelf instead of ErrNotFound. This means that the wait for refreshment mechanism can be bypassed and we can unnecessarily jump to the Store or Fail part.

Here not-in-AOR could wait, while in-AOR could "wait but triggering preemptive receipt signing on a shorter 10s timeout" or "send back receipt but start separate go routine that waits for closer peers to become pushable"

[zsgsp]

@lat-murmeldjur do you consider the review part ready with those comments, or shall we wait for a review at the actual PR, too?

[lat-murmeldjur]

Oh yes my review was mostly about "what else could contribute to the observed issues", the original 3 observations of this issue all seem valid problems, the solutions in the PR seem reasonable, I don't plan to make any more comments for now. Not sure if my last 2 comments here were read, it would probably be beneficial to do some sort of improvement based on these observations.