Retry fix (#18/#24) merged but never released — v0.7.2 still fully exposed; two related gaps remain

[b2-jared]

Summary

We investigated a customer report of a B2 client process (via restic, which depends on this library) hanging indefinitely — one observed case for ~80 hours — while uploading a small object, with a goroutine dump pointing into blazer's upload/retry code.

Tracing it back: the currently released version of this library, v0.7.2 (tagged 2025-01-23, still the latest tag, and what restic and presumably most Go-module consumers pin via go.sum), has exactly the unbounded-retry bug already described and fixed by #18 / #24 — but #24 was merged 2025-03-08, after v0.7.2, and has never been released. So every consumer using a tagged version is still fully exposed to the original bug that #18/#24 fixed.

Primary ask: please cut a new release

master (commit 349db35 and later) contains the fix for #18. There's been no tagged release since v0.7.2 (Jan 2025) that includes it. Given this bug can cause indefinite hangs in production backup workloads (confirmed impact on restic users), we'd like to request a new release/tag be cut so downstream consumers actually get the fix.

Two related gaps that appear to remain even on current master

While verifying the fix, we noticed two related issues in the retry path that #24 doesn't appear to address:

1. Server-supplied Retry-After duration has no upper bound.

In base/base.go, Backoff() converts a response's Retry-After header directly into a time.Duration with no clamp:

func Backoff(err error) time.Duration {
	e, ok := err.(b2err)
	if !ok {
		return 0
	}
	return time.Duration(e.retry) * time.Second
}

This value flows straight into withBackoff's (b2/backend.go) and simpleWriteFile's (b2/writer.go) delay calculation:

retry.DynamicDelay(func(attempt uint, delay time.Duration, err error) time.Duration {
	bo := ri.backoff(err)
	if bo > 0 {
		return bo  // <-- no upper bound, unlike the retry.Backoff() fallback which caps at 30s
	}
	...
})

Unlike the library's own exponential-backoff fallback (retry.Backoff, capped at 30s), a large or malformed Retry-After value from the server (or a misbehaving proxy in between) is used verbatim as a single sleep duration. With #24's new attempt caps this is no longer an infinite hang, but it could still produce a single, very long stall on one attempt out of the (now-bounded) retry budget.

2. The upload-URL pool can recycle a broken connection/URL after a hard failure.

In b2/writer.go, simpleWriteFile:

func (w *Writer) simpleWriteFile() error {
	ue, err := w.getUploadURL(w.ctx)
	...
	// This defer needs to be in a func() so that we put whatever the value of ue
	// is at function exit.
	defer func() { w.o.b.urlPool.put(ue) }()
	...

This unconditionally returns ue to the shared per-bucket urlPool (b2/b2.go) even when simpleWriteFile is about to return a hard, non-retryable error. The pool has no liveness/expiry check, so a subsequent unrelated Writer on the same bucket can pull that same potentially-broken upload URL/connection back out via urlPool.get() and immediately hit the same failure.

Repro context (from the customer report)

• Client: restic, B2 native backend, pinned to github.com/Backblaze/blazer v0.7.2.
• Workload: many hosts, each writing to its own bucket, small lock-file object writes (a few hundred bytes) under a locks/ key prefix — i.e. exactly the Writer.simpleWriteFile code path.
• Symptom: process hangs for tens of hours after the main backup data completes, stuck (per goroutine dump) inside blazer's upload/retry code.
• Notably, restic's own maintainers hit a version of this class of bug back in 2022 and worked around it — but only for the initial b2_authorize_account connection setup, not for Save()/uploads: restic/restic@dc2db2d ("If the connection to B2 fails, the library enters an endless loop.")

We're preparing a PR for the two remaining gaps above and will link it here once it's up. Happy to help verify a fix/release against the restic repro path if useful.

[b2-jared]

PR up: #32