You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As noted in #2123 and #1395, we are seeing more instances of "missing buckets" than we expect from mere misconfiguration (which sometimes causes it, but not always).
On investigation with a node operator, we discovered a state that is best understood to be caused by our failure to fsync bucket writes:
Bucket went missing at moment of node crash (cloud provider killed host)
SQL database matched public network state (postgres WAL integrity maintained)
Once restored, subsequent buckets diverge with difference appearing just to be missing entries or updates from lower levels (i.e. some merge restarted against a zero-sized bucket)
There are three necessary fixes for this. This bug tracks one of them: start properly fsync()'ing when we close a bucket (or any XDR file) so we don't lose them even if the OS crashes.
The other two fixes are better diagnostics (tracked with #2123) and verification of hashes while reading buckets (#2203).
The text was updated successfully, but these errors were encountered:
As noted in #2123 and #1395, we are seeing more instances of "missing buckets" than we expect from mere misconfiguration (which sometimes causes it, but not always).
On investigation with a node operator, we discovered a state that is best understood to be caused by our failure to fsync bucket writes:
There are three necessary fixes for this. This bug tracks one of them: start properly fsync()'ing when we close a bucket (or any XDR file) so we don't lose them even if the OS crashes.
The other two fixes are better diagnostics (tracked with #2123) and verification of hashes while reading buckets (#2203).
The text was updated successfully, but these errors were encountered: