Miners running out of disk space could no longer delete anything, and could not
recover on their own.
Deleting a blob moves it into trash/ so it stays restorable for the retention
window. On a full filesystem that move cannot happen — creating the destination entry
needs the space that only a delete can free. Every delete then failed, nothing was ever
reclaimed, and the node stayed wedged indefinitely:
ERROR Delete: failed to move blob to trash error=No space left on device (os error 28)
WARN inventory: failed to mark shard trashed error=database or disk is full
The blast radius was network-wide: an affected node accepts an authorised delete, fails
to execute it, and every delete fan-out including that node comes back partial.
What changed
When, and only when, the filesystem reports it is out of room, delete() now falls back
to a plain unlink, which needs no extra space. Both fallible steps — creating the trash
directory and the rename — reach the fallback.
is_out_of_space() matches ENOSPC and EDQUOT only. EACCES, EIO, ENOTDIR
and EPERM keep propagating: unlinking a blob because of a permission or I/O fault would
destroy recoverable data.
The restore window is given up on a full disk. That is deliberate — losing the ability to
undo a delete is strictly better than losing the ability to delete at all.
Verification
Exercised on a node held at zero bytes free: the fallback fired 7 times in 6 minutes,
delete failures went from 29 per 15 minutes to 0, no panics, and the node came back
healthy with its shard inventory intact.
cargo test -p miner --release: 50 passed, 0 failed.
Upgrading
Nodes on 0.1.30 pick this up automatically. Note the fallback prevents a node from
getting stuck; it does not free space by itself — a node whose disk is genuinely full
still needs headroom to keep accepting new shards.
What's Changed
- fix(miner): fall back to a permanent unlink when the disk is full by @masterdubs in #8
Full Changelog: v0.1.30...v0.1.31
