feat: curio: storage index gc task

[magik6k]

Related Issues

Proposed Changes

• Add a harmonytask helper for defining "singleton" tasks - tasks which run as at most one instance across the whole cluster
  • This construction also helps with scheduled tasks which only run every so often without requiring a separate table for each task - which will be handy for other periodic cleanup tasks.
• Implement a storage path endpoint cleanup task
  • This task scans the sector index pinging all endpoints, and removes the ones which appear to be dead for a long time
    • (lotus-miner never had that, but because the index there was in-memory, cleanup was done by restarting the process..)
  • Curio needs this because the index is now in the database, and after migrating from lotus-miner there are guaranteed to be stale endpoints in the sector DBIndex. The remote storage code can cope with those, but it's not great to keep dead references in the index.

Additional Info

TODO:

• Do a dry run on a setup with stale references
• Do a real run
• Increase intervals, drop the dryrun guard

Checklist

Before you mark the PR ready for review, please make sure that:

• Commits have a clear commit message.
• PR title is in the form of of <PR type>: <area>: <change being made>
  • example: fix: mempool: Introduce a cache for valid signatures
  • PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
  • area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
• If the PR affects users (e.g., new feature, bug fix, system requirements change), update the CHANGELOG.md and add details to the UNRELEASED section.
• New features have usage guidelines and / or documentation updates in
  • Lotus Documentation
  • Discussion Tutorials
• Tests exist for new functionality or change in behavior
• CI is green

[snadrus]

This is a great basis for a "follows" task that reviews sector paths to determine what was on a defunct storage path. It could:

• warn
• increase the count of available copies
• unseal to recreate the pieces and/or unsealed copy/copies.
• re-seal (with appropriate context) to recreate what sealed copies were lost.

We could also verify daily. We could verify copy count too.