Syncs should eventually skip unavailable content and progress

[SidSethi]

Description

What is the purpose of this PR? What is the current behavior? New behavior? Relevant links (e.g. Trello) and/or information pertaining to PR?

Full details on Notion

Old behavior: if a secondary sync fails to retrieve any content (for any reason), it will immediately reject sync.
New behavior: a sync should eventually progress past unretrievable content and succeed.

Implementation:

• new skipped Boolean column in Files table
• sync ops will record every time a sync fails due to failure to retrieve/process files; once this failure count for a given user has crossed a specified threshold, the sync will mark those files as skipped in DB and succeed sync
• new SkippedCIDRetryService to retry all content in DB marked as skipped to correct for transient unavailability. Will only process content created after specified date to prevent repeat processing of old, permanently unavailable content
• expose skipped info in /users/clock_status/<wallet> route for later consumption

Other details:

• configurable option saveFileForMultihashToFSIPFSFallback for sync to not check IPFS after gateway. defaulted to true for back-compat
• change file retrieval to not error so sync always processes all files instead of immediately short-circuiting on first failure

Tests

List the manual tests and repro instructions to verify that this PR works as anticipated. Include log analysis if possible.
❗ If this change impacts clients, make sure that you have tested the clients ❗

1. new unit tests for /users/clock_status<wallet>
2. Maddog only tests happy path - never covers content unavailability, manual testing approach below

Manual:

• bring up local setup with at least 5 CN (not sure why this is failing with 4)
• on node startup, confirm db migration works
• on node startup, confirm retryQueue starts and is processing correctly
• run maddog base test on 1 user, confirm base cases work (content verification checks should confirm sFFM (saveFileForMultihashToFS()) still works i.e. no regression
• force sync skipping
  • force CID unavail
    • randomly select a previously created CID for 1 user, add code in sFFM to reject save
    • override saveFileForMultihashToFSIPFSFallback to false
  • force 1 user Reconfig -> add code in snapback to mark 1 of user's secondaries as unhealthy
    • confirm sync to newly selected replica fails N times
    • confirm sync to newly selected replica eventually succeeds and marks CID as skipped
• new clockstatus route
  • test with new param
  • test without new param
  • check user that exists
  • check user that doesn't exist
  • user that has no skips
  • user that has skips
  • check with param on primary and secondary
• confirm retryQueue works
  • confirm queue retries skipped CID and fails
  • confirm createdAt date calc works: let it run and fetch skipped CID -> modify createdAt threshold to be after that CID -> confirm CID is not fetched -> reset threshold
  • remove code on primary to never stream CID -> ensure retryQueue eventually fetches CID + records it, also updates DB record to skipped = false
  • confirm this is reflected in users/clock_status route

❗ Reminder 💡❗:
If this PR touches a critical flow (such as Indexing, Uploads, Gateway or the Filesystem), make sure to add the requires-special-attention label. Add relevant labels as necessary.

How will this change be monitored?

For features that are critical or could fail silently please describe the monitoring/alerting being added.

TODO specify logs here

[raymondjacobson]

Looks great! Super straight forward to me

[dmanjunath]

One kind of structural comment, otherwise looks good

[dmanjunath]

This looks great! Some small NIT's but overall this looks exactly right.