[CON-510] Migrate legacy storage paths in background

[theoilie]

Description

• Adds a task that runs in the background to search for rows in the Files table with legacy storagePaths, copy (or move) the files to an updated path, and update the row to reflect the new path. Gated by env var migrateFilesWithLegacyStoragePath
• Makes background tasks (this new legacy storagePath migration + disk deletion) run when cluster is disabled (they previously only ran when cluster was enabled)
• Changes image dir validation to be storagePath-agnostic and make sure it's safe to change type=image rows and their corresponding type=dir row non atomically

Database impact

• I paginated using fast cursor (>) pagination instead of LIMIT+OFFSET to avoid sequential scans
• Finding all legacy files without batching (for manual inspection):
  • CN5 (1.7M rows in 280s): SELECT * FROM "Files" WHERE "storagePath" NOT LIKE '/file_storage/files/%' AND "storagePath" LIKE '/file_storage/%';
  • CN2 (9.3M rows in 158s): SELECT COUNT(*) FROM "Files" WHERE "storagePath" NOT LIKE '/file_storage/files/%' AND "storagePath" LIKE '/file_storage/%';
• Finding all legacy files on CN2 with batching: these queries take milliseconds. Example: SELECT "storagePath" FROM "Files" WHERE "multihash" > 'Qma' AND "type" != 'dir' AND "storagePath" NOT LIKE '/file_storage/files/%' AND "storagePath" LIKE '/file_storage/%' LIMIT 100;

File storage impact

• Files at legacy paths are deleted after they're successfully copied to a new path and their db row is updated. If we left the files at legacy paths then disk space usage would increase substantially. An upper bound estimate of per-node disk space increase if we didn't delete legacy files would be:
  • avg track+image file size * # of tracks and files on CN2
  • # of tracks and files on CN2 is ~9M according to this query: SELECT COUNT(*) FROM "Files" WHERE "type" != 'dir' AND "storagePath" NOT LIKE '/file_storage/files/%' AND "storagePath" LIKE '/file_storage/%';

Testing Legacy Path Migration

1. Set migrateFilesWithLegacyStoragePath to false in config.js
2. A seed clear; A seed create-user; A seed upload-track; A seed upload-image
3. In db: UPDATE "Files" SET "storagePath" = CONCAT('/file_storage', SUBSTRING("storagePath" FROM 20)) WHERE true;
4. docker exec -it cn3_creator-node_1 /bin/sh and move each file manually: cd /file_storage/files; ls then for each dir: mv <dir>/<CID> /file_storage
5. Set migrateFilesWithLegacyStoragePath to true in config.js
6. After a minute, storagePath in the db should be updated to what it was before, and in the container /file_storage should only have /files. Also each file in storagePath under the db should exist on disk

Other tests

• Added tests to ensure extracting CIDs from legacy paths works for directories and non-directories

Monitoring - How will this change be monitored? Are there sufficient logs / alerts?

• We should eventually see every node's value stop increasing for the filesMigratedFromLegacyPath metric
• Search for occurrences of Error migrating legacy storagePaths, _copyLegacyFiles() could not remove, or Error updating legacy path db rows
• When the following queries return 0 results, then there are no remaining legacy storage paths on the node:
  • SELECT * FROM "Files" WHERE "storagePath" NOT LIKE '/file_storage/files/%' AND "storagePath" LIKE '/file_storage/%' ORDER BY "multihash" ASC LIMIT 100;

[SidSethi]

generally really nicely done, main thing we should talk about is the dir / image / nondir stuff, i need to handle that in my pr as well

[SidSethi]

really really clean and thorough code. PR desc and code both very easy to follow. great work

p much good to go imo, 1 question about terminal condition of the while loop
and we should fast-follow the prom metric