Files Storage Developer Guide

Files storage: server folder or S3 — Developer Guide

5.2.0 · #283 · PR #318 by cloudcogsio · User guide: Files § Storage · Review: Files S3 Storage - PR review · Module guide: Files - Developer Guide

🔑 The one-line version

Files never touches the disk or the network itself. It asks a StorageDriver for a key: LocalDriver (a folder) or S3Driver (any S3-compatible bucket, plain curl and a SigV4 signer, no SDK). When the location changes, a background job copies every file, checks it, and only then reads it from the new place. The old copies are deleted only when someone asks, and only after it has been proved, file by file, that the new copy is there.

📁 Files

File	What
includes/storage/driver.php	The StorageDriver interface; StorageNotFound (answers 410) and StorageUnavailable (answers 503); storageDriver($cfg, $store); storageLocationRelation() (same / different / unknown); storageAddressRefusal() and storageCheckedAddress() (the address guard)
includes/storage/local_driver.php	LocalDriver - <root>/<key>, refuses .., a leading /, \ and NUL
includes/storage/s3_driver.php	S3Driver over curl: range reads, multipart above 64 MiB, a 5 s connect timeout and a 15 s first-byte timeout, every request checked and pinned (below)
includes/storage/s3_sign.php	s3SignRequest() (SigV4, checked against the AWS test-suite vector), s3ObjectUrl() (path style forced for a dotted bucket over https), s3NormaliseEndpoint(), s3NormalisePrefix()
includes/files/storage.php	filesActiveStore() for writes; filesStoreFor($key) for reads, which uses the old store until that key is verified; filesServeBytes(), which sends headers only once the store has answered; filesStorageKeyIsValid()
includes/services/files_storage.php	FilesStorageService - the only writer of the storage settings: describe(), test(), save(), queueIfMoved(), migrationAction(), migrationSummary()
includes/files/storage_migration.php	storageMigrateBatch() - the copy and the cleanup; storageMigrateEnd() - the only way a move ends
cron/storage_migrate.php	The job storage_migrate (CLI only, every minute while needed, under a lock)
api/files/settings.php	?tab=storage, storage_test, storage_migration - all requireCapabilityJson(Cap::FILES_STORAGE)
api/files/upload.php, download.php, view.php	Go through the driver; the folder permission checks are unchanged
assets/js/files-settings.js (v=5), files/settings/index.php	The Storage tab

🗄️ Data

CREATE TABLE storage_migration_items (        -- one row per stored file while a move runs
    id BIGINT AUTO_INCREMENT PRIMARY KEY,
    store VARCHAR(20) NOT NULL,                -- 'files' (#284 adds the other stores)
    object_key VARCHAR(255) NOT NULL,          -- files_versions.storage_path
    size_bytes BIGINT NOT NULL, sha256 CHAR(64) NULL,
    state VARCHAR(10) NOT NULL DEFAULT 'pending',   -- pending | verified | failed | removed | skipped
    attempts INT NOT NULL DEFAULT 0, last_error VARCHAR(500) NULL, updated_datetime DATETIME NULL,
    UNIQUE KEY uq_storage_migration_key (store, object_key),
    KEY ix_storage_migration_state (store, state)
);

Settings (system_settings): files_storage_driver local | s3, files_storage_root, files_s3_endpoint, files_s3_region, files_s3_bucket, files_s3_access_key_id, files_s3_secret (encrypted, in MASKED_SETTING_KEYS, never sent to the browser - only secret_set), files_s3_prefix, files_s3_path_style, files_storage_previous (the old location as encrypted JSON - its presence is "a move is running"), files_storage_cleanup ('1' = remove old copies).

A stored path keeps its format (2026/11/08/<32 hex>.bin). In S3 it becomes <prefix>files/<key>, so no row is ever rewritten.

🔁 A move, start to finish

save()            new location tested; storageLocationRelation(new, old) !== 'same'
                  -> every distinct files_versions.storage_path queued 'pending',
                     files_storage_previous = old config (encrypted)
reads             filesStoreFor($key): verified -> new store, anything else -> old store
storageMigrateBatch  pending: read old -> size + SHA-256 checked -> put new -> size checked
                  there -> 'verified'   (an outage stops the batch, charging no attempt;
                  5 real failures -> 'failed')
uploads meanwhile go to the new store; one that chose its store just before the save
                  is queued by queueIfMoved() as 'pending'
Remove / Finish   remove_old sets files_storage_cleanup = 1; finish_keep calls storageMigrateEnd()
cleanup           (pending rows first) -> prove different -> per key: new copy there at
                  size_bytes? delete old : back to 'pending' -> storageMigrateEnd()

🛑 The cleanup - where files could be lost

// storageMigrateBatch(), while removing old copies
$deleting = $cleanup && $pendingLeft === 0;          // a file queued meanwhile is COPIED first
if ($deleting) {
    $rel = storageLocationRelation($active, $prev);
    if ($rel === 'same')      { /* end the move, delete nothing */ }
    if ($rel !== 'different') { $r['unavailable'] = true; return $r; }   // unknown: try again later
}
// ... per row:
$there = $active->size($key);
if ($there !== (int)$row['size_bytes']) { /* back to 'pending', keep the old copy */ continue; }
$prev->delete($key);

🔑 storageLocationRelation() only ever proves "different". It writes a marker through $a, checks $a can see it and $b cannot, then does the same the other way round. Anything else - an error, a timeout, a side that cannot see its own marker - is unknown, and unknown deletes nothing. Two settings that look different can be one place: a folder spelt two ways, a symlink, one bucket behind two endpoint spellings. A "move" between them verifies every file against itself, and removing the "old" copies would then delete the only ones.

🔑 storageMigrateEnd() clears only finished rows (verified, removed, skipped). A pending or failed row keeps files_storage_previous, so the move goes on and its file is read from where it really is. The keys set aside as missing are written into the audit entry.

🌐 The address guard

The server sends requests to whatever an administrator types. A link-local address is the cloud metadata service, which hands out the server's own credentials.

// S3Driver::once(), before every request
$port = (int)($p['port'] ?? (($p['scheme'] ?? 'https') === 'http' ? 80 : 443));
$ip   = storageCheckedAddress((string)$p['host']);   // A + AAAA, every address checked
$opts[CURLOPT_RESOLVE]   = [$host . ':' . $port . ':' . $ip];   // pinned: no second lookup
$opts[CURLOPT_PROTOCOLS] = CURLPROTO_HTTP | CURLPROTO_HTTPS;
// CURLOPT_FOLLOWLOCATION stays false

• The host checked is the one the request really goes to. With virtual-hosted addressing that is <bucket>.<endpoint host>, which a check of the endpoint alone never sees.
• Refused: 169.254.0.0/16, 0.0.0.0/8, fe80::/10, fd00:ec2::254, ::, and IPv4 written as ::ffff:.
• Allowed on purpose: private and loopback addresses, because a MinIO on the LAN, or beside FreeITSM in Docker, is a normal set-up.
• FilesStorageService::endpointRefusal() uses the same check, to give the person typing a quick answer.

📥 Reading

filesServeBytes() asks the store for the range and calls $begin() (the headers) only when the first chunk arrives. So a store that cannot be reached still gets a clean 503 with "storage can't be reached", not a half-sent download. A missing object is 410. Range requests work against S3 too, so a video can be scrubbed.

⚠️ Traps

• TRAP: never treat "could not check" as "different places" before deleting. Use storageLocationRelation() and delete only on 'different' (PR #318 review).
• TRAP: never delete an old copy on the strength of a row alone. Check the new copy is there, at size_bytes, at that moment.
• TRAP: never end a move with DELETE FROM storage_migration_items. Go through storageMigrateEnd(), or an upload queued a moment earlier is dropped and its file answers 410.
• TRAP: check the address the request actually connects to, and pin it. Checking the endpoint's name and then letting curl resolve again (or prepend the bucket) is no check.
• TRAP: files_s3_secret is never sent to the browser. Only secret_set is. The save keeps the stored secret when the field comes back as it was sent (secretUnchanged()).

🧪 Tests

Test	Needs	Checks
tests/storage-local-driver.php	nothing	12 - keys, traversal, ranges
tests/storage-s3-signing.php	nothing	34 - SigV4 against the AWS vector, addressing, the address guard (refused / allowed addresses, a metadata endpoint refused by the driver, the bucket's own host checked)
tests/storage-s3-live.php	a real bucket (optional)	6 without one
tests/files-storage-backend.php	test DB	11 - which store a key is read from
tests/files-storage-settings.php	test DB	16 - save, the S3 secret, a move queued, a second change refused, the upload race
tests/files-storage-migration.php	test DB	27 - copy, checksum, outage, five failures, cleanup, probe same / different / unknown, cleanup refusing on unknown, a missing new copy re-copied before its old copy goes, a file queued during cleanup copied first, ending a move keeping an unfinished row

Database tests run with FREEITSM_TEST_DATABASE=<the throwaway db>, in one transaction that is rolled back. The four that use temporary folders normalise them to forward slashes, so they pass on Windows as well as Linux.

---

See also: Files - Developer Guide · Scheduled tasks · Files S3 Storage - PR review