File size optimisation / playing around with the FS.

[tiphaineruy]

Hello @Barre ,

I don't want to open an issue for this since I'm not really using this as it probably should be. But if you have an insight it´d be interesting:

config:

• Default settings for pretty much everything
• Scaleway ( they support locking since a few days agohttps://feature-request.scaleway.com/posts/843/support-locking-when-using-objects-storage-as-terraform-state-backend )

[cache]
dir = "${HOME}/.cache/zerofs"
disk_size_gb = 10.0
memory_size_gb = 1.0

[lsm]
# l0_max_ssts = 64                 # Max SST files in L0 before compaction
# max_unflushed_gb = 1.0           # Max unflushed data before forcing flush (in GB)
max_concurrent_compactions = 12

[servers.ninep]
addresses = ["127.0.0.1:5564"]
unix_socket = "/tmp/zerofs.9p.sock"

[servers.webui]
addresses = ["127.0.0.1:5565"]
uid = 1000
gid = 1000

[servers.rpc]
addresses = ["127.0.0.1:7000"]
unix_socket = "/tmp/zerofs.rpc.sock"

What i'm running: ( fuse mounting a folder in ~ )

./zerofs run -c zerofs.toml              
./zerofs mount 127.0.0.1:5564 ./mnt

Usage:

• pretty random, ill mount / unmount then start stop in random order the thing, upload a lot of small file, few big files, mixed files , create 1000 10mb files at once etc.

The error I get: ( pretty reliably after 5/10 min of tinkering )

2026-06-16T19:58:53.660675Z ERROR slatedb::compactor: error executing compaction [error=ObjectStoreError(
    NotFound {
        path: "zerofs-data/compacted/01KV9000D9HRZRWKW1TJ73KZJ2.sst",
        source: RetryError(
            RetryErrorImpl {
                method: GET,
                uri: Some(
                    https://s3.fr-par.scw.cloud/zerofs/zerofs-data/compacted/01KV9000D9HRZRWKW1TJ73KZJ2.sst,
                ),
                retries: 0,
                max_retries: 10,
                elapsed: 5.904071ms,
                retry_timeout: 180s,
                inner: Status {
                    status: 404,
                    body: Some(
                        "<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><RequestId>txg5ac47bf5f10546329601-006a31aafd</RequestId><HostId>txg5ac47bf5f10546329601-006a31aafd</HostId><Resource>/zerofs/zerofs-data/compacted/01KV9000D9HRZRWKW1TJ73KZJ2.sst</Resource></Error>",
                    ),
                },
            },
        ),
    },
)]

Question:
Not sure how to go about this, from what I understand of the Architecture there isnt a lot you can do if you "lose" one of the .sst and the filesystem is bricked at that point.

But is there something I shouldn't do with the Fs ? From what is saw there is nothing stoping you having 2go+ files but maybe there is a config combination of sst size or limitations that I'm hitting that I'm not aware ?

[Barre]

Hi,

It seems that you are doing nothing wrong.

But is there something I shouldn't do with the Fs ? From what is saw there is nothing stoping you having 2go+ files but maybe there is a config combination of sst size or limitations that I'm hitting that I'm not aware ?

There's nothing that you can configure that could break the filesystem in that way. It's a bit tricky to debug because it could be many things such as:

1. Read-after-write inconsistency: the compactor PUTs .../compacted/<ulid>.sst, gets a 200, and the manifest is updated to point at it. A later GET is routed to a node/replica that hasn't yet seen the write and returns NoSuchKey.
2. List-after-write / list-after-delete inconsistency ("partial list replies"): a LIST returns an object that was already deleted, or omits one that was just written.
3. Conditional-write (CAS) correctness: ZeroFS uses conditional puts to commit the manifest and fence writers.
4. Non-durable PUTs under high concurrency: the burst of small files means a lot of concurrent PUTs; if one is acked before it's durably committed, every later GET for it is a hard 404.

To narrow it down:

• Next time it happens, try to fetch that exact key (.../compacted/01KV9000D9HRZRWKW1TJ73KZJ2.sst) from the bucket directly. If it's there now, it was a transient consistency 404 and the data is fine; if it's genuinely gone, it was a lost write.
• If you can, run the same workload against a known-good store: S3, R2, GCS, or Azure Blob. If it doesn't reproduce there, that points squarely at your provider consistency/CAS; if it does, then it's on ZeroFS side and I'll dig in.

If you can't reproduce on a known-good store (and I'd bet you won't, otherwise the CI would be failing all the time) it's worth compiling all of this and sending it to them.

• the failing key and the RequestId / HostId from the error body (txg5ac47bf5f10546329601-006a31aafd)
• ideally the RequestId of the PUT that wrote the same key, so you can show a 200-acked write followed by a 404 GET for the same object;
• the bucket and region (fr-par), the rough timestamp, and that it correlates with bursty concurrent writes;
• the fact that the same workload doesn't reproduce on S3/R2/GCS/Azure.

[tiphaineruy]

Thanks for the insight.
Love the project BTW.

This isn't a transient error, the key isn't there and once the error starts firing I get it every compaction attempt. And the FS is bricked, can't mount / access it anymore.

I'll try and make a reproducible workload, you're probably right it might be on the provider side.
I'll reopen if I can reliably brick it.

thanks