Bug: RocksDB blob storage does not get cleaned / purged neither by maintenance nor by api calls

[chrilep]

Issue Description

Cross reference: Issue at Bulwark's repository (this web client can do JMAP file management).

---

When users upload files to Stalwart 0.15.5 (current version as of April 4th 2026) via JMAP, it uses blob storage to save them to disk. See https://stalw.art/docs/storage/blob/ for details.

Unfortunately even after users delete their files from their Stalwart account these files stay in the storage forever. The maintenace run and the api calls to these endpoints do nothing about that:

• Purge Blob Store
• Purge Data Store
• Purge In-Memory Store (yeah, why should it)
• Purge All Accounts

This being said, any user can kill your Stalwart installation by uploading like thousands of files even though the max upload size is set to a low value (default is 50M bytes ~ 47.68 MB). Just upload, delete, upload delete, ...

---

This was found while doing chunked encoding upload tests for Bulwark issue #96.

⬇️ Endpoint tests done ⬇️

Purge blob storage and get a successful return message JSON:

# curl -X GET -u 'admin:pasword>' https://stalwart.domain.com/api/store/purge/blob
{"data":null}

See your blob storage stay unchanged:

These files with GB in size are from the tests of one user account. The files inside this account are already deleted before maintenance run took place at night and after calling all the purge methods of the various storage backends.

Means: Even unreferenced blob data is being kept forever!

Expected Behavior

Unreferenced data in blobs should be removed after undelete grace period.

Actual Behavior

Blobs stay forever.

Reproduction Steps

1. Upload file via JMAP
2. Check blob storage. File is there.
3. Delete file via JMAP
4. Check blob storage. File is there.
5. Wait for next maintenance run and/or do api calls against blob store
6. Check blob storage. File is there.

Tests done without enterprise license against QA system.

Relevant Log Output

none

Stalwart Version

v0.15.x

Installation Method

Binary (Linux)

Database Backend

RocksDB

Blob Storage

RocksDB

Search Engine

Internal

Directory Backend

Internal

Additional Context

• Installed on an LXC (Debian 12, Stalwart 0.15.5 binary) managed by a HA Proxmox cluster.
• Data in /opt/stalwart/data/
• API method calls as admin and as the user who uploaded and deleted the files.

I acknowledge that:

• I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.
• I have searched the Stalwart repository (both open and closed Discussions and Issues) and confirm this is not a duplicate.
• I have set the logging level to trace and included relevant log output if applicable.
• I agree to follow the project's Code of Conduct.

[mdecimus]

Stalwart uses reference counting to manage blob storage. Blobs are only deleted from the store once all references to them have been removed. Off the top of my head, these are the possible reference types:

• Hard links to objects such as emails, files, or Sieve scripts, removed when the email or object is deleted.
• Temporary queue links expire once message delivery completes.
• Temporary upload links expire automatically after a short period.
• Spam filter links can persist for up to 3 months (or whatever the configured hold period is).
• Undelete links persist for the duration of the undelete hold period.

Of all these, I'd say the most likely culprit is the spam classifier holding a reference to the blob. Currently there's no way to unlink these outside of the database console, but in v0.16 you'll be able to delete training samples directly.

[gdewey]

Thanks for the explanation, @mdecimus. Just to confirm: does this mean that all emails, including those that were in the inbox, are retained for training until they are indexed, and only then deleted?

Also, in v0.16, will deleting training samples be exposed as an API parameter?

[mdecimus]

does this mean that all emails, including those that were in the inbox, are retained for training until they are indexed, and only then deleted?

Only those that were incorrectly classified or those that the user moved to the Junk folder.

Also, in v0.16, will deleting training samples be exposed as an API parameter?

Yes.

[chrilep]

Thank you for clarification.

In my case I uploaded some big test files (up to 2 GB in size) to test Bulwark's files feature (uses JMAP FileNode API).

The file should not be referenced by anything but my Stalwart user's account.

Deleting the file should have removed the blob since it has no references any more. And, it should neved had a reference to mail, cards or cals. Or maybe there is some kind of Trash folder?

[mdecimus]

What do you plan for 0.16.x regarding storage?

There are no changes in storage in v0.16.

Are there any analysis tools for Stalwart's storages?

Install Stalwart on a fresh container:

• Reduce the blob TTL to 5 minutes. Reload the config.
• Upload your large file.
• Link it to a FileNode
• Delete this file from FileNode
• Run the purge task after 6 minutes (or whatever blob TTL).

If the blob is still there, use the database console to dump all blob links and post the results here.

AI suggests to compile tools like ldb and work on a copied database.

Don't trust AI, it hallucinates when it doesn't know a software well enough.

[mdecimus]

The files I show in the initial post are still there after a week. File TTL is at 1h (default).

That is a different issue, RocksDB needs to run the compaction task. RocksDB decides when to do this and Stalwart cannot trigger it manually.

[chrilep]

Great, I will do this. Please clarify: ... use the database console to dump all blob links ...

[mdecimus]

Great, I will do this. Please clarify: ... use the database console to dump all blob links ...

https://stalw.art/docs/management/database

However, the issue in your case seems to be that RocksDB has not yet compacted the database. You can try using PostgreSQL or SQLite to confirm.

[gdewey]

very interesting how after the first purge my S3 storage when from 3.5 million to 900,000 files and from 500 GB to 100. but then getting thousands of this logs

WARN Training sample not found (spam.train-sample-not-found) reason = "Blob not found"

any idea ? I understand the log. but I just dont understand how I got to this point of getting missed blob files.

[tonyhardcode]

Scenario: Stalwart v0.16.5 using Store type: Filesystem.

I imported around 2 GB of email into a single account’s Inbox and Sent folders over IMAP. After deleting all messages, emptying Trash, and waiting for Data Retention → Expunge and Cleanup to complete, I noticed the account’s blobs still remain in the server’s blob-store directories. There appears to be 1 GB left on the server. I inspected the 52 character blob files via a text editor and they are the emails that I deleted.

Is this expected behavior, or could blob cleanup not be running?

[rathlinus]

Support moved from discord, github etc to the forum, so you should probably post there