Filesystem storage, many small (mail) files, performance impact

[benrubson]

Hi,

Infos:

• Used Zammad version: 5.3.0-1669151657.5fb17b02.buster
• Installation method (source, package, ..): packages
• Operating system: Debian 10.13
• Database + version: MySQL 5.8+1.0.5
• Elasticsearch version: OSS 7.10.2
• Browser + version: Firefox, Chrome, Safari, last versions

Expected behavior:

• When choosing filesystem storage, received mails should be proceeded and stored into database, attached files should be stored on filesystem.

Actual behavior:

• Received mails are (IMO) wrongly stored both in database and on filesystem.
• Attached files are properly stored on filesystem only.

Steps to reproduce the behavior:

• Enable filesystem storage, and look at stored files. Mails should not be stored on filesystem, as they are already stored in database, and Zammad will certainly have the best performance results using them from database rather than from filesystem. Storing mails on filesystem leads to many many small files, and as a result to performance issues.

Yes I'm sure this is a bug and no feature request or a general question.
Could then this be investigated ?
Many thanks 👍

[MrGeneration]

Are you really sure that the complete raw email is stored in database AND filesystem for the exact same article?
That would be new to me. I also don't see the connected performance issue you're trying to describe here.

I'm afraid this issue is potentially more of a technical question. Especially the performance part. Or you don't describe the actual issue but just the root cause. That won't help I'm afraid.

Database + version: MySQL 5.8+1.0.5

One of your performance issues rather lays there.

[benrubson]

Many thanks @MrGeneration for your reply.

Are you really sure that the complete raw email is stored in database AND filesystem for the exact same article? That would be new to me.

The complete raw email (headers + body) is stored on filesystem.
What is stored in database is the body "only".

But then, what's the goal to keep the raw email on filesystem ?
Once the article is in database, the file should not be used anymore by Zammad, right ?

Note that the emails sent by the triggers are also stored on the filesystem.
Makes me feel like all these email files are sort of temporary files, which could perhaps be deleted once used ?

I also don't see the connected performance issue you're trying to describe here.

Filesystems generally do not like managing lots of small files, leading to global performance impact when the number of small files increases.
And on a Zammad instance with a fair amount of new tickets per day, the number of small files increases... a lot :)

I'm afraid this issue is potentially more of a technical question. Especially the performance part. Or you don't describe the actual issue but just the root cause. That won't help I'm afraid.

My question is not about the performance part (I'm used to performance analysis).
But rather why Zammad stores these email files, and why they are not deleted.
Why sounds like a bug to me, perhaps somewhere in the code the rm command is missing, once the email file has been processed :)

Database + version: MySQL 5.8+1.0.5

One of your performance issues rather lays there.

I plan to follow the migration guide proposed in the 5.3 changelog.

Thank you again 👍

[MrGeneration]

Sorry, but this seems to be a technical question.
Please note that this repository is the wrong place for technical questions.
Instead, please refer the Zammad community over at: https://community.zammad.org .

In case this turns out the be a bug and no technical question, we'll transfer the relevant information to this repo!

If you require commercial grade support and are no hosted or support contract customer, you can check out our support contracts here: https://zammad.com/pricing#selfhosted
If you're paying customer already, please consult your Zammad support for assistance! :-)

[benrubson]

https://community.zammad.org/t/filesystem-storage-many-small-mail-files-performance-impact/11778/1