[CW-1588] Email & attachments gets fetched again and again but no conversation is created

[bfuchs]

Describe the bug

A specific email gets fetched by an email inbox via IMAP but no conversation will be created. The worker will create the attachments in storage. The next time the worker fetches that specific email, all attachments will be created again and again.

Steps To Reproduce

Not sure whats wrong with the specific email. It is a newsletter email with lots of html and lots of images with a size of around 2.5Mb. Should I upload the raw email source code?

Expected behavior

Email should gets fetched via IMAP and a conversation should be created. Attachments should not get stored again and again every minute.

Screenshots

The email got received at 9:30. Disk usage increases. The email got resend multiple times to check whats wrong. At 11:30 the resent copies got deleted from the mail server. The original was kept but it got not fetched again. I guess because it fetches only the last n emails? After that disk usage gets normal again.

Server logs

I reproduced the bug by resending the email. The output is always the same:

Fetching starts

Jul 21 11:19:02 ubuntu-fra1-01 chatwoot-worker.1[729]: I, [2022-07-21T11:19:02.102482 #729] INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f493e8da-6730-4a4d-b5d9-4ed0f5cdb85e] Performing Inboxes::FetchImapEmailsJob (Job ID: f493e8da-6730-4a4d-b5d9-4ed0f5cdb85e) from Sidekiq(low) enqueued at 2022-07-21T11:19:02Z with arguments: #<GlobalID:0x000056023e56a770 @uri=#<URI::GID gid://chatwoot/Channel::Email/4>>

Then attachments gets stored. I ommited most of them as they look the same.

Jul 21 11:19:05 ubuntu-fra1-01 chatwoot-worker.1[729]: I, [2022-07-21T11:19:05.058117 #729] INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f493e8da-6730-4a4d-b5d9-4ed0f5cdb85e] Disk Storage (1.1ms) Uploaded file to key: cwox63adqlkk9cntlf4uyefi771l (checksum: os3A30UZKHMt7k8ohl7HOg==)

Jul 21 11:19:05 ubuntu-fra1-01 chatwoot-worker.1[729]: I, [2022-07-21T11:19:05.064246 #729] INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f493e8da-6730-4a4d-b5d9-4ed0f5cdb85e] Disk Storage (1.2ms) Uploaded file to key: a75rnsfpwko0cswmm6lf8c05jsw3 (checksum: sYFhRFkrO1fWZcO0i+Ys9A==)

Then there is an error

Jul 21 11:19:05 ubuntu-fra1-01 chatwoot-worker.1[729]: E, [2022-07-21T11:19:05.088996 #729] ERROR -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f493e8da-6730-4a4d-b5d9-4ed0f5cdb85e] wrong number of arguments (given 0, expected 1..2)

And its done

Jul 21 11:19:05 ubuntu-fra1-01 chatwoot-worker.1[729]: I, [2022-07-21T11:19:05.097905 #729] INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f493e8da-6730-4a4d-b5d9-4ed0f5cdb85e] Performed Inboxes::FetchImapEmailsJob (Job ID: f493e8da-6730-4a4d-b5d9-4ed0f5cdb85e) from Sidekiq(low) in 2995.1ms

Jul 21 11:19:05 ubuntu-fra1-01 chatwoot-worker.1[729]: 2022-07-21T11:19:05.098Z pid=729 tid=2ebd class=Inboxes::FetchImapEmailsJob jid=cf65b98fd059b7ee3feac2b7 elapsed=3.009 INFO: done

Environment

Self hosted Linux VM installation on DigitalOcean. Installed with the installer according to the documentation.

4 vCPUs / 8 GB Memory / 80 GB Disk / FRA1 - Ubuntu 20.04 (LTS) x64

_CW-1588

[bfuchs]

I can reproduce the bug on your cloud hosted platform.

I created an account on the Chatwoot cloud platform and created an email inbox. I did not use IMAP this time. I just used the forwarding address provided by the email inbox.

I sent a test email with just text which created a new conversation and worked as it should. Then I sent the email that could not get fetched to the forwarding address and it did not create a conversation.

Maybe you can check the logs on your side?

Account ID: 73633
Inbox ID: 17137

I could also forward the problem email to some support email of yours, so you can resend it yourself. Maybe this helps to find out why the mail gets not converted to a conversation.

[BrutalBirdie]

I can confirm this as well.
The storage bloats so badly that it grows for a small IMAP inbox multiple GB a day.

2022-10-06T20:50:03.000Z 2022-10-06T20:50:03.783Z pid=13 tid=2hgd class=Inboxes::FetchImapEmailsJob jid=720125f3e707c4dc825b6cd7 INFO: start
2022-10-06T20:50:03.000Z I, [2022-10-06T20:50:03.789413 #13]  INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f41e5013-b830-494a-9058-aeba06384407] Performing Inboxes::FetchImapEmailsJob (Job ID: f41e5013-b830-494a-9058-aeba06384407) from Sidekiq(low) enqueued at 2022-10-06T20:50:03Z with arguments: #<GlobalID:0x00005617d1695228 @uri=#<URI::GID gid://chatwoot/Channel::Email/1>>
2022-10-06T20:50:04.000Z I, [2022-10-06T20:50:04.986310 #13]  INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f41e5013-b830-494a-9058-aeba06384407] Enqueued Avatar::AvatarFromGravatarJob (Job ID: 79a7e939-831d-44ff-9908-3d4eab7d3470) to Sidekiq(low) at 2022-10-06 20:50:34 UTC with arguments: #<GlobalID:0x00007f85ec02abb0 @uri=#<URI::GID gid://chatwoot/Contact/11169>>, "REDACTED@DOMAIN.TLD"
2022-10-06T20:50:05.000Z I, [2022-10-06T20:50:05.106191 #13]  INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f41e5013-b830-494a-9058-aeba06384407] �[36m  Disk Storage (2.5ms) �[0m�[32mUploaded file to key: d7sg6h5me03fx6y42q3jalk2ohvv (checksum: 8vV9ktmyCiwRzIoZC1a2lQ==)�[0m
2022-10-06T20:50:05.000Z I, [2022-10-06T20:50:05.111148 #13]  INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f41e5013-b830-494a-9058-aeba06384407] �[36m  Disk Storage (1.6ms) �[0m�[32mUploaded file to key: d0pial04uf832spmwh8if3hqssyn (checksum: I1sZoigidhnPnobZoOxXXQ==)�[0m
2022-10-06T20:50:05.000Z I, [2022-10-06T20:50:05.116620 #13]  INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f41e5013-b830-494a-9058-aeba06384407] �[36m  Disk Storage (1.9ms) �[0m�[32mUploaded file to key: d288g177bv9buypxkvdhickrhzgo (checksum: ii2yBiCGWQPRoevu/YjdhQ==)�[0m
2022-10-06T20:50:05.000Z I, [2022-10-06T20:50:05.121968 #13]  INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f41e5013-b830-494a-9058-aeba06384407] �[36m  Disk Storage (1.9ms) �[0m�[32mUploaded file to key: owui7kyay5jf89z6yxj2owobohox (checksum: /f38dyH4kmvBGCeYtQGbqw==)�[0m
2022-10-06T20:50:05.000Z I, [2022-10-06T20:50:05.127312 #13]  INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f41e5013-b830-494a-9058-aeba06384407] �[36m  Disk Storage (1.8ms) �[0m�[32mUploaded file to key: 2w51a904dgcjytro1laf8mzk11fh (checksum: j0KTM5yQJ/2VT4xNIMJRdA==)�[0m
2022-10-06T20:50:05.000Z I, [2022-10-06T20:50:05.132863 #13]  INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f41e5013-b830-494a-9058-aeba06384407] �[36m  Disk Storage (1.9ms) �[0m�[32mUploaded file to key: 25j8xtbejh3f2egum2ss1t4nfj8b (checksum: M0RaTcLUuYLub1LvX5Yd8w==)�[0m
2022-10-06T20:50:05.000Z I, [2022-10-06T20:50:05.138262 #13]  INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f41e5013-b830-494a-9058-aeba06384407] �[36m  Disk Storage (1.9ms) �[0m�[32mUploaded file to key: qpeob2fukk46apvsgaetdu34dv7q (checksum: 5j2pQkGqnKQb6se+3gecIQ==)�[0m
2022-10-06T20:50:05.000Z I, [2022-10-06T20:50:05.142665 #13]  INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f41e5013-b830-494a-9058-aeba06384407] �[36m  Disk Storage (1.1ms) �[0m�[32mUploaded file to key: rc2965c7mvhyh460g6f6o9ce6wru (checksum: y/2tiHh9C5v3Hk3UNnJqDA==)�[0m
2022-10-06T20:50:05.000Z I, [2022-10-06T20:50:05.148948 #13]  INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f41e5013-b830-494a-9058-aeba06384407] �[36m  Disk Storage (2.3ms) �[0m�[32mUploaded file to key: b7169ohvymndz8c9jtz9up1lvviv (checksum: 61s7++LkifzY4Bc6k8jlZA==)�[0m
2022-10-06T20:50:05.000Z I, [2022-10-06T20:50:05.154309 #13]  INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f41e5013-b830-494a-9058-aeba06384407] �[36m  Disk Storage (1.8ms) �[0m�[32mUploaded file to key: x4bo5stcalqrr79yz6ypj3oe8w2d (checksum: 0wvVIuOPubglqbr3wydjSg==)�[0m
2022-10-06T20:50:05.000Z I, [2022-10-06T20:50:05.163813 #13]  INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f41e5013-b830-494a-9058-aeba06384407] �[36m  Disk Storage (4.5ms) �[0m�[32mUploaded file to key: nlijwl3n7gf0p4f5ib3j1hjko8eq (checksum: hCD+62DUNButQl1JWh0FwQ==)�[0m
2022-10-06T20:50:05.000Z I, [2022-10-06T20:50:05.170068 #13]  INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f41e5013-b830-494a-9058-aeba06384407] �[36m  Disk Storage (1.6ms) �[0m�[32mUploaded file to key: zo4ze3hgybrp4jt51anpn34b2yv9 (checksum: 9TTsc5VQbrWISMSJPGXsSA==)�[0m
2022-10-06T20:50:05.000Z I, [2022-10-06T20:50:05.173491 #13]  INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f41e5013-b830-494a-9058-aeba06384407] �[36m  Disk Storage (1.0ms) �[0m�[32mUploaded file to key: lcwk7lq1xpc3xazln9o9cwqjs07j (checksum: RX1uFnIkYNH7PKun6M9s2g==)�[0m
2022-10-06T20:50:05.000Z I, [2022-10-06T20:50:05.178439 #13]  INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f41e5013-b830-494a-9058-aeba06384407] �[36m  Disk Storage (1.5ms) �[0m�[32mUploaded file to key: 3sg0o0xzj16xy15zn9kbdkmmwj9c (checksum: 0vhjdc2Y/gDEgSchxtvvXw==)�[0m
2022-10-06T20:50:05.000Z I, [2022-10-06T20:50:05.184013 #13]  INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f41e5013-b830-494a-9058-aeba06384407] �[36m  Disk Storage (2.1ms) �[0m�[32mUploaded file to key: 2ju3hp3eo31lulywm8wa0nh0yxwq (checksum: 9HPMk6fDhPwXwUuJJKakrw==)�[0m
2022-10-06T20:50:05.000Z I, [2022-10-06T20:50:05.188067 #13]  INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f41e5013-b830-494a-9058-aeba06384407] �[36m  Disk Storage (1.1ms) �[0m�[32mUploaded file to key: oqi9jkmy3avl5nwta3zscqht12h2 (checksum: jzdeeKpRHpbIwlCzsRIglA==)�[0m
2022-10-06T20:50:05.000Z I, [2022-10-06T20:50:05.192665 #13]  INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f41e5013-b830-494a-9058-aeba06384407] �[36m  Disk Storage (1.4ms) �[0m�[32mUploaded file to key: p9rdwhtypsawvuswneg1jrq6dce8 (checksum: e1e9iPKqSumC/7u1yjMK0A==)�[0m
2022-10-06T20:50:05.000Z E, [2022-10-06T20:50:05.199467 #13] ERROR -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f41e5013-b830-494a-9058-aeba06384407] wrong number of arguments (given 0, expected 1..2)
2022-10-06T20:50:05.000Z I, [2022-10-06T20:50:05.200013 #13]  INFO -- : [ActiveJob] [Inboxes::FetchImapEmailsJob] [f41e5013-b830-494a-9058-aeba06384407] Performed Inboxes::FetchImapEmailsJob (Job ID: f41e5013-b830-494a-9058-aeba06384407) from Sidekiq(low) in 1410.46ms
2022-10-06T20:50:05.000Z 2022-10-06T20:50:05.200Z pid=13 tid=2hgd class=Inboxes::FetchImapEmailsJob jid=720125f3e707c4dc825b6cd7 elapsed=1.416 INFO: done

[BrutalBirdie]

Digging and analysis:
The E-Mail was identified. It contained 12MB as attachments.
It was received on the 29.09.2022
This E-Mail is getting added every minute.
12MB * 1440 Minutes (1 Day) = 17280 aka. 17.280 GB/Day.

BoI! Can I back this up with monitoring, sure can do!

[BrutalBirdie]

More background for the devs for debugging.

This mail attachment contains:

• 17 elements
• jpg and pdf files
• file names with special chars and spaces like © ┬® ,

If you need more information about the E-Mail let me know.

ps / edit:
After backing up the E-Mail for further analysis, I deleted the Mail and now the issue does no longer persist.
So there must be something with this E-Mail creating this issue.

[BrutalBirdie]

Could this be the culprit?

chatwoot/app/models/message.rb

Line 35 in 7ad5dda

NUMBER_OF_PERMITTED_ATTACHMENTS = 15

[BrutalBirdie]

Sorry for the direct ping @vishnu-narayanan but this issue did not get any (real) attention yet.
Hope you don't mind 😬

[bfuchs]

I checked my problem email again to add some additional information. It contains 31 attachments. All of them are either .jpg or .png. All filenames are properly slugified though, no weird special characters. One file was larger than the others with a size of 1.6 MB and 1663 × 2157 pixels.

[BrutalBirdie]

Bump.

Does no one else see the implication here?
This can be abused as a DDoS attack if the system has limited storage.
If the system is hooked up to a scalable filesystem then this could lead into immense costs.

[bfuchs]

We stopped our testing of chatwoot completely after we noticed this bug. We would love to use it for our company, but we are afraid of user requests and support cases not being fetched and going unnoticed. Using the paid hosted platform does not help either, because it is happening there too.

But using it as a DDoS attack is another valid concern. The monitoring @BrutalBirdie provided, shows how one simple email can completely swamp the server storage because of the short fetching interval. Imagine sending hundreds of these emails to the same server.

Maybe it makes sense to hash the email when the fetching starts and save this hash as pending fetch. If the fetching is a success, the saved hash can be deleted again. But if the fetching fails at some point, it will not get fetched again or it will only get fetched a pre-defined amount before this hash gets flagged as a problem email.

[BrutalBirdie]

It's rather shocking that this did not get any attention so far.
Yes, this is used as a DDoS vector for self hosted instances, that is also why I advised multiple customers not to use chatwoot / the IMAP feature of Chatwoot.

[nickmonad]

@sojan-official Any chance this will be addressed in the next release? We have hit this issue again and it does pose a large risk to our self-hosted instance.

[bfuchs]

Any update on this? Do you want the original email to investigate? @sojan-official

[dmpanch]

Any update on this?