BUG: Email tracking issue

[Rosencrantz]

Describe the bug
Aleph does not always show the full contents of an email requiring the user to download and open the email in another tool. Additionally, for some emails, attachments are not displayed.

Examples:

• https://aleph.occrp.org/entities/112188415.bcc58378c895e612b7ca7d8722bb3a93e4aa08ef
• https://aleph.occrp.org/entities/112171227.5b3ab48a19d3d22ece8996d6f5bea969da3b096f
• https://aleph.occrp.org/entities/112176241.92b574a8eac98e74de854f2c2ae2566033ffeb41

This is due to multiple related problems in ingest-file, followthemoney, and Aleph. I‘ve broken them down into separate issues. I’d recommend tackling them roughly in the following order:

• FEATURE: Add an option to switch between HTML and plaintext preview for email #3162
• BUG: safeHtml is derived only from first value of bodyHtml #3163
• Maintain insert order for multi-valued properties followthemoney#1139
• BUG: ingest-file doesn’t properly handle multipart/mixed emails #3161
• BUG: Attachments missing for some ingested emails #3165
• BUG: ingest-file "inlines" contents of attached emails as if they were part of the original email #3164

Please note that due to the fact part of the root cause for this bug is in ingest-file, we won’t be able to resolve it completely for existing entities without re-ingesting them.

Implementing #3162 will allow users to at least view all contents (even if an email was ingested before the fix is rolled out). However, users will need to be aware that they might need to switch between the HTML and plaintext view to see all contents. We should consider it has a quick fix/temporary workaround while we work on #3161.

Once #3161 is fixed, users should be able to see complete email previews even without switching between HTML and plaintext view. However, the fix will not apply to emails that have been ingested before it has been rolled out.

Regarding #3165, it’s likely that the fix will also not apply to emails that have been ingested before it has been rolled out. However, we don’t know yet what exactly is the root cause here, so it’s a bit too early to give a definitive answer.

[jlstro]

Hi,

I did more digging and made a very strange observation. I found at least one case where Aleph was showing part of the email, but the rest seems to be indexed - it shows up in a search when I search for a keyword that's in the non-visible part. (I can't share the details on what to search for and where here, it's an ongoing investigation).

In that case, the email contains a forwarded email that's attached inline. Then there's an email body above the inline attachment. Aleph only shows the forwarded email, not the original email body (but it appears to be searchable).

The original email also has 4 attachments, Aleph shows none.

[jlstro]

Here is another one that's displayed incorrectly / does not have attachments.

[tillprochaska]

@jlstro @brrttwrks I have updated the issue description with more details based on what I’ve found debugging the issue. This is actually a bigger issue with multiple separate causes. However, we will likely be able to implement at least a temporary workaround in a timely manner.

The technical details are unfortunately a bit more complex. If you’re interested in the details, feel free to look at the descriptions of the individual issues mentioned in the top-level comment or the Google Doc I shared with you. If something is unclear or confusing, feel free to reach out!

[tillprochaska]

Going to close this as all but one issues have fixes that will be included in the next Aleph release.