Stats page: Discrepancies in numbers observed and reported by ThirdStats

[rfwolff]

I noticed discrepancies in the number of emails given on the stats page of ThirdStats and the number reported by Thunderbird. This appears to affect both, an email account as well as local folders, please see below a comparison of numbers for a specified time period.

Office365 account (in Thunderbird | in ThirdStats)

• Total: 2224 vs. 2199
• Sent: 657 sent vs. 4
• Received: 1567 vs. 2195

Local folders (in Thunderbird | in ThirdStats)

• Total: 2500 vs. 2474
• Sent: 704 vs. 65
• Received: 1796 vs. 2409

The O365 has one email address (default identity) linked to it. Some minor discrepancy could be caused by 15 emails sent by me (from default identity) to an email list I am a member of, i.e. me sending emails to myself.

For the local folders, all sent emails are in subfolders of the standard sent folder. All addresses I am using are defined as "Local Identities" in the ThirdStats Options. Similar to the O365 account, there are some emails which I sent myself, approx. 30 (can get the exact number, if needed).

Please let me know if you need further information.

Thunderbird 78.7.0 (64-bit)
ThirdStats 1.2.0

[devmount]

Thank you for reporting this. The differences in total counts may be caused by folders that are not available for global search. You can try the following: Do a right click on every folder (e.g. Trash), click Properties and make sure that "include messages in global search result" is checked (it appears like WebExtension APIs can only process messages in folder that are allowed for global search).

Now test the numbers again and let me know, if this changes anything.

[rfwolff]

Thanks for looking into this.

I have checked this now. Apart from the "deleted" and "junk" folders, all folders had the option "Include messages in this folder in Global Search results" ticked.

When I selected that option for the "deleted" and "junk" folders (which are usually empty or nearly empty), the numbers did not change.

[devmount]

I see. ThirdStats first looks for the sender, so if any of the configured identities matches the emails author, that ThirdStats handles this email as 'sent', doesn't matter if this identity is also a recipient or not.

It may be caused by a contact format "<name, prename> email@address.org" vs "email@address.org". I currently implemented the extraction of email addresses myself, so there's a chance that there are cases, I don't know of. I could make use of the fromMe or toMe query attributes from the corresponding WebExtension API and see, if I can reproduce the dixcrepancies you're observing.

Another possible (minor) cause can be, that some of the emails you're accounts are holding are flagged as spam. Those emails are currently not queried (TB bug) and therefore cannot be taken into account by ThirdStats.

[rfwolff]

Thanks for looking into this.

It might be related to the contact format issue your describe, i.e. it would be good if this would sort this issue. Happy to report back once the change has been made.

I double-checked and can confirm that emails being flagged as spam should not be causing the issue, i.e. out of several thousand emails there were six that were marked as junk.

[devmount]

I wasn't able to find the cause of this issue on my end. However, I created a debug version of ThirdStats for you. Please install the following ThirdStats add-on (rename the file extension from .zip to .xpi first):
third-stats_v1.5.0-alpha-sent-received-recognition.zip

Open the console and process the affected account. It will log all messages into the console showing you, if ThirdStats recognized them as sent or received like this:

You now have the possibility to go through that list and see, why ThirdStats doesn't correctly recognized your sent mails. Note that this will produce one console entry per email, so be patient as it can take some time to process everything. Let me know, if you found anything of interest.

[rfwolff]

@devmount - Many thanks for this. I will run this and aim to get back to you at the weekend.

[rfwolff]

@devmount - I now had a chance to run the debug version which you kindly provided.

It looks as if this gave some idea on how to resolve the issue, at least for the local folders but not for the Office365 account. Briefly, I suspect it due to capitalisation of email addresses.

As described earlier, the total number of emails counted was correct but, according to the ThirdStats output, I hardly ever sent emails and had more emails received than I got adding up the number of emails in the relevant folder. This suggested something going on with email identities I was sending from. In the ThirdStats settings, I defined all of my email identities but this did not seem to do the trick.

When analysing the local folders, the address listed as most received from is "robert@somedomainname.com". Of note, I defined "robert@somedomainname.com" as local identity. Running the debugger, I realised that ThirdStats seems to capitalise the first letter of email addresses, i.e. handled "robert..." as "Robert..." when counting the email addresses (unlike the output where the email is listed as "robert...").

Subsequently, I added the capitalised version, i.e. "Robert@somedomainname.com" to the list of local identies. Running the analysis for the pre-specified time period again is getting very close to the numbers I reported earlier (see overview below, third column "fix"). The small discrepancies (compared to the first column) could be due to accepted invitations, moved and deleted messages etc. since the last count (16 days ago) for the defined period. Happy to investigate further, if needed. However, I think we found the main issue. Please note that this is obviously not addressing the issue with the Office365 account (not local).

Please find the updated numbers below:
Office365 account (in Thunderbird as reported previously | in ThirdStats previously | in ThirdStats after fix)

Total: 2224 vs. 2199 vs. 2197
Sent: 657 sent vs. 4 vs. 4
Received: 1567 vs. 2195 vs. 2193 

Local folders (in Thunderbird as reported previously | in ThirdStats previously | in ThirdStats after fix)

Total: 2500 vs. 2474 vs. 2459
Sent: 704 vs. 65 vs. 706
Received: 1796 vs. 2409 vs. 1753

Thanks again for providing the debugger. Please let me know if you need anything further.

[devmount]

Thank you for these detailed insights.

When analysing the local folders, the address listed as most received from...

This was the hint I needed. When creating the "most contacted" charts, all email addresses are converted to lowercase to prevent duplicate output. However, I don't lowercased identities itself, so that's what I'm going to improve now. 👍🏻

Please note that this is obviously not addressing the issue with the Office365 account (not local)

Just to be sure again: Your O365 account only has the default identity configured in Thunderbird? What does Debug-ThirdStats say when processing that O365 account?

[rfwolff]

Great, I am glad that was helpful.

Regarding O365, the problem looks very similar, i.e. the identity was defined as "robert@..." but according to the console, emails sent from that identity were handled as "Robert@..."

To confirm that this indeed caused the issue, I added "Robert@..." as an additional identity for the O365 account and get these numbers which are pretty much in line with my earlier count (manually adding up folders in Thunderbird; as explained in previous post small discrepancies possible):

Office365 account (in Thunderbird as reported previously | in ThirdStats previously | in ThirdStats after adding capitalised identity)

Total: 2224 vs. 2199 vs. 2197
Sent: 657 sent vs. 4 vs. 665
Received: 1567 vs. 2195 vs. 1532

[devmount]

Ok, taking the minor causes into account (junk and deleted mails), this issue should be fixed now with #254. It's part of v1.5.0 which I will release soon. If you observe large discrepancies again, feel free to open this or a new issue again.