Sent mail statistics are zero (0) in 90.0a1 (Windows 64-bit)

[koobs]

Describe the bug

Statistics for Sent mail are zero (0) for at least the past couple of weeks. I am not sure when the issue was first introduced.

To Reproduce
Steps to reproduce the behavior:

1. Go to ThirdStats Dashboard
2. Observe no stats for "Mails sent"
3. Try to rebuild cache
4. Observe no stats for "Mails sent" again

Expected behavior

Up-to-date and accurate statistics for Sent mail

Screenshots

System (please complete the following information):

• OS: Windows 10 Pro 20H2 x64 19042.1023
• Thunderbird Version: 90.0a1 (2021-05-30) (64-bit)

Additional context

• Isolation tests: rebuild cache. enable service workers (warning in dev console)

Developer Console messages:

[devmount]

Thank you for this report. I'm unable to reproduce this on Win10/TB90.0a1 so let's try to narrow it down:

1. Since it stopped working at some point I assume all your identities are correctly configured? Is this a local or an IMAP account?
2. Do you have "Messages to Self" enabled in the add-on options?
3. Do you only use Thunderbird Daily or do you also have a Thunderbird Beta / Stable on your machine to confirm, that this bug is related to the Daily version?

[koobs]

1. No changes to accounts/identities since started using TS. Google IMAP Account
2. Tested with Reset Options which disables Self messages.
3. I only use Daily, but I tested back to 2021-04-03 (89.0). Unfortunately, there's the virtual folder bug before 2021-04-29. Build bisection:

2021-04-03 (89.0a1) - TS doesn't complete processing - Error: Folder not found: <virtualfoldername>
2021-04-17 (89.0a1) - TS doesn't complete processing - Error: Folder not found: <virtualfoldername>
2021-04-24 (90.0a1) - TS doesn't complete processing - Error: Folder not found: <virtualfoldername>
2021-04-28 (90.0a1) - TS doesn't complete processing - Error: Folder not found: <virtualfoldername>
2021-04-29 (90.0a1) - Not OK 
2021-04-30 (90.0a1) - Not OK 
2021-05-03 (90.0a1) - Not OK 
2021-06-02 (91.0a1 )- Not OK

4. I've also reset almost all custom preferences back to defaults.

[koobs]

Inspecting extension data storage:

...snip...

...snip...

...snip...

[devmount]

I see, sorry to hear that you are affected by the virtual folder bug (I thought that was fixed long before TB 90 😮).

Since I haven't been able to find what's causing this issue on my end, I created a debug version of ThirdStats for you. Please install the following ThirdStats add-on (rename the file extension from .zip to .xpi first):
third-stats_v1.8.0-alpha.sent-received-recognition.zip

Open the stats tab, open the console, clear it and process the affected account again. It will log all messages to the console showing you, if ThirdStats recognized them as sent or received like this:

You now have the possibility to go through that list, analyze the message objects and hopefully see, why ThirdStats doesn't correctly recognized your sent mails. Note that this will produce one console entry per email, so be patient as it can take some time to process especially large accounts. Let me know, if you found anything of interest.

[koobs]

@devmount Only saw virtual folder issue using older versions while bisecting :)

Thank you for the debug version. Will test and report back.

Relatedly, I think it'd be worthwhile, if its possible, to add a 'Debug' option to ThirdStats to allow for quicker and simpler user debugging in the future for situations like these. Certainly save the time going back and forth :)

Do you want a new issue for that?

[devmount]

Only saw virtual folder issue using older versions while bisecting :)

Bisecting? Sorry if there was a joke, I didn't get it 🙈

Relatedly, I think it'd be worthwhile, if its possible, to add a 'Debug' option to ThirdStats to allow for quicker and simpler user debugging in the future for situations like these. Certainly save the time going back and forth :)

Great suggestion. As I stated here, the demand for such a feature is too small by now, but I totally see that this would be useful.

Would be awesome if you could create a new issue for that - as I really like reading issues from you anyway 😍

[koobs]

Hey Andreas. Tried to let the debug/refresh run three times to completion but thunderbird crashes every time after consuming all memory (16gb)

Can we descope the debug to only include sent, or exclude received or something?

Also I think I found an unrelated bug ;)

[devmount]

Whoa, I see - so logging 64k mail objects to the console wasn't the brightest idea here 🙈 I rebuilt the debug version without logging the whole object, so this should fit now into your memory (at least I hope so):

third-stats_v1.8.0-alpha.sent-received-recognition.2.zip

Also I think I found an unrelated bug ;)

Looking forward to your bug report ;)

[koobs]

Test results:

• All log entries are prefixed [RECEIVED]. I see no other entries.
• Only two errors observed, one which I've seen before, and don't know if its related to thirdstats or not, but im guessing not (The 5 minute activity manager thread possible?) (see screenshot). I see this error at intervals unrelated to thirdstats processing too. The other error is that which is included in the screenshot in the original report.
• I mentioned "unrelated bug" in my last post. I mention here in case its relevent :)
  • Some entries contain [RECEIVED] foo@barbsd.org not found in ["foo@BarBSD.org"] (note: same email, case difference). My naive expectation would be that this would or should be a match, if the check was case-insensitive. Still, this is only for [RECEIVED] (I see no [SENT]). I can report this separately if its unrelated to this issue.
  • I also see entries like [RECEIVED] foo@bar.tld some other text not found in ["foo@BarBSD.org"]. The actual examples of 'some other text' are like firstname lastname, or (nickname something). My naive expectation would be whatever API is being used to extract email addresses, would only return email addresses, not email addresses plus additional text.

[devmount]

Thanks again for your detailed report.

Error 1: As you already guessed, this isn't related to ThirdStats.
Error 2: This is indeed a ThirdStats issue. Actually, everything should be normalized to lower case, but thanks to your report I found some places where this transformation was still missing. I also improved the email address recognition algorithm in strings. Here is a fixed version, can you test this again?

third-stats_v1.8.0-alpha.sent-received-recognition.3.zip

[koobs]

You win @devmount. Sent mail stats are back. So root causes was a case-insensitive match and I'm one of those annoying users who breaks things? :)

[koobs]

Have a couple more comments to add, but need some more time to flesh out

[devmount]

Great! Thanks for confirming the fix, will be part of the next release.

Have a couple more comments to add, but need some more time to flesh out

Take your time.