Systray double-counts unread messages in sub-folders of Inbox

[mjeveritt]

Describe the bug
Systray double-counts (occasionally more!) unread messages in sub-folders. This then misreports unread messages in the tray icon!

To Reproduce
Steps to reproduce the behaviour:

• Receive mail.
• Mail is filtered into subfolder
• New messages are double (or more) -counted by systray.

Expected behavior
Correct count of new messages is maintained across all tracked subfolders.

Screenshots
If applicable, add screenshots to help explain your problem.

• Difficult to capture this meaningfully ..

Desktop (please complete the following information):

• OS: Linux Ubuntu 20.04.6 LTS
• 64 bit
• KDE/Plasma 5
• Thunderbird 102.13.0 (64-bit) (from Packages repo)
• Run echo $XDG_SESSION_TYPE in a terminal: x11
• Add-on version: 0.9.0
• App version: 0.9.5 - build 904
• Install type: deb
• Used Qt library: default system

Additional context
Add any other context about the problem here.

[Ximi1970]

You still have an old add-on that does the counting. Did you install one manually? Please remove it and let the system add the new one. There should be no systray-x addon installed in your own profile directory. (~/,thunderbird/xxxx.default/extensions)
Can not replicate the bug with a clean install and TB 115 for now. Will try the older version.

[mjeveritt]

I've cleaned the old version out, and reinstalled 0.9.5, but it's definitely still misbehaving. Are you testing by manually moving messages around, as I have about 20 pre-installed Mail Filters in place, which move messages based on content/sender, and it could be this that's causing a race condition somewhere ..

[Ximi1970]

For testing I manually moved them in a test setup. But I am using filters in my main setup with more then 20 mail accounts but no filters moving mails inside the inbox dir. I use the filters to move some mails to an archive account. The complete archive account is also part of the the count filters of the addon. But I never saw a double count...
But I can imaging double counting if my count is done parallel with the mail filtering going on.

What happens if TB is in rest and you mark one message in an inbox as unread? Is the updated unread count then valid?

Could you also have a look at the debug console of TB? The count filters are listed in the debug output of the addon.
(Tools->Develop Tools, then confirm the remote debugging by pressing "Ok" in the dialog, then select the "console" tab)
Find the filter list in the console by scrolling.

Stored filters: [{"accountId":"account13","version":"0.9.5","folders":["/INBOX"]}......

Check if there are somehow double entries listed.

[mjeveritt]

Could you also have a look at the debug console of TB? The count filters are listed in the debug output of the addon.
(Tools->Develop Tools, then confirm the remote debugging by pressing "Ok" in the dialog, then select the "console" tab)
Find the filter list in the console by scrolling.

Stored filters: [{"accountId":"account13","version":"0.9.5","folders":["/INBOX"]}......

Check if there are somehow double entries listed.

I'm only getting a bunch of 'spam' about calendars appearing in the Tools/Developer Toolbox/Console log; and nothing [meaningful] relating to Mail Filtering nor the Counting of messages from Systray-X. I'm probably missing something fundamental here, so I'd appreciate any pointers.
I also tried the method of going into Tools/Debug Add-ons, but there is nothing whatsoever posted there when I select Systray-X from the list that pops up.

For the most part, counting is working fine in individual folders, and decrements fine also .. but its funny when you read a New message in a sub-folder, and, like, 6 drop off the Notification counter .. it's definitely incorrect!!

Happy to continue the debugging quest, as I suspect I'm probably not the only unfortunate victim of this problem!

[Ximi1970]

Moved my testing to Ubuntu 20.04. Setup a message filter on the Inbox moving some mails to an "Inbox/Test" dir. But still no problems detected.
Are you using any other addons voor TB?

Forgot they changed the logging/debugging in the latest version. To get the debug output:
Menu->Tools->Developer Tools->Debug addons-> Search for Systray-X in the "Extensions" listbox -> Click "Inspect" and select in the newly opened tab the sub tab "console". At the bottom of the page there should be a filter list visible.
In my test case it lists:

Stored filters: [{"accountId":"account1","version":"0.9.5","folders":["/INBOX","/INBOX/Test"]}] defaults.js:261:15

[mjeveritt]

This is what I've got in that window:

https://dpaste.com/HK26955CA
(set to expire in a year)

[Ximi1970]

I can see that the addon detects 5 email accounts from the forced filters setting (but only 1 account is listed partially in the account listing).
I can also see, you never selected the folders which it should count. If you do not, it should automatically count the mails in the "Inbox" folders only (not any subdirs in them)
So it is still a mystery would happens here.
Maybe there are still some old settings which are not displayed anymore but somehow still used. Cannot figure out how that could happen.
Please could you use the "Reset" button in the TB preferences of the addon to clear all old and new preferences of SysTray-X?
Afterwards you have to set your preferences again and mark the accounts, folders you want to add to the count.

[mjeveritt]

Unless I'm mistaken, I had to do a reset recently either when upgrading, or whether it was porting my profile to another computer.
I can certainly do it again, but if the problem prevails, it's still a problem with the plugin, and not my install! ... :)

[mjeveritt]

Hmm, well this is interesting, on restarting TB, with the Toolbox window open, the Stored filters is defined, as such:

Stored filters: [{"accountId":"account1","version":"0.9.5","folders":["/INBOX","/INBOX/--old-tagged--","/INBOX/--older-tagged--","/INBOX/--Support","/INBOX/Ad&AOk-lie","/INBOX/Amazon","/INBOX/ANTS","/INBOX/auctions","/INBOX/aufs","/INBOX/Autism-stuff","/INBOX/Autism-stuff/--old-tags","/INBOX/Autism-stuff/ADDitude","/INBOX/Autism-stuff/DAANA","/INBOX/Autism-stuff/Devon autism card","/INBOX/Autism-stuff/Devon autism card/crap","/INBOX/Autism-stuff/Devon Autism Guide","/INBOX/Autism-stuff/DFA-meetings","/INBOX/Autism-stuff/DFA-meetings/2018","/INBOX/Autism-stuff/DFA-meetings/2019","/INBOX/Autism-stuff/DFA-meetings/2020","/INBOX/Autism-stuff/DFA-meetings/2021","/INBOX/Autism-stuff/DFA-meetings/2022","/INBOX/Autism-stuff/DFA-meetings/2023","/INBOX/Autism-stuff/DFAdocs-old","/INBOX/Autism-stuff/DMHA","/INBOX/Autism-stuff/GP-WG","/INBOX/Autism-stuff/GTDS","/INBOX/Autism-stuff/interview","/INBOX/Autism-stuff/ND-Rebel","/INBOX/Autism-stuff/NHS-ND-CEN","/INBOX/Autism-stuff/oh-Alex!","/INBOX/Autism-stuff/Torbay-miniconf","/INBOX/Banking-Finance","/INBOX/Banking-Finance/HSBC-CustomerStudio","/INBOX/Barnstaple-Pi-Jam","/INBOX/BenD","/INBOX/Blogtrottr","/INBOX/Blogtrottr/Hue","/INBOX/Blogtrottr/kernel","/INBOX/Dad-Martin-Everitt","/INBOX/Dad-Martin-Everitt/Estate-probate-etc","/INBOX/Dad-Martin-Everitt/Funeral","/INBOX/Dad-Martin-Everitt/Tributes","/INBOX/dclug","/INBOX/fleabay","/INBOX/fleabay/dbx-kit","/INBOX/fleabay/Orders","/INBOX/Gitlab-hub","/INBOX/Gitlab-hub/chatterino","/INBOX/Gitlab-hub/etcher","/INBOX/Gitlab-hub/Gentoo","/INBOX/Gitlab-hub/Gentoo/audio","/INBOX/Gitlab-hub/Gentoo/portage","/INBOX/Gitlab-hub/pi-hole","/INBOX/Gitlab-hub/systray-x","/INBOX/Gitlab-hub/whatsapp","/INBOX/Hatherleigh","/INBOX/Hatherleigh/Cycle_path","/INBOX/Hatherleigh/Energy","/INBOX/Kernel","/INBOX/Kernel/releases","/INBOX/LifeCare-Radio","/INBOX/Matrix-Element","/INBOX/musl","/INBOX/openPower","/INBOX/Parted","/INBOX/Pentoo","/INBOX/Phoronix","/INBOX/python","/INBOX/SocialFixer","/INBOX/SocialFixer/data","/INBOX/SocialFixer/dev","/INBOX/TripLog crap","/INBOX/Utilities"]}] defaults.js:261:15

which looks a little more sensible...

[Ximi1970]

Found some problems with the new / unread mail listeners. Refactored the counting a bit using a new method for TB 115 and up due to extra available parameters in the API. Updated the listeners for the older versions. Still testing and looking for problems. Did not have much time lately...

[mjeveritt]

Found some problems with the new / unread mail listeners. Refactored the counting a bit using a new method for TB 115 and up due to extra available parameters in the API. Updated the listeners for the older versions. Still testing and looking for problems. Did not have much time lately...

Much appreciated, and also I understand the time pressures!

Could you share your insights on the listeners issue, and or reference this issue in your testing code branch perhaps? I would be happy to do some testing if that helps you conclude the problem(s) ! I can't remember off the top of my head whether it's possible to modify-in-place a mozilla plugin (I use greasemonkey scripts extensively in Firefox and Chrome - it's a lovely ecosystem) and again, if there's supplementary debug information I can provide to ease development, only too happy to do so!! 😃

[Ximi1970]

The listeners were activated after a 5 second delay (can be tweaked in the settings). The delay was needed for pre? 115 TB because the accounts and folders where not immediately available using the API for setting up the count filters.
The listeners are now installed immediately at the start of the add-on. All events are cached until the delay timer fires and hopefully the account data is available.
The changes are now in the develop branch. I will generate a new release tomorrow.
Hopefully it will solve your problem too. I really can not replicate your specific problem.

You could use the git version of the add-on to test:

sudo bash -c 'echo "deb https://download.opensuse.org/repositories/home:/Ximi1970:/Mozilla:/Add-ons:/Git/xUbuntu_20.04 ./" > /etc/apt/sources.list.d/systray-x.list'
sudo apt update
sudo apt upgrade

[Ximi1970]

And problems with the build service. Packages are not published at the moment.
Whoops looks like all of opensuse.org is down.

[mjeveritt]

And problems with the build service. Packages are not published at the moment. Whoops looks like all of opensuse.org is down.

I should be able to build from git - I'll figure out how! But sounds like you have bigger problems!! 😬🤨

[mjeveritt]

I should be able to build from git - I'll figure out how! But sounds like you have bigger problems!! 😬🤨

Successfully built from the git repo , develop branch, noting #160 !
Installed and running .. so I'll see how I get on...

Further to original post, TB is now 102.15.1 (still installed from Ubuntu repository, security.ubuntu.com).

[mjeveritt]

ok I'm pretty sure I've got the addon updated from the git branch, but I think the separate 'app' part isn't loading from wherever it should be .. how does this normally get spawned? do I need to move the executable from the build folder somewhere for it to be 'detected'?!

Thanks in advance!

[Ximi1970]

First you need to uninstall the package from the repository.

Then install the config file:

mkdir -p ~/.mozilla/native-messaging-hosts
cp -f app/config/linux/SysTray_X.json ~/.mozilla/native-messaging-hosts/

Have a look in the SysTray_X.json. It should contain the path to the SysTray-X binary in the build directory.
Manually install the xpi from the github dir (install from file in the add-on/themes dialog).

If you ever install the package from the repositoy again, please do not forget to remove this file again.

I have updated the README a few day ago. It should be in the Build section.

[mjeveritt]

Have a look in the SysTray_X.json. It should contain the path to the SysTray-X binary in the build directory. <snip>

Ooops! yeah I'd unwittingly built it as 'root' user and copied to my home folder, not correcting the path! 🤦‍♂️
Now fixed (manually) and I think everything is golden. I had seen a suspicious error in the Debug Log window:

Uncaught (in promise) Error: Attempt to postMessage on disconnected port
    postSysTrayXMessage moz-extension://bf674758-ea3d-458a-a039-7df2727bad79/background.js:644
    sendMailCountPre115 moz-extension://bf674758-ea3d-458a-a039-7df2727bad79/js/defaults.js:444
    listenerFolderInfoChanged moz-extension://bf674758-ea3d-458a-a039-7df2727bad79/background.js:258
    init moz-extension://bf674758-ea3d-458a-a039-7df2727bad79/background.js:109
    start moz-extension://bf674758-ea3d-458a-a039-7df2727bad79/background.js:1006
background.js:644:32
Uncaught (in promise) Error: Attempt to postMessage on disconnected port
    postSysTrayXMessage moz-extension://bf674758-ea3d-458a-a039-7df2727bad79/background.js:644
    focusChanged moz-extension://bf674758-ea3d-458a-a039-7df2727bad79/background.js:874
    promise callback*focusChanged moz-extension://bf674758-ea3d-458a-a039-7df2727bad79/background.js:873

.. which would equally make sense if it couldn't "connect" with the host 'app' so that should now be fixed! 👍

Will update further with some more news Soon™. Thanks for the tip! 😄

[Ximi1970]

Any updates?

[mjeveritt]

Any updates?

Ah, sorry for not getting back sooner!
I have one folder which reliably/consistently mis-counts, which gives me a reproducer.. !

I probably need to pull your latest git code to make sure the problem persists, but I don't believe you've made any significant changes to the 'counting' code?

I've seen some debugging statements in the code (or think I have) but not identified them in the Debug Console output in Thunderbird itself. For the time being I'm sticking with TB102.15 and Not upgrading to 115 Yet, so hopefully we can track down whatever this supposed race-condition might be...

Any debugging tips, and I'll get some more 'running data' to send you... 👍 TIA!

[mjeveritt]

Aha, yes I see some //console.debug statements in the code, but so far I will need to rebuild the plugin again.

I wonder if we could add a 'verbose' type option to block-toggle these print statements On/Off to watch the internals easily.. wdyt?

Gotta get some sleep in .. but will note this for a timeslot Soon™...

[Ximi1970]

For TB 115 you can use API count method for unread messages. I will not fully count the unread messages anymore but use the unread attribute of the folders. There is a bit of a delay by TB updating this attribute.
New messages are still counted the same as always and is slow when you have a lot of messages. I need to load the info for every message to see if it is new (and/or unread)....
So you could check the API count method in the preferences but do not use any new mail options to see if the new method is more accurate.
I also start now counting immediately when the addon is started by TB. The old version only started counting after a 5 sec delay build in due to a bug in TB not providing the accounts instantly on startup. You can change this delay in the preferences but it could result a more inaccurate count.
From TB 121 there is finally a new count attribute for a folder. No more counting done by me anymore. Only available for TB121 and up AND only on branch feature-fix-tb121 for now. But be careful using this branch because the database containing the count folders will be upgraded and is not compatible anymore with the current addon. Once the preferences/settings are automatically converted by the new addon, you can not use an older addon version without resetting all preferences.
I will only merge the branch into develop when I am sure the TB guys are done with their API changes...

The "verbose" option is indeed a good idea. I am getting a bit annoyed with myself continuously enabling and disabling the console.debug statements.