Skip unread and Next / Previous sibling folder don't follow tree order

[fredstluka]

I've found 3 QF features that all seem to go to the wrong "next" folder.

When I hit Shift-S (skip unread folder), it correctly skips the unread messages in the current folder. But I don't understand how it chooses the next folder to skip to.

Ideally (in my opinion) it would go to the next subfolder in the folder tree that has unread messages. By "next", I mean the same folder that the native TB "n" keystroke (next unread) would go to, if I had first read all the messages in the current folder. Instead, it goes to some other folder, but it seems random to me. I don't know how it chooses which folder.

I see the same problem in other features of QF. The same (seemingly random, but apparently not since it's repeatable) sequence of folders is used by:

1. Opt-Left/Right (cycling through folders in the current tab in same sequence), if enabled in the QF settings
2. Left/Right arrow buttons in the QF toolbar

The desired order (order used by "n") IS used by the "Move to the next [unread] message" button in the QF toolbar. So, perhaps the same code used by that button to choose the "next" folder should also be used by the 3 cases above (Shift-S, Opt-Left/Right, Left/Right arrows in QF toolbar)?

Thanks!
--Fred

[RealRaven2000]

Ideally (in my opinion) it would go to the next subfolder in the folder tree that has unread messages. By "next", I mean the same folder that the native TB "n" keystroke (next unread) would go to, if I had first read all the messages in the current folder. Instead, it goes to some other folder, but it seems random to me. I don't know how it chooses which folder.

ideally I would agree, not aware that it wouldn't go to a sibling next folder (within the parent) if there are unread folders, provided the folder iterator passes this on; I need to deep dive into the code to see how it was implemented.

Opt-Left/Right (cycling through folders in the current tab in same sequence), if enabled in the QF settings

that's a different algorithm, but that should go through the different tabs in visual order. The tabs are stored in a big array and painted / iterated in that order, even if you use a filtered set of tabs. tested here (in Windows) using ALT+Left and ALT+Right, it works as expected. Does yours skip tabs or even walk into the wrong direction (go left when you click right)?

Left/Right arrow buttons in the QF toolbar

do you mean the current folder (navigation) toolbar?

The desired order (order used by "n") IS used by the "Move to the next [unread] message" button in the QF toolbar. So, perhaps the same code used by that button to choose the "next" folder should also be used by the 3

not easily possible as it uses an internal function which does too much (for instance I couldn't use it for skip folder), it uses a command handler with predefined actions, again let me look at it in more detail. It's possible that it uses the C++ layer to do these things and it's not easy to convert / reinvent that particular wheel with JavaScript. As far as I remember I am using a folder iterator that accepts the folders in an order that is given to me by Thunderbird.

[fredstluka]

Opt-Left/Right (cycling through folders in the current tab in same sequence), if enabled in the QF settings

that's a different algorithm, but that should go through the different tabs in visual order. The tabs are stored in a big array and painted / iterated in that order, even if you use a filtered set of tabs. tested here (in Windows) using ALT+Left and ALT+Right, it works as expected. Does yours skip tabs or even walk into the wrong direction (go left when you click right)?

Axel, it just occurred to me that you are probably talking about QF tabs. I'm talking about native TB tabs. I have several native TB tabs open, but no QF tabs. I really LOVE QF, but it use it mostly for the Shift-M (Move), Shift-J (Jump) and Shift-T (Copy) functions. I'm starting to use Shift-S (Skip unread folder) more often now too. And may soon start using the Search function, since native TB doesn't support regular expressions.

So, when I hit Opt-Left/Right, I'm seeing that TB stays at the same TB tab, but that tab now shows a different folder. I haven't figured out how it chooses the folder to go to next. Maybe it's confused because I don't have any QF tabs defined?

I expected it to move in the same sequence as the native TB "n" (Next unread). That is, I expected it to do a depth-first tree traversal of the TB folder tree, expanding collapsed folders as needed. So, if the TB folder tree were already fully expanded, it would be moving vertically down by one folder/subfolder in that tree. That's the behavior I see with the native TB "n" (except of course that it skips folders with no unread messages).

do you mean the current folder (navigation) toolbar?

Yes.

The desired order (order used by "n") IS used by the "Move to the next [unread] message" button in the QF toolbar. So, perhaps the same code used by that button to choose the "next" folder should also be used by the 3

not easily possible as it uses an internal function which does too much (for instance I couldn't use it for skip folder), it uses a command handler with predefined actions, again let me look at it in more detail. It's possible that it uses the C++ layer to do these things and it's not easy to convert / reinvent that particular wheel with JavaScript. As far as I remember I am using a folder iterator that accepts the folders in an order that is given to me by Thunderbird.

OK. I hope you can figure out a way to reuse the existing TB "n" code. If not, perhaps replicate its behavior by doing your own depth-first tree traversal of the TB folder tree?

If this is a lot of work, don't worry about it. I currently do pretty much all of my navigation via Shift-J, and "n". I'd use Shift-S occasionally, if it followed the sequence I was expecting. But I can live without it because I've carefully named all of my folders/subfolders with numeric prefixes to force TB to display them in the order I want.

So I open TB each day and just hit "n" over and over to go through all of my email in priority order. Each time I fetch new messages, I use Shift-J to jump to the first folder in my tree, and resume using "n" over and over. And, of course, I use Shift-M and Shift-T constantly, to file away messages as I read them and deal with them.

The only time I need Shift-S is when a folder contains a lot of new messages, but my priorities have changed so I want to skip them until I've seen the rest of my new mail. If that happens often enough, I rename that folder, giving it a larger numeric prefix, so it moves to the end of the folder tree. Then I don't have to skip it as often.

As always, thanks for your excellent support!
--Fred

[RealRaven2000]

So, when I hit Opt-Left/Right, I'm seeing that TB stays at the same TB tab, but that tab now shows a different folder. I haven't figured out how it chooses the folder to go to next. Maybe it's confused because I don't have any QF tabs defined?

no - left right is explicitly designed to navigate QuickFolders Tabs, nothing else. Best to disable the feature if you do not have any, I am not making an exception for the small number of users who use QuickFolders without defining any QuickFolders.. (But I am aware that use case exists as well)

I expected it to move in the same sequence as the native TB "n" (Next unread). That is, I expected it to do a depth-first tree traversal of the TB folder tree, expanding collapsed folders as needed. So, if the TB folder tree were already fully expanded, it would be moving vertically down by one folder/subfolder in that tree. That's the behavior I see with the native TB "n" (except of course that it skips folders with no unread messages).

from what I can see here, when the folder tree (which I never use) has focus and no QuickFolders tab is currently selected, ALT+left / right here goes to the next sibling folder of the same parent. It's been a while since I wrote the code, so I only recalled by debugging, but here it is:

          case "ArrowLeft":
            if (!this.goPreviousQuickFolder()) {
              this.goPreviousSiblingFolder();
            }
            isHandled = true;
            break;
          case "ArrowRight":
            if (!this.goNextQuickFolder()) {
              this.goNextSiblingFolder();
            }
            isHandled = true;
            break;

this does the same as the 2 left / right navigation buttons on the Current Folder (Navigation) bar center screen. Since Thunderbird 88 the child folders of a folders a are stored in an array, I follow their "natural order" - which seems to work in Windows, even in the latest beta (Tb 134.0b3):

In the example on my test instance it followed the path of sibling folders alphabetically (from issue 181, through issue 228 etc. until test1), however then special characters like Umlaut Ä seem to come after lowercase ASCII characters in the order but not in the tree which seems to be ordered lexicographically:

it certainly would be interesting to find out how the "stored" order of the array differs exactly, and if I am only traversing top level subfolders, re-sorting lexicographically (if that's really what is shown in the folder tree, I am just guessing) may be not overly expensive performance wise. I guess that people with many folders have them in deeper hierarchies (rather than a folder that has 10,000 child folders).

The only time I need Shift-S is when a folder contains a lot of new messages, but my priorities have changed so I want to skip them until I've seen the rest of my new mail. If that happens often enough, I rename that folder, giving it a larger numeric prefix, so it moves to the end of the folder tree. Then I don't have to skip it as often.

that's exactly what I designed this one for, Skip is important when you have a few social folders you want to follow up on (or read) later and want to skip to the more important parts of your daily workload. I probably don't quite have as many in that group (my estimate is I probably traverse around 30-45 folders every morning), I spend most of my time in the support and license folders, to work through my fan / shopping mails, before I check out any mailing lists. Part of the discipline is to answer user queries / problems first, in chronological order, this way I am avoiding large back logs. And of course github helps me staying on top!

[RealRaven2000]

I added a lexical sorting routine for siblings, which should hopefully follow the folder tree order:

QuickFolders-mx-6.8.3pre30.zip

can you check whether it is reasonably performant on a folder with many subfolders? It sorts the subfolders using localeCompare every time the next / previous sibling button is pressed.

[fredstluka]

no - left right is explicitly designed to navigate QuickFolders Tabs, nothing else. Best to disable the feature if you do not have any, I am not making an exception for the small number of users who use QuickFolders without defining any QuickFolders.. (But I am aware that use case exists as well)

OK. Fair enough. I'll be happy to disable the feature in the QuickFolders Options. Or to leave it enabled and just not use it. As I said, I wasn't making much use of the Opt-Left/Right keys or the Left/Right buttons of the QuickFolders toolbar anyhow. No problem!

However, this behavior DOES still affect the sequence used by Shift-S, which I may use occasionally (though still not often).

from what I can see here, when the folder tree (which I never use) has focus and no QuickFolders tab is currently selected, ALT+left / right here goes to the next sibling folder of the same parent.

I've played with it some more... As expected, the Opt-Left/Right keys and the Left/Right buttons of the QuickFolders toolbar both seem to use the same sequence. As you said, they move among the siblings of a single parent folder. And they never move to any folder outside of that parent folder.

However, they do NOT move through those siblings in alphabetic order. Instead, they seem to move through them in some other order, according to some pattern that I haven't yet detected. Perhaps the order in which the folders were created or something? That might correspond to their order in the array, with the alphabetic order that I see in the folder tree being produced by sorting on the fly.

They ARE repeatable. That is the same order within a single parent is always used. And they ARE symmetric. That is the left and right operations DO go through the same sequence, but in opposite directions.

Also, I've just noticed that Shift-S does NOT use the same sequence. It does NOT stick to sibling folders of a single parent folder. Instead it goes to some siblings of one parent, then to some siblings of another parent, then perhaps back to more siblings of the first parent, etc.

... however then special characters like Umlaut Ä seem to come after lowercase ASCII characters in the order but not in the tree which seems to be ordered lexicographically:

Yeah, non-ASCII chars lead to questions about which sorting order is preferred. As do upper/lower case chars (case-sensitive or insensitive sort), and also numeric digits (alphabetic or numeric sort).

None of that matters much to me though. I tinker with the names (adding leading zeros to achieve numeric order when it's really doing an alphabetic sort, etc.) until I get the TB folder tree to display them in the order I want. Then I use "n" which traverses the entire tree in that same order.

It would be nice (but again, low priority for me) if Shift-S used that same order.

it certainly would be interesting to find out how the "stored" order of the array differs exactly, and if I am only traversing top level subfolders, re-sorting lexicographically (if that's really what is shown in the folder tree, I am just guessing) may be not overly expensive performance wise. I guess that people with many folders have them in deeper hierarchies (rather than a folder that has 10,000 child folders).

I agree. People either have relative few folders, or they organize their large number of folders into a tree. Less likely to have a massive number of sibling folders in the same parent. So, sorting lexicographically on the fly should be fast enough. As I said above, I suspect that the native TB code sorts them on the fly, and it seems fast.

that's exactly what I designed this one for, Skip is important when you have a few social folders you want to follow up on (or read) later and want to skip to the more important parts of your daily workload.

Yes, that's exactly what appealed to me about Skip.

I probably don't quite have as many in that group (my estimate is I probably traverse around 30-45 folders every morning), I spend most of my time in the support and license folders, to work through my fan / shopping mails, before I check out any mailing lists. Part of the discipline is to answer user queries / problems first, in chronological order, this way I am avoiding large back logs. And of course github helps me staying on top!

Same for me. I hit about 40-50 folders each morning. Each folder has 0-15 or so siblings. The parents are things like Clients, Personal, Alerts, Finance, Social, News, Lists, etc.) So, in my case, sorting each group of siblings should be VERY fast.

For me, each folder has typically 0-10 new messages. I read the messages, deal with them, and delete them or file them away (via Shift-M -- thanks!). So, there are usually very few total messages in each folder that received filtered incoming (new) messages. I have another 600+ folders that are much larger, that I file messages away into for years or even decades.

--Fred

[fredstluka]

I downloaded and installed 6.8.3pre30. It works perfectly, and VERY fast, to solve the problem with the Opt-Left/Right keys and the Left/Right buttons of the QuickFolders toolbar. Perfect! Thanks!

However, it does NOT yet have any effect on Shift-S.

I don't have any folders with more than a couple dozen subfolders. So, I can't really comment on its performance with huge numbers of subfolders. However, as we discussed, I suspect that's an unusual use case.

Thanks for the quick response! Excellent support as always!
--Fred

[RealRaven2000]

I downloaded and installed 6.8.3pre30. It works perfectly, and VERY fast, to solve the problem with the Opt-Left/Right keys and the Left/Right buttons of the QuickFolders toolbar. Perfect! Thanks!

However, it does NOT yet have any effect on Shift-S.

yes because that is a search algorithm, and it has a different order - it is not alphabetical but applies different algorithm to match the closest match which can also depend on the number of characters and position of the text within the string (I think if you only enter 2 letters it only looks at the beginnings of "words" which also includes text that is preceded by various special characters such as . + - & _ and of course spaces. It's complicated. the idea is of course the more letters you enter, then less hits (and the more relevant results) you should get.

I was also considering sorting either by most recently used or by "most used" depending on frequency, but I haven't really had enough conviction on these to embark on an implementation journey. Note that you can narrow down the number of results byL

• only entering 2 letters (the algorithm concentrates on word beginnings)
• add the name of the parent folder, e.g. to get to customers/acme, just enter c/a or cu/ac

[fredstluka]

Axel,

However, it does NOT yet have any effect on Shift-S.

yes because that is a search algorithm...

I think you're talking about Shift-M (Move), Shift-T (Copy) and Shift-J (Jump).
I LOVE all those features that you described, of the search algorithm that they use. I've watched as you made those refinements one after another, always improving my search experience, often with direct input from me along the way.

However, in this case, I'm talking about Shift-S (Skip).
It does not need to do any searching, word-prefixes, etc. It just needs to immediately jump to the "next" folder that has an unread message.

I think the "next" folder should be determined by the same algorithm you're now using for the Opt-Left/Right keys and the Left/Right buttons of the QuickFolders toolbar, with one exception.
Instead of simply iterating through the siblings in a parent folder, ideally I'd like to see it do a full depth-first traversal of the folder tree.

That is, when I'm at a sibling in a parent folder, and hit Shift-S, it would skip to the next folder with unread messages using the following definition of "next":

1. If the sibling has no children of its own, go to the lexicographically next sibling, exactly as your recent change does for the Opt-Left/Right keys and the Left/Right buttons of the QuickFolders toolbar.
2. If the sibling DOES have children, go to the lexicographically first sibling among those children.
3. When the sibling is the lexicographically LAST sibling in its parent folder, go to the sibling of the parent folder that is lexicographically next after the parent. Do NOT cycle back to the lexicographically first sibling inside the parent folder, as your recent change does for the Opt-Left/Right keys and the Left/Right buttons of the QuickFolders toolbar. And if that sibling of the parent folder is the LAST sibling in ITS parent folder (in the grandparent folder), go to the sibling of the grandparent folder that is lexicographically next after the grandparent. And so on, up the tree as necessary.

I realize this is more work than simply cycling through the list of siblings. And it would require an additional lexicographical sort each time it moved up or down a level of the tree. But I think it would still be very fast because most users don't have more than a couple of levels of nesting.
Even with my 700 or so folders, I never go more than 5 levels deep. And among the folders that get new messages filtered to them, I never go more than 3 deep.

That would be the most useful algorithm for me, but as I said, I can live without Shift-S if this is too much trouble for you to implement. As we've discussed, I'm kind of an edge case for you, "using QuickFolders without defining any QuickFolders".

On the other hand, once you've implemented this depth-first search algorithm, I think you'll find other uses for it. So maybe as an investment in the future?

Hopefully yours,
--Fred

[RealRaven2000]

The skip folder function (QuickFolders.Interface.onSkipFolder()) calls my utility function QuickFolders.Util.getNextUnreadFolder(currentFolder) to determine the next unread folder target. (It cannot use a built in method of Thunderbird because they tend to remain in the current folder even if you are below remaining unread mails in the timeline.

In order to determine the list of folders this calls my utility function QF.Util.allFoldersIterator(false) which creates an iterator (<=function that lists enumerable objects, such as an array) across all folders.

The iterator uses an array of the built in function MailServices.accounts.allFolders() - which can return 10,000s of folders, and then iterates over the array, thereby excluding some folders (such as the ones that are excluded from search via Tab-specific properties and excluded accounts), and then builds a new array of final folders to consider for the jump.

My understanding is that the folders returned into this Array should mimick the order in the folder tree, both as regards visual order of accounts and the actual folder tree nodes. If there are no other unread folders left and it ends up in the current folder again, it will return the current folder.

Just testing the iterator vs the tree while debugging, there is a slight inconsistency here:

acctMgr.allFolders.map(a => a.prettyName);

once the array is returned, the getNextUnreadFolder function iterates it and calls folder.getNumUnread(false) to find only folders that have unread mails in the first level (not their children) and then runs logic on them.

The main loop in getNextUnreadFolder() then goes through the array from the top until it hits the current folder, then goes to the next folder until it hits another unread folder where it breaks its flow.

We could potentially rebuild a much shorter array of all unread folders first and then try to lexicographically sort all clusters that have the same parent folder (obviously not sort all folders) this may not be as resource intensive than trying to sort the complete array, before applying the logic of finding the next (unread) folder.

[RealRaven2000]

Considerations for rebuilding the Array - (even without sorting it) - in order to make a complete array we need to call getNumUnread() on every folder in all included accounts - there is no instant shortcutting at the next unread folder, so this may perform slightly (or significantly) worse with many folders if the unread folders are early on / on top of a long list of folders. I would need to do some performance testing with a large account to see how the unread counter performs, then think of the resorting logic next.

Ideally that may be a little complicated as an algorithm (clustering along parent folders) but shouldn't affect performance assuming there are not silly amounts of unread folders. (Anything below 1,000 unread folders can probably be handled with ease).

[RealRaven2000]

OK, it took a while to come up with a reliable algorithm, here is a test version that sorts the unread folders in lexical order. Making the algo was quite tricky as it first only wanted to group together and sort folders of the same parent but not keep subfolders within them in the group but appended them to the end. So we came up with a way of creating an artificial folder path using prettyName instead of the real (often encoded) parts of the URL, and then were able to use localCompare for generating the same lexical order that I suspect the folder tree is in:

QuickFolders-mx-6.8.3pre44.zip

In order to keep the correct order of servers / accounts we replaced them with id001..id999. It would be interesting to see performance on a system with many folders. Since sorting only affects unread (plus the current) folder it shouldn't take too much extra time, but I am considering putting the algorithm behind an option.

Here are the main lines I added to getNextUnreadFolder() :

   // Step 1: Build a map to store server identifiers
    const accountMap = new Map();
    let serverIdCounter = 1;

    function getServerUri(f) {
      return f.URI.match(/^([a-zA-Z]+:\/\/[^\/]+)/)[0]; // Extract the server part (e.g., "imap://example.com")
    }

    // Step 2: Recursive function to build the path based on prettyName
    function buildPath(folder) {
      // Base case: if the folder has no parent, return its own prettyName
      if (!folder.parent) {
        return folder.prettyName;
      }

      // Recursively build the path from parent folder + / + prettyName
      return buildPath(folder.parent) + "/" + folder.prettyName;
    }

    util.logDebug("iterating all folders");

    // Step 3: Build the list of folders, respecting unread mail filter and account map
    const allFolders = [];
    for (let folder of util.allFoldersIterator(false)) {
      if (folder == currentFolder) {
        // we must always include current folder even if it is unread!
      } else {
        // Only consider folders with unread mails
        if (folder.flags & unwantedFolders) continue;
        if (!folder.getNumUnread(false)) continue;
      }
      allFolders.push(folder);

      // Step 4: Assign a unique identifier for each server if not already assigned
      const serverUri = getServerUri(folder); // Extract the server part (e.g., "imap://example.com")
      if (!accountMap.has(serverUri)) {
        accountMap.set(serverUri, `id${String(serverIdCounter).padStart(3, "0")}`);
        serverIdCounter++;
      }
    }

    util.logDebug(
      "Unread folders + account map",
      allFolders.map((a) => a.parent.prettyName + "/" + a.prettyName),
      Array.from(accountMap.entries())
    );


    // Step 5: Build a new "full path" for sorting, using prettyName and server identifier
    const folderPaths = allFolders.map((folder) => {
      const serverUri = getServerUri(folder); // Extract the server part
      const serverId = accountMap.get(serverUri); // Get the server identifier
      const relativePath = buildPath(folder); // Get the path from prettyName

      // Combine the server identifier with the folder's relative path for sorting
      const fullPath = `${serverId}/${relativePath}`;

      return { folder, fullPath };
    });

    // Step 6: Sort the folders by their full path (which includes the server ID)
    folderPaths.sort((a, b) => {
      // Split both full paths into components
      const pathA = a.fullPath.split("/");
      const pathB = b.fullPath.split("/");

      // Compare each component of the paths using localCompare
      for (let i = 0; i < Math.max(pathA.length, pathB.length); i++) {
        const componentA = pathA[i] || ""; // Use empty string for undefined components
        const componentB = pathB[i] || "";

        // Use localCompare to compare the components lexicographically
        const comparison = componentA.localeCompare(componentB, undefined, { sensitivity: "base" });

        if (comparison !== 0) {
          return comparison; // Return the result of the comparison
        }
      }

      // If the paths are identical, return 0 (they are equal)
      return 0;
    });

    util.logDebug("Map of sorted paths: ", folderPaths);
    // Step 7: Extract the sorted folders from the result
    const sortedFolders = folderPaths.map((item) => item.folder);

[jaysmithgit]

Please do put this (anything that requires reading the entire folder structure) "behind an option".

I have not counted recently, but our IMAP has more than 10,000 folders and certainly well over 100,000 (maybe over 250,000 by now) emails. To what extent the folders/emails would be classified as "unread" or not depends on factors beyond my knowledge. Every couple of years, each of the users' IMAP local summary summary becomes corrupted to the extent that I have to (for each user!!!) delete TB's local (per user) summary folders/files and rebuild them. The rebuilding is a very long (takes hours) process with a lot of manual intervention (at least on the version of TB that we are using) to "open" (expose) each level of folders so that TB can create the summary folders and files. However, until the user actually clicks on a folder in tree, TB does not display how many emails are in the folder nor their read/unread status. So, that suggests to me that -- at some level or by some parts of the TB or QF programs -- they would all be considered to be "unread" following such a rebuild process. Since we have a folder structure around 10 levels deep, reading that depth of structure has in the past, but I am not sure about currently caused TB to lock up after a couple hours of trying to automatically (i.e. if IMAP local summary rebuilding were done by the automatic feature) navigate the full structure -- thus the rebuilding is done manually. This may no longer be a problem for TB -- I have not wanted to find out!

=-=-=

BTW, we handle the general issue of isolating and viewing unread messages by creating an "incoming" folder structure. Everything that is coming in from the outside either goes into the user's own Inbox OR is filtered at the server level (i.e. dovecot sieve) (and not emails filed into a folder by another IMAP user), into a modest-size group of folders in an incoming structure. In our case, that "incoming" structure is on the "Public" side of the IMAP (i.e. Shared), so all our users can have access to those emails. In our case, our business email addresses/users are role-based so most incoming email ends up in the shared "incoming" structure. (If some of our users need private non-shared email, we give them a different email address that filters into their personal Inbox.)