afew is very slow

[greened]

For some reason afew is taking a very long time (like a half hour or more) to process mail. Admittedly there are many mails (hundreds of thousands) but afew sits at the "Commit changes to N messages" for long periods.

Even when only processing new mail, afew takes much longer than I would expect.

[GuillaumeSeren]

Hello,
maybe you should check your backup before continue 🤣

Joke aside, to help you we need more informations,
find a command that produce the problem each time, some config param etc

I have a nearly 900k mail in my boxes and it works fine

[DearRude]

Even when only processing new mail, afew takes much longer than I would expect.

Same here. Even when there is no new mail, afew takes some seconds to process for me.

[GuillaumeSeren]

Hey @greened , @DearRude ,
can you get the duration of the request in notmuch vs afew (you can see those in verbose mode).

[DearRude]

dearrude@topol:~ » time notmuch new  
No new mail.
notmuch new  0.01s user 0.00s system 57% cpu 0.013 total
dearrude@topol:~ » time afew --new --tag -v
INFO:afew.filters.ListMailsFilter.ListMailsFilter:Tagging mailing list posts
INFO:afew.filters.ArchiveSentMailsFilter.ArchiveSentMailsFilter:Archiving all mails sent by myself to others
INFO:afew.filters.BaseFilter.Filter:Delete all spams
INFO:afew.filters.BaseFilter.Filter:Committing changes to 269 messages
INFO:afew.filters.BaseFilter.Filter:Filter from git
INFO:afew.filters.BaseFilter.Filter:Committing changes to 291 messages
INFO:afew.filters.BaseFilter.Filter:Filter all OSM related
INFO:afew.filters.BaseFilter.Filter:Committing changes to 2178 messages
INFO:afew.filters.BaseFilter.Filter:Sent from mail provider admin
INFO:afew.filters.BaseFilter.Filter:Committing changes to 6 messages
INFO:afew.filters.BaseFilter.Filter:All newsletters
INFO:afew.filters.BaseFilter.Filter:Committing changes to 114 messages
INFO:afew.filters.BaseFilter.Filter:No message specified for filter
INFO:afew.filters.BaseFilter.Filter:Committing changes to 596 messages
afew --new --tag -v  6.04s user 3.14s system 95% cpu 9.592 total

[GuillaumeSeren]

Hello @DearRude,
well nothing very strange to me here.

No new mail so nothing to do for notmuch 0.01s

Then each line in afew is doing a real query and adding some tag, and ~10s of afew run is not so slow if you consider how much stuff it does (it could be faster of course).

If you want to compare, you need to get a single process like:

INFO:afew.filters.BaseFilter.Filter:Filter from git
INFO:afew.filters.BaseFilter.Filter:Committing changes to 291 messages

Add a more verbose output to get the actual query, run each changes manually + time,
then compare it to a run of afew of this specific filter.

[DearRude]

Is it necessary for afew to query all mails even when there is no new mail? I don't want the existing mails to change at all.

[GuillaumeSeren]

Is it necessary for afew to query all mails even when there is no new mail? I don't want the existing mails to change at all.

Probably as afew is not the database itself, it has to rely on notmuch to return the list,
even if the list is empty, but maybe I missread you comment please explain.

[DearRude]

Committing changes to 269 messages

As listed in the verbose output, afew is committing changes to all mails. However, I'm running it with --new flag which only have to affect new mails. Does afew have to commit change to all mails even if --new flag is passed? Or simply afew can't tell the difference between new and old mails in my case?

[GuillaumeSeren]

Hey @DearRude ,
afew rely on notmuch for this, you need to configure notmuch to tag new mail, like:

[new]
tags=new
...

So if you use the --new option it will only target those new mail, if you increase the verbose level more,
you will see the query.

I was trying to say before, afew may be slow but you have to compare it to all the queries / changes it does.

[DearRude]

Ah got it. Thank you!

[DearRude]

Hey @GuillaumeSeren
Is there any chance you could add the feature of customizing the new tag from afew?

What I mean is that afew looks for other tags like unread instead of new?