Search is broken/ useless

[jdaviescoates]

Describe the bug
If I do a search for suma or even specifically search subjects for Suma delivery day change it does not find an email with the subject Suma delivery day change

To Reproduce
Steps to reproduce the behavior:

1. Search forsuma or even specifically search subjects for Suma delivery day change
2. Don't find the email I am looking for
3. Pull my hair out
4. Give up and use Roundcube instead :(

Expected behavior
Search should work.

[the-djmaze]

Search does work for me.

The browser developer console Network tab reveals the search:

epoch reveals the time and date the search is made.
If that value is lower than the current epoch, the search is not executed and fetched from browser cache.

Code starts here:

snappymail/snappymail/v/0.0.0/app/libraries/MailSo/Mail/MailClient.php

Line 614 in ad6b51f

public function MessageList(MessageListParams $oParams) : MessageCollection

And in there the strlen($sSearch) part is the important searching code.
You will also notice that message_list_limit changes the behavior of search.

[jdaviescoates]

Search does work for me.

It's not that it doesn't work at all for me. Some results are shown, but loads that ought to be, aren't.

I'll investigate further when I find the time, but I'm fairly certain it isn't working as it should (because if it was search the subject field for the exact subject of the email I couldn't find would find that email).

[the-djmaze]

When logging is active, it should log something like:
Get Serialized UIDS from cache ("INBOX" / REVERSE DATE / suma) [count:0]
and
Save Serialized UIDS to cache ("INBOX" / REVERSE DATE / suma) [count:0]

What does yours say?

[virtualizeone]

Hello, I am having the same problem, I found out that these emails, when you see the source are with broken encoding.
like Subject: =?utf-8?B?VHJhbnNmZXLDqm5jaWFzIHJlY2ViaWRhcyBCQiAtIDEyLzA=?= , really don´t know why it happens to some emails and others don´t. But that is why we can´t find them in the search.

[the-djmaze]

Interesting. It seems your IMAP server can't decode base64 utf-8 text.

I will test if Dovecot also has issues with them.

[chsleemsi]

Hello, I am having the same problem.
[2023-11-16 08:49:35.771][def771e2] [INFO]: Get Serialized UIDS from cache ("INBOX" / / UNDELETED) [count:23]
[2023-11-16 08:49:35.771][def771e2] [INFO]: Get Serialized UIDS from cache ("INBOX" / / OR OR OR FROM "VMware End of Support information" TO "VMware End of Support information" CC "VMware End of Support information" SUBJECT "VMware End of Support information" UNDELETED) [count:0]
[
Actually, it is the 23rd message but it said no message found.

[the-djmaze]

Interesting. It seems your IMAP server can't decode base64 utf-8 text.

I will test if Dovecot also has issues with them.

Dovecot works fine.
Which IMAP daemon are you using?

[chsleemsi]

Sorry, I am not sure what do you mean by IMAP daemon. I am using Nextcloud v26 running Snappymail. The IMAP server secure is SSL/TLS, 993

[the-djmaze]

Sorry, I am not sure what do you mean by IMAP daemon.

We need the name of your IMAP software.

[chsleemsi]

Do you mean the IMAP server which is mail.psmail.net?!

[mschilli87]

@chsleemsi: I think they refer to the software that is responding to IMAP requests made to that server on that server. For example, I use dovecot on my server, but somebody else might opt for mailu instead. There are many options out there and I think that's what the question is aiming at.

[chsleemsi]

Thanks mschilli87! Sorry for my ignorant. However, I don't know how to find out what software I am using. We use SnappyMail v2.29.4 on Nextcloud v26.0. The email domain is set up through SnappyMail - Admin Panel. The Nextcloud is hosted on Ubuntu VM.

We used RainLoop before but it is not compatible with Nextcloud v26.0 that was why we migrated to SnappyMail recently and which RainLoop suggest.

There was no search problem on RainLoop but all of the users facing the problem about Search after we have migrated to SnappyMail.

[mschilli87]

@chsleemsi:

I don't know how to find out what software I am using.

Maybe you could consult https://info.psmail.net/xsupport/?

[jdaviescoates]

OK, so searching for the emails SnappyMail couldn't previously find as mentioned in the OP does now work...

BUT I've just hit another even more serious example of search being broken/ useless! 😭

I just searched for an email from a specific email address by searching for the exact email address and it doesn't find it! Again, Roundcube finds it fine.

If I do:

from:natasha

It actually finds the email fine.

But if I do:

natasha@example.com

It doesn't find anything at all.

Same with:

from:natasha@example.com

What's going on?!?

[the-djmaze]

To find out, we need the log.
https://m.youtube.com/watch?v=FR-REDevAzk

[chsleemsi]

Morning djmaze, don't know why but the search working now. I guess someone from the server side did something. Thank you so much!

[jdaviescoates]

To find out, we need the log.
https://m.youtube.com/watch?v=FR-REDevAzk

Haha, I thought that was perhaps going to be handy view about exactly which logs you want and where exactly to find them! But now, it's a clip from Ren and Stimpy (I used to enjoy that show).

I've just enabled logging here at my-nextcloud-url/apps/snappymail/?admin#/config

But not sure what (if anything) to put in the path?

I also checked this box here in my-nextcloud-url/settings/admin/additional

What next?

[the-djmaze]

By default the lognis in the data directory of Nextcloud.
Something like data/app_snappymail/default/logs

Haha, I thought that was perhaps going to be handy view about exactly which logs you want and where exactly to find them! But now, it's a clip from Ren and Stimpy (I used to enjoy that show).

Me too, and it is about "log" 🤣

[jdaviescoates]

OK @the-djmaze so here are some recent logs:

https://paste.uniteddiversity.coop/?d30b147d210a9f32#CU7VfW1gy6Rj1xna47c8tzSReXhyPcZCGhfKZcVB3zBp (pw: snappy)

I'm still finding the search to be pretty useless 😭 I just searched for inov8 and it didn't find most of the related emails. Then again when searching for enquiries@inov-8.com it didn't find any of the emails from that address.

I may have to just go back to using Roundcube as search works pretty well there. But that'd be a shame as I otherwise like SnappyMail and how it's nicely integrated into Nextcloud. Hopefully the logs will shed some light on why it's not working.

[the-djmaze]

SnappyMail relies on the search capabilities of the IMAP server.
Roundcube downloads all messages and seaches in its own database.

[jdaviescoates]

from=emailaddress still doesn't work which is rather frustrating

SnappyMail relies on the search capabilities of the IMAP server.

Is this a SnappyMail issue or is it because of the search capabilities of the IMAP server?

How can I help to investigate why this isn't working?

[the-djmaze]

Is this a SnappyMail issue or is it because of the search capabilities of the IMAP server?

Probably the IMAP server.

Using dovecot i can even search unicode:

But this fails:

But the IDN version works:

[jdaviescoates]

Using dovecot i can even search unicode:

Dovecot is the exact same IMAP server that my system is running too though 🤔

[jdaviescoates]

What's odd is that e.g. searching for admin@sustainablebridport* works, but searching for admin@sustainablebridport.com fails to display any results.

admin@sustainablebridport.* works too, but as soon as there are any characters after the dot it fails

[the-djmaze]

Open a message from the search that works.
Look in the source From, To, boundary etc. headers for issues.

You can also forward th email as attachment to issue-1096@snappymail.eu and i will take a look/test.