ISSUE: Error searching emails: SEARCH command error: BAD [b'Error in IMAP command SEARCH: Invalid characters in atom (0.001 + 0.000 secs).']

[segdy]

Describe the bug
I have set this up with my dovecot server. Unfortunately I do not receive notifications in home assistant.
Looking in the log file, I see lot of lines:

2022-06-17 14:32:56 ERROR (SyncWorker_8) [custom_components.mail_and_packages.helpers] Error searching emails: SEARCH command error: BAD [b'Error in IMAP command SEARCH: Invalid characters in atom (0.001 + 0.000 secs).']

This seems to be similar to jstedfast/MailKit#572

Environment (please complete the following information):

• OS: Ubuntu 20.04.4 LTS
• Type of device: Odroid C4
• Home Assistant version: 2022.6.6
• Component version: 0.3.4-2 (latest via HACS)

Logs
See above

[firstof9]

Check if the latest beta version still produces the error.

[segdy]

Just switched to 0.3.5-b3 but sadly the same.

[segdy]

If it helps, I added debug messages and this is a query that fails (non UTF8 mode):

(OR OR OR FROM "USPSInformedDelivery@usps.gov" FROM "USPSInformeddelivery@email.informeddelivery.usps.com" FROM "USPSInformeddelivery@informeddelivery.usps.com" FROM "USPS Informed Delivery" SUBJECT "Your Daily Digest" SINCE 17-Jun-2022)

[firstof9]

That's valid IMAP syntax. What mailserver?

[segdy]

# dovecot --version
2.3.13 (89f716dc2)
# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye

How could I test this manually against the mail server to ensure mail server is OK? In other words, what are the raw commands I need to send?

The following one does not:

# openssl s_client -showcerts -connect mail.example.com:993 -servername mail.example.com
[...]
* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ AUTH=PLAIN AUTH=LOGIN] Dovecot (Debian) ready.
01 LOGIN USERNAME PASSWORD
01 OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY STATUS=SIZE SAVEDATE LITERAL+ NOTIFY] Logged in
SEARCH (OR OR OR FROM "USPSInformedDelivery@usps.gov" FROM "USPSInformeddelivery@email.informeddelivery.usps.com" FROM "USPSInformeddelivery@informeddelivery.usps.com" FROM "USPS Informed Delivery" SUBJECT "Your Daily Digest" SINCE 17-Jun-2022)
SEARCH BAD Error in IMAP command: Invalid command name (0.001 + 0.000 secs).

[firstof9]

You have to select the folder first if I recall, then you can do a SEARCH

[segdy]

Thanks! Ok, I tried this and it seems to work fine:

# openssl s_client -showcerts -connect mail.example.com:993 -servername mail.example.com
[...]
* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ AUTH=PLAIN AUTH=LOGIN] Dovecot (Debian) ready.
01 LOGIN USER PASS
01 OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY STATUS=SIZE SAVEDATE LITERAL+ NOTIFY] Logged in
02 SELECT INBOX
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
* 4 EXISTS
* 0 RECENT
* OK [UNSEEN 4] First unseen.
* OK [UIDVALIDITY 1655449399] UIDs valid
* OK [UIDNEXT 5] Predicted next UID
02 OK [READ-WRITE] Select completed (0.001 + 0.000 secs).
03 SEARCH (OR OR OR FROM "USPSInformedDelivery@usps.gov" FROM "USPSInformeddelivery@email.informeddelivery.usps.com" FROM "USPSInformeddelivery@informeddelivery.usps.com" FROM "USPS Informed Delivery" SUBJECT "Your Daily Digest" SINCE 17-Jun-2022)
* SEARCH 4
03 OK Search completed (0.004 + 0.000 + 0.003 secs).
04 LOGOUT
* BYE Logging out
04 OK Logout completed (0.001 + 0.000 secs).
closed

[firstof9]

I'd need to see the full debug output to pin down where the issue is.

[segdy]

Thanks; can you help me, how do I generate the debug output? (I assume from the extension)

[firstof9]

We cover that in the wiki. 😄

[segdy]

Awesome, with the debug output I even saw an issue myself. It's the "Amazon forwarded email address" field. Something does not get quoted properly, so an uneven number of quotation marks. Example:

2022-06-17 17:24:52 DEBUG (SyncWorker_2) [custom_components.mail_and_packages.helpers] DEBUG imap_search: (FROM " "my-email@example.com" SUBJECT "Delivery update:" SINCE 17-Jun-2022)

My question: What is exactly this field, why is it needed, what should I put in and in which format?

Previously I had my two email addresses in, both in quotation marks, separated by a comma. Now I just have the single email address I have registered via my Amazon account, without quotation marks and I think the error is gone.

EDIT: To make it clear, my E-Mail address that I have registered with Amazon is amazon@example.com and I forward it to notifications@example.com. The documentation is unclear about the purpose and format of this field...

[firstof9]

What is exactly this field, why is it needed, what should I put in and in which format?

It's used for those who forward their package emails to an alternate email. If you don't do this enter "" into this field.

If you do use it, which your edit sounds like you do, only enter the email address, it shouldn't need quotes, of the email address the forwarded emails will be coming from.

[segdy]

It's used for those who forward their package emails to an alternate email. If you don't do this enter "" into this field.

May I ask why this is needed? Can't we just filter based on Amazon's address (similar as it's done for the other providers)?

If you do use it, which your edit sounds like you do, only enter the email address, it shouldn't need quotes, of the email address the forwarded emails will be coming from.

So this would then be the address of my Amazon account, right?

If my Amazon account is registered to amazon@example.com. And in this account (which I own) I created a sieve script that forwards the mail to notifications@example.com (which is used by "Mail and Packages" to login), then I would enter amazon@example.com there, correct?

[firstof9]

May I ask why this is needed? Can't we just filter based on Amazon's address (similar as it's done for the other providers)?

It is searched for that way, but not all providers leave the mail headers intact, Gmail does forward without mangling the headers so it looks like the email came directly from Amazon to the forwarded account.

If my Amazon account is registered to amazon@example.com. And in this account (which I own) I created a sieve script that forwards the mail to notifications@example.com (which is used by "Mail and Packages" to login), then I would enter amazon@example.com there, correct?

Yes use amazon@example.com, unless the email headers don't get mangled and they report the FROM address to be the proper amazon email addresses.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.