iCloud messages indexed but not displayed #5181
Comments
This one still puzzles me. Could you try with detailed logging as documented at https://github.com/nextcloud/mail/blob/master/doc/admin.md#logging? I'd like to see the IMAP communication where an UID isn't a number. |
|
Thank you for the reply Christoph. I was able to retrieve the nextcloud.log and the horde_imap.log but couldn't find the smtp one, Personal info replaced with USERNAME and LOCALIP. Thank you for looking. |
|
Sorry, wasn't thinking clearly. |
|
This is a problem with the Horde_Imap_Client_Ids::_toSequenceString method. The IDs array contains an empty string:
Line 317 In Horde_Imap_Client_Ids
|
|
Hi and thanks very much @miaulalala for having a look. |
|
Hi Minnietheandroid, it looks like it's just the first issue, but I haven't been able to load my mailbox contents now. So it seems this empty string from the iCloud IMAP is causing more than just this issue. I'll debug further and let you know what I find. |
|
Right, I'm so confused. I had the one and only message in my inbox displaying in Mail, but as soon as I went to open it, it disappeared as if by magic.... Also this:
What? |
|
@miaulalala I don't know how the code works but I'm up for testing or anything else that may help. |
|
So after spending some time debugging the mysteriously vanishing message, it turns out that we're receiving the same UIDs for changed and vanished messages from the iCloud IMAP server. At least my Inbox does, which leads to the message being deleted at the first partial sync. @Minnietheandroid how proficient are you with PHP? Would you be able to confirm this from your side? If you add this function $in = array_filter($in, static function($id){
return is_int($id);
});to the Horde_Imap_Client_Ids::_toSequenceString function like so:
you should be able to get your mailbox set up. The first sync ( If you set some break points in In case you miss the intial sync, you can set the sync tokens to I closed the frontend and triggered the sync via command line with the |
|
@miaulalala I've added the function to - The 'Error: A non-numeric value encountered etc'' has stopped in the admin loggging page, Moving on to setting sync tokens to null and I'm quite lost. I can't figure out how to trigger a new sync. |
|
@Minnietheandroid thanks so much for taking the time! If you want some help, join us in the public groupware dev channel! |
|
Thank you @miaulalala, that looks really interesting! I rechecked with 3 fresh Hanssen Nextcloud VM instances today, VM1. Installed and patched the /Horde/Imap/Client/Ids.php before adding the icloud account. VM2. Left unpatched. Just added the icloud account. VM3 Unpatched using a different (non icloud) imap account, no error and shows the messages. Not sure where that leaves us, at least some of the way there though. |
|
@Minnietheandroid thanks again for checking, I think this kind of confirms my theory on vanished and changed IDs! Not sure what the next steps are yet, but I'll keep you updated in any case. |
|
Just wondering if you all landed this one? At this point I’d pay for the right dev to help all of us out. Love this app, and want to fully use NextCloud. |
|
Hello, I, too, have this exact same issue, using Nextcloud 22.2.3 and configuring the email app with iCloud. When you are concerned with privacy it makes sense to host Nextcloud and use iCloud (Apple) as an email provider, specially since iCloud recently added support for custom domains. I strongly appreciate the high quality work being done by Nextcloud developpers and hope this issue will get to a resolution some time soon. iCloud is far from being a small email provider and it fits perfectly with Nextcloud values. Thank you again and keep going the excellent work. |
|
@NQuebec is your instance of Nextcloud sitting behind SWAG or a Reverse Proxy by chance? I'm continuing to self diagnose to see if its something within my setup. I noticed the basic data elements are coming through (email count, inbox quota), which is most likely transported over 443, but maybe the actual email comes through 993. Which is currently something I'm trying to configure correctly within SWAG (which is a reverse proxy for security). When I look at the logs, I can see Mail iterating through the list of emails, but without the actual guts of the email. I'm going to focus on configuring SWAG with 993 to see if that changes anything. My stack: No luck yet, but hopeful that the time off this holiday will let me dig in a bit further. SIMILAR THREAD: #3401 @ChristophWurst - in issue #3401 you had asked me to increase the IMAP timeouts, just forgot to follow-up. Here are the current settings.
|
|
@jkrack yes my instance is behind a reverse proxy (nginx). I have the exact same behavior as you describe: email count and folders name get refreshed but emails don't show. Your hypothesis is interesting, I'll try too to forward port 993 within my reverse proxy. My Nextcloud instance isn't installed through docker but native with PHP-FPM and proxied via nginx. One last thing: Few weeks ago I had downloaded and installed the bitnami image of Horde Groupware email to test it and got the exact same issue we are discussing here: folders and emails count were updated but emails weren't downloaded. I'll have to check if this setup involved a reverse proxy. |
|
While experimenting with other webmail clients (Roundcube, Afterlogic and Rainloop) I found that enabling emails threads on a specific email account with iCloud prevents the webmail client to get the messages, but not the folders and attribures. I didn't find such an option ("use email threads") in the config of Nextcloud mail app but if you find it and disable it then I am 99% confident this issue will be resolved. |
|
This is really great insight @NQuebec, I tipped you with the small amount of Brave credit that I had (more just testing out the feature, :) ) . @ChristophWurst any thoughts on how we could help dig deeper into this one? Super appreciative of all of the work on the Mail app. My only thought is dig into which app is using email threads and see if I could change the behavior. One question I do have for you though, do you know of anyone that has iCloud email setup successfully? |
|
@jkrack We've changed some stuff in the sync logic since then so I'm not sure how up to date this issue still is. If you have any insight, I would be super grateful if you are willing to share :) I recently tried setting up an iCloud account again and I'm running into a endless loop for the sync now. |
|
Hey @miaulalala, I'm using Mail app version 1.11.5, on Nextcloud Hub II (23.0.0) and still experiencing the issue. I've removed and added the icloud account again, but same behavior.
|
|
Thanks @jkrack for letting me know. I think I am on the way to at least knowing the reason for this behavour. The MIN:MAX search for the highest and lowest known UID return 0 for my iCloud account. If anyone here is capable of doing a bit of PHP debugging, I would be super grateful if you could set a breakpoint at mail/lib/IMAP/MessageMapper.php Lines 102 to 118 in 2287909 $metaResults, $min, $max and $total
|
|
I'm not saavy enough to debug, just dangerous enough to set a debug level and share logs. :) Appreciate you continuing to look into this one! |
|
I've taken a peek at this, today; and it looks like Apple's servers legitimately omit the MIN and MAX fields from search responses, even when they're explicitly requested. I've been having some luck just treating the servers like they don't support the |
|
It seems like Apple's implementation has some issues with their --- Socket.orig.php 2022-02-21 04:25:42.377899468 +0000
+++ Socket.php 2022-02-21 04:45:29.390136621 +0000
@@ -2359,39 +2359,39 @@
$esearch = false;
// Check for ESORT capability (RFC 5267)
- if ($this->_capability('ESORT')) {
- foreach ($options['results'] as $val) {
- if (isset($results_criteria[$val]) &&
- ($val != Horde_Imap_Client::SEARCH_RESULTS_SAVE)) {
- $results[] = $results_criteria[$val];
- }
- }
- $esearch = true;
- }
+ //if ($this->_capability('ESORT')) {
+ // foreach ($options['results'] as $val) {
+ // if (isset($results_criteria[$val]) &&
+ // ($val != Horde_Imap_Client::SEARCH_RESULTS_SAVE)) {
+ // $results[] = $results_criteria[$val];
+ // }
+ // }
+ // //$esearch = true;
+ //}
// Add PARTIAL limiting (RFC 5267 [4.4])
- if ((!$esearch || !empty($options['partial'])) &&
- $this->_capability('CONTEXT', 'SORT')) {
- /* RFC 5267 indicates RFC 4466 ESEARCH-like support,
- * notwithstanding "real" RFC 4731 support. */
- $esearch = true;
-
- if (!empty($options['partial'])) {
- /* Can't have both ALL and PARTIAL returns. */
- $results = array_diff($results, array('ALL'));
-
- $results[] = 'PARTIAL';
- $results[] = $options['partial'];
- $partial = true;
- }
- }
-
- if ($esearch && empty($this->_init['noesearch'])) {
- $cmd->add(array(
- 'RETURN',
- new Horde_Imap_Client_Data_Format_List($results)
- ));
- }
+ //if ((!$esearch || !empty($options['partial'])) &&
+ // $this->_capability('CONTEXT', 'SORT')) {
+ // /* RFC 5267 indicates RFC 4466 ESEARCH-like support,
+ // * notwithstanding "real" RFC 4731 support. */
+ // //$esearch = true;
+
+ // if (!empty($options['partial'])) {
+ // /* Can't have both ALL and PARTIAL returns. */
+ // $results = array_diff($results, array('ALL'));
+
+ // $results[] = 'PARTIAL';
+ // $results[] = $options['partial'];
+ // $partial = true;
+ // }
+ //}
+
+ //if ($esearch && empty($this->_init['noesearch'])) {
+ // $cmd->add(array(
+ // 'RETURN',
+ // new Horde_Imap_Client_Data_Format_List($results)
+ // ));
+ //}
$tmp = new Horde_Imap_Client_Data_Format_List();
foreach ($options['sort'] as $val) {
@@ -2417,40 +2417,40 @@
$results = array();
// Check if the server supports ESEARCH (RFC 4731).
- if ($this->_capability('ESEARCH')) {
- foreach ($options['results'] as $val) {
- if (isset($results_criteria[$val])) {
- $results[] = $results_criteria[$val];
- }
- }
- $esearch = true;
- }
-
- // Add PARTIAL limiting (RFC 5267 [4.4]).
- if ((!$esearch || !empty($options['partial'])) &&
- $this->_capability('CONTEXT', 'SEARCH')) {
- /* RFC 5267 indicates RFC 4466 ESEARCH-like support,
- * notwithstanding "real" RFC 4731 support. */
- $esearch = true;
-
- if (!empty($options['partial'])) {
- // Can't have both ALL and PARTIAL returns.
- $results = array_diff($results, array('ALL'));
-
- $results[] = 'PARTIAL';
- $results[] = $options['partial'];
- $partial = true;
- }
- }
-
- if ($esearch && empty($this->_init['noesearch'])) {
- // Always use ESEARCH if available because it returns results
- // in a more compact sequence-set list
- $cmd->add(array(
- 'RETURN',
- new Horde_Imap_Client_Data_Format_List($results)
- ));
- }
+ //if ($this->_capability('ESEARCH')) {
+ // foreach ($options['results'] as $val) {
+ // if (isset($results_criteria[$val])) {
+ // $results[] = $results_criteria[$val];
+ // }
+ // }
+ // $esearch = true;
+ //}
+
+ //// Add PARTIAL limiting (RFC 5267 [4.4]).
+ //if ((!$esearch || !empty($options['partial'])) &&
+ // $this->_capability('CONTEXT', 'SEARCH')) {
+ // /* RFC 5267 indicates RFC 4466 ESEARCH-like support,
+ // * notwithstanding "real" RFC 4731 support. */
+ // $esearch = true;
+
+ // if (!empty($options['partial'])) {
+ // // Can't have both ALL and PARTIAL returns.
+ // $results = array_diff($results, array('ALL'));
+
+ // $results[] = 'PARTIAL';
+ // $results[] = $options['partial'];
+ // $partial = true;
+ // }
+ //}
+
+ //if ($esearch && empty($this->_init['noesearch'])) {
+ // // Always use ESEARCH if available because it returns results
+ // // in a more compact sequence-set list
+ // $cmd->add(array(
+ // 'RETURN',
+ // new Horde_Imap_Client_Data_Format_List($results)
+ // ));
+ //}
/* Charset is optional for SEARCH (RFC 3501 [6.4.4]).
* If UTF-8 support is activated, a client MUST NOT
@@ -2974,11 +2974,11 @@
// Remove the last dot from the string.
$cmd = substr($cmd, 0, -1);
- if (!empty($val['decode']) &&
- $this->_capability('BINARY')) {
- $main_cmd = 'BINARY';
- $pipeline->data['binaryquery'][$key] = $val;
- }
+ //if (!empty($val['decode']) &&
+ // $this->_capability('BINARY')) {
+ // $main_cmd = 'BINARY';
+ // $pipeline->data['binaryquery'][$key] = $val;
+ //}
break;
case Horde_Imap_Client::FETCH_HEADERS:
@@ -3007,11 +3007,11 @@
break;
case Horde_Imap_Client::FETCH_BODYPARTSIZE:
- if ($this->_capability('BINARY')) {
- foreach ($c_val as $val) {
- $fetch->add('BINARY.SIZE[' . $val . ']');
- }
- }
+ //if ($this->_capability('BINARY')) {
+ // foreach ($c_val as $val) {
+ // $fetch->add('BINARY.SIZE[' . $val . ']');
+ // }
+ //}
break;
case Horde_Imap_Client::FETCH_ENVELOPE:If anyone knowledgeable about Horde wants to make this into a proper workaround, that'd be appreciated. |
|
Thanks a lot for the thorough debugging @ktemkin. We could check if there is an alternative approach to finding the min/max UIDs if icloud has issues with the existing search operation. |
|
Hi, I'm looking to use iCloud email with nextcloud on truenas. Has there been any updates to this issue? |
|
No, not yet |
|
Resetting the status because this got stale again. |
|
@ChristophWurst I do realize you and your team have other priorities and I would like to help fixing this issue. I am an experienced developer and an SRE so I can debug things and fix them. However, I lack useful knowledge about PHP and IMAP so I need some initial guidance. With Google forcibly switching their old Google Apps accounts to paid plans many people move their mailboxes to iCloud because it allows to attach a custom domain with no extra charge, if one's using a paid storage plan. With that, I am ready to invest my time and effort into fixing this issue or finding a useful workaround for it but I kindly ask for your or one of your teammates' assistance to at least start looking in a right place. @ktemkin - perhaps we can team up on this. |
This comment was marked as spam.
This comment was marked as spam.
Hi.
Trying out the app (it looks fantastic btw) and came across this error trying to connect an iCloud mailbox.
This or a similar issue came up a few days back and was closed for various reasons. #3401
Wasn't sure if I should continue it or not, but seeing as I'm on NC Snap, with Apache thought I'd do a fresh report.
Apologies if that was the wrong decision. Nextcloud error attached and any advice would be most appreciated.
[PHP] Error: Error: A non-numeric value encountered.txt
Expected behavior
Mailbox displays messages in right hand pane
Actual behavior
Mailbox reports correct number of messages/unread items in mailbox but doesn't display the messages or content
Mail app
Mail app version: 1.9.5
Mailserver or service: iCloud
Server configuration
Operating system: Ubuntu 20.04 - Nextcloud Snap
Web server: Apache 2.4
Database: MySQL 8
PHP version: 7.4
Nextcloud Version: Snap 21.01
Client configuration
Browser: Firefox 89.0, Brave 1.25.68
Operating system: Manjaro 21
The text was updated successfully, but these errors were encountered: