"failed to move message" error after sending message

[wedens]

Expected or desired behavior

After sending message and receiving new mail I can continue browsing my mail without errors.

Actual behavior

After sending a message and doing mu4e-update-mail-and-index I get an error when trying to open any message:

error in process filter: mu4e-error-handler: Error 2: failed to move message: cannot read file /home/wedens/Maildir/xxxx/sent/cur/1588217564.0d954d3a7860b217.wedens-pc:2,S: No such file or directory (line was:'(move :docid nil :msgid "87tv111wdf.fsf@xxxx" :flags "+S-u-N" :maildir nil :rename nil :no-view t)')

And there is indeed no such file. But this file (same message, different flags?) /home/wedens/Maildir/xxxx/sent/cur/1588217564.0d954d3a7860b217.wedens-pc,U=277:2,S exists.

If I close mu4e, do mu index (reports 1 cleaned-up message) and re-open mu4e, the problem disappears.

This happens every time I send a message.

mu4e log (sanitized): mu4e.log

Steps to reproduce

• Compose a new message
• Send the message
• mu4e-update-mail-and-index
• Try to open any message and get the error

Versions of mu, mu4e/emacs, operating system etc.

mu/mu4e: 1.4.1
emacs: 27.0.90
GNU/Linux (NixOS)

Any other detail

I've updated mu/mu4e to 1.4.1 from 1.2.0 recently and re-initialized mu database (mu init, mu index). I didn't have such issue with 1.2.0.

My mu4e configuration includes these vars that might be relevant: (setq mu4e-index-cleanup nil), (mu4e-sent-messages-behavior . sent), (mu4e-change-filenames-when-moving . t).

mu4e-index-cleanup is set to nil because one of mailboxes is not always mounted and I don't want to re-index it every time I mount it.

[djcb]

What are you using to sync your messages? It seems that tool may be changing the message names.

[theophilusx]

Since upgrade from original 1.4.1 to 1.4.3, I'm also seeing occasional errors like this. It does seem to be some interaction between mu4e and mbsync (seems to happen only when automated updated runs in the background while either reading or composing a message - not sure yet, error is intermittent enough to be tricky to track down/reproduce).

…

-- Tim Cross

[wedens]

What are you using to sync your messages? It seems that tool may be changing the message names.

mbsync 1.3.1. The same tool I was successfully using before mu4e upgrade with the same mu4e configuration.

[fintelkai]

Same combination of mbsync 1.3.1 (not recently changed or updated; worked previously) and recently updated mu(4e) 1.4.3. Same intermittent errors. Two recent samples:

error in process filter: mu4e-error-handler: Error 2: failed to move message: cannot read /home/fintel/.local/share/mail/INBOX/cur/1588344124.2c30e8916c51ca3c.kaix1:2,S (line was:'(move :docid 30520 :msgid nil :flags "+T-N" :maildir "/Trash" :rename t :no-view nil)')

and

error in process filter: mu4e-error-handler: Error 5: could not create message: something went wrong (line was:'(move :docid 30523 :msgid nil :flags "-N" :maildir "/Archive" :rename t :no-view nil)')

[cacology]

I think I've made a little progress on this issue: it turns out that the server for mu4e is dropping the local Uid when renaming messages. It's easiest to see this with some unread messages. I.e.

Messages retrieved by mbsync

1588435213.670aca2005a700b6.Grumpys-MacBook.local,U=6336:2,
1588435214.25af345705bce094.Grumpys-MacBook.local,U=6337:2,R
1588435216.f1c199f505cef091.Grumpys-MacBook.local,U=6338:2,P

After viewing them in mu4e

1588435772.21bf050726fa28f5.Grumpys-MacBook.local:2,S
1588435773.7a150fca270a75b5.Grumpys-MacBook.local:2,PS
1588435773.b6d1e3f627024bae.Grumpys-MacBook.local:2,RS

This causes mbsync to assign a NEW higher UID on the next pass, which breaks the mu index, repeat.

Is there a user switch to tell mu to leave the U=#### portion of a filename intact when appending new flags?

[djcb]

mu4e only changes the base-part of the file-name when asked to so, with mu4e-change-filenames-when-moving.

[djcb]

Can people still reproduce this with the latest (ie. >= 1.4.4)?

[linwaytin]

Hi,

I just run into similar issues.
After upgrading to 1.4.4, when I want to delete a mail, I see the error.
I suppose that is related to the (setq mu4e-change-filenames-when-moving t).
After commenting out this line, I can delete mails normally.
(However then I had the UID issue that makes mbsync not work.)

[linwaytin]

I noticed the similar thing as @cacology .
Every mail in my Maildir has a filename like '1588696288.206535_54.WT-HP-x360,U=54:2,S'.
However, after moving a mail to trash in mu4e, I have a new file like 1588697486.1222683ddcc1b2da.WT-HP-x360:2,ST.
The U= part is gone.
I have mu4e-change-filenames-when-moving set to t.

If I keep the filename like the above, run mbsync, then I won't see the mail in the Trash in Gmail (but it still exists in the server, and I can find it through search), and it's gone in my Maildir.

This seems related to the OP's issue.

[djcb]

Yes, if you have mu4e-change-filenames-when-moving then mu4e will change the (base) filename as requested; otherwise it won't; (the whole reason for mu4e-change-filenames-when-moving is to help mbsync, apparently it gets confused otherwise).

Anyway, can any of this be reproduced without invoking mbsync? It seems there's not much more I can do from the mu/mu4e side.

[cacology]

@djcb the change seems to have fixed it for me. (thank you! And thank you for a GREAT piece of software.) I was intermittently getting the "failed to move message" error too, which hasn't come back in the last ten-ish hours yet, nor has the escalating UIDs from mbsync issue.

Just to document my setup so that people trying to debug theirs can see what's different, I have both mbsync and mu index running in the background by a script. I've disabled mu4e auto-updates and never really use the manual update (only when debugging something or if something goes wrong). I do have mu4e-change-filenames-when-moving set to t. Here's the script I'm using to get email--Periodic just gets inboxes, All gets everything.

#!/bin/bash

while :
do
    gtimeout --foreground 600 mbsync All
    gtimeout --foreground 600 mu index
    date
    sleep 300
    gtimeout --foreground 600 mbsync Periodic
    gtimeout --foreground 600 mu index --lazy-check
    date
    sleep 300
    gtimeout --foreground 600 mbsync Periodic
    gtimeout --foreground 600 mu index --lazy-check
    date
    sleep 300
    gtimeout --foreground 600 mbsync Periodic
    gtimeout --foreground 600 mu index --lazy-check
    date
    sleep 300
    gtimeout --foreground 600 mbsync Periodic
    gtimeout --foreground 600 mu index --lazy-check
    date
    sleep 300
done

[linwaytin]

Does mu4e-change-filenames-when-moving remove the UID in the filenames?
Sorry this is likely a dumb question, but I just have used mbsync and mu4e for a few days.
Also, should I config mbsync in order to work with mu4e?
Thank you.

[wedens]

Anyway, can any of this be reproduced without invoking mbsync? It seems there's not much more I can do from the mu/mu4e side.

But it was working fine in mu4e 1.2.0, so there is probably some change between 1.2.0 and 1.4.* that (probably indirectly) introduced the problem.

[djcb]

@wedens (and other mbsync users): if you can point out the exact difference in what happens, that would be useful, since I don't use mbsync, i.e. on the filesystem level:

1. I received a message with filesystem path x
2. mu opens it, now the message has filesystem path y. Can you still successfully open it in mu4e?
3. mbsync
4. now the message has filesystem path z. Can you still successfully open it in mu4e?

And you can try with mu4e-change-filenames-when-moving set to t or to nil, and with mu4e-index-cleanup t, and mu4e-index-lazy-check nil. Does it fail in all these cases, and what exactly do you see?

(also see #1670).

[podiki]

For what it's worth, I haven't run into these issues, using mbsync/isync, mu (always git version), and gmail. I update frequently and use mu4e a lot, but it's possible I missed some of the versions with the commits mentioned previously. mu4e does an update on a timer, and otherwise mbsync is run when new messages are detected (by goimapnotify). I have mu4e-change-filenames-when-movin' set to t, mu4e-index-cleanup is t, and mu4e-index-lazy-check is nil.

[jperras]

Adding in another data point: I experienced a similar issue to that described, but it was resolved with mu 1.4.6 (was on 1.4.3 previously).

Previously results from `mu find "maildir:/some/INBOX" would not actually correspond to what mu4e would display. This would happen after I refiled/trashed/etc. (basically any message moved), and would persist until I reindexed outside of mu4e.

[djcb]

Great, thanks for the testing. Closing this.

[kaz-yos]

I encountered a similar issue after sending every email with a setup with very frequent mbsync invocation via fswatch. Switching to mbsync's AltMap UID scheme (no U=* UID in file names) has mostly(?) solved the issue.

• Related draft duplication issue: Duplicate draft messages #1373
• On mbsync AltMap: https://sourceforge.net/p/isync/mailman/message/35435686/

EDIT Feb 2, 2021. For the occasional "failed to move message: cannot read" error messages, I started using the following automatic invocation of mu4e-update-index.

  ;; When this is encountered index cleanup to remove missing files from the index is necessary.
  (defun my-mu4e-error-handler (errcode errmsg)
    "Handler function for showing an error."
    ;; don't use mu4e-error here; it's running in the process filter context
    (cl-case errcode
      (4 (user-error "No matches for this search query."))
      (t (cond
          ((string-match "failed to move message: cannot read" errmsg)
           (let ((mu4e-index-cleanup t)
                 (mu4e-index-lazy-check t))
             (mu4e-update-index)
             (error "Error %d: %s" errcode errmsg)))
          (t (error "Error %d: %s" errcode errmsg))))))
  (advice-add #'mu4e-error-handler
              :override #'my-mu4e-error-handler)

[theophilusx]

Note also that extremely frequent mbsync calls can be problematic if you fail to ensure that a previous mbsync call has completed before the next one starts. This can occur if you have an external process performing the mbsync and don't ensure a previous call has ended before starting the next one. This can be particularly important if you have multiple clients using imap for the same account (e.g. desktop and mobile devices) as you can cause race conditions that can result in inconsistent behaviour. If you have multiple clients all checking too frequently, problems are likely. I think it is important to remember email is not instant messaging. Some delay in email delivery should be expected. While it is rarer these days due to the speeds and connectivity of networks, the mail routing protocol is designed to pass mail through multiple servers, sometimes with significant delay between 'hops' (these days, there are typically very few hops due to network connectivity). I have found most (all?) of the issues I have had with mbsync can be avoided if you ensure there are multiple minutes between mbsync calls (at least 5 min - I use 15 min). This seems particularly important with Microsoft's brand of imap. I have also found the setting (setq mu4e-change-filenames-when-moving t) can be important for preventing uuid issues. This is also how clients like mutt work by default.

…

-- Tim Cross