offlineimap erroneously maps T => \Deleted

[rnkn]

I'm using FastMail + offlineimap + mu4e.

The default FastMail Trash folder behaviour is to auto-purge messages in the folder after 60 days and immediately purge all messages with the \Deleted IMAP flag.

The mu4e "trash" behaviour is to move to the Trash folder and apply the Maildir T (trashed) flag. The "delete" behaviour is to permanently delete the file.

This pretty standard behaviour is obviously to allow a buffer of time between "trashing" an email and having it erased.

However, offlineimap is mapping T => \Deleted which causes emails to be immediately deleted. The T flag is not equivalent to the \Deleted IMAP flag (see http://cr.yp.to/proto/maildir.html).

Both FastMail staff and mu/mu4e creator @djcb (see https://groups.google.com/forum/#!topic/mu-discuss/m4ORymDlf0E) have confirmed that this is erroneous behaviour.

Can you please stop mapping T => \Deleted?

[spaetz]

I just read the thread you linked above, and nobody seems to be saying that the Maildir "T" flag does not correspond to the IMAP \deleted flag. This behavior has been ingrained in offlineimap since the dawn of time. And the "trashed" Flag and the "\deleted" flag are very much equivalent from everything I have seen so far.

If Fastmail "auto-expunges" deleted/trashed emails then that is their servers policy, but why would you set the "trashed" flag in the first place? This is the first time, I have seen a complaint about this behavior .

[rnkn]

It was my take that @djcb was saying that T and \Deleted are not equivalent in this message: https://groups.google.com/d/msg/mu-discuss/m4ORymDlf0E/4HPCFmAl49MJ. I may have misunderstood. Anyway, from the info I've found:

Maildir T flag, from http://cr.yp.to/proto/maildir.html:

Flag "T" (trashed): the user has moved this message to the trash; the trash will be emptied by a later user action.

IMAP \Deleted flag, from https://tools.ietf.org/html/rfc3501#section-2.3.2:

\Deleted
Message is "deleted" for removal by later EXPUNGE

Is EXPUNGE a "user action"? This may be a semantic issue. I read the above as distinct: an EXPUNGE can be initiated by a user action, but every EXPUNGE is not a user action, which is probably most true when an email provider purges \Deleted messages automatically (as with FastMail and I think Gmail).

Also, it seems like a message flagged T ought to be in the Trash folder, whereas a message flagged \Deleted can be anywhere. (Maybe not relevant?)

In my own searching, I found some other confused souls...

• Criteria for deleting messages? #107
• https://groups.google.com/forum/#!topic/mu-discuss/z4w_Gy1w6vg
• https://groups.google.com/forum/#!topic/mu-discuss/y6T-MpWKXh4

Please let me know if you'd prefer using the mailing list.

[chris001]

If I may add a comment on this. To add understanding to this issue.

From my understanding of the spec - and use of the mail client Mulberry - a very nice, compact binary client that respects the IMAP standards very well...

The Maildir flag "T" means "soft" - the user has requested the mailreader app to move message to Trash folder. Similar to the Mac OS desktop Trash can, or Windows Recycle Bin. The user might change their mind, and un-Trash the message. There is a generous margin of time for the user to un-Trash the message... days... months.. years... unlimited... anyone at any time with access to the desktop can manually "empty the trash". If not, maybe it depends on the service level agreement and/or system administrator need's to recover disk space.

The IMAP flag "\deleted" means "hard" Trash - user has deleted a message from inside the Trash folder. On Mulberry and similar mail clients, you can see a horizontal line thru the "\deleted" message - crossing it out - when the app's configuration option is set to show messages that are "pending Expunge". That message is headed for certain deletion. Either instantly, upon user action, (menu option - "Expunge mailbox now") or at a future scheduled garbage collection, during a cron job, nightly, weekly, or at the manual action of the email system admin.

So I agree, that Maildir "T" flag and the IMAP "\deleted" flag, correspond to different user intentions, have different meanings, and should be acted on differently by offlineimap.

Let "T" messages stay in the Trash folder. Only expunge the messages with the "\deleted" IMAP flag, with a configurable delay, such as, 30 days after the date the message was marked "\deleted".

[rnkn]

@chris001 put it beautifully.

If I may digress in an unwarranted literary analysis... The use of metaphor in computational language helps the layperson understand very quickly what to expect from certain processes, e.g. in this case, the recipient of some physical mail may decide to place the mail in an archive, or throw it in the trash, or maybe take the drastic step of burning it the fireplace. However, if, after having thrown the mail in the trash, the recipient then changes his/her mind, the piece of mail still exists in the trash can (and can be retrieved if one doesn't mind fishing through last night's dinner scraps). By contrast, the mail thrown into the fireplace has since become ash.

Thus, to treat "trash" and "delete" as equivalent, i.e. by erasing email that is sent to the Trash folder, we break this metaphor and change the trash can into a magical vortex in which mail disappears into the ether ;)

[xmaillard]

Paul Rankin writes:

Thus, to treat "trash" and "delete" as equivalent, i.e. by erasing
email that is sent to the Trash folder, we break this metaphor and
change the trash can into a magical vortex in which mail disappears
into the ether ;)

Beautifully said !

Sent with my mu4e

[spaetz]

The thing is, that fastmail seems to have decided to burn your mails. When I mark my mails as deleted, they get the "\delete" flag on the IMAP server, which is my equivalent of putting them in an archive (or not having them shown by default anymore). Your use case seem to simply move mails to the "trash" folder, why would you use the "T" flag for that. Simply map your "d" key to "move to Trash", or is that not what you ultimately want?

Anyway, I am not making a call here of what will or will not be implemented, just saying that this behavior has been part of OfflineImap since (as far as I can tell) 2001 without significant complaints. If that were completely faulty behavior some of our IMAP experts would have had complained long ago :-).

[rnkn]

When I mark my mails as deleted, they get the "\delete" flag on the IMAP server, which is my equivalent of putting them in an archive

This seems ideal, however this treatment of the \Deleted flag is not how many common email providers seem to operate. Here's FastMail's rationale behind their expunging practice https://www.fastmail.com/help/clients/deleteissue.html

Whether that's the right or wrong way to do things, that's the way they operate, so I fear the issue will keep cropping up.

Your use case seem to simply move mails to the "trash" folder, why would you use the "T" flag for that.

I agree. Applying the "T" flag is superfluous. However it is the default behaviour of mu4e, and the author appears to believe this way is also the right way to do things.

Simply map your "d" key to "move to Trash"

This is what I've done, and it works well.

this behavior has been part of OfflineImap since (as far as I can tell) 2001 without significant complaints.

I don't think that's true. The threads linked above show people experiencing the same frustration.

Given that there seem to be two opposed schools of thought on what the right way is, maybe allowing for both is in everyone's best interests?

[nicolas33]

Given that there seem to be two opposed schools of thought on what the right way is, maybe allowing for both is in everyone's best interests?

Don't know but since you could fix your issue, I'm closing.

[xmaillard]

I do not call that a "fix". Don't you think you should take the time to analyze whether it is a good proposition or not ? We are a few expecting such modification.

If you are not willing to do it, could you at least explain where we could try by ourself in the code ?

Regards

[nicolas33]

Well, the "workaround" of re-assigning the "d" key binding can fix the operations to what you expect.

Other than that, I'm closing the issue mostly because I don't want do debate of what means a non-standard flag (then not well-defined by definition according to software programming).

Though and to make it clear, I'm willing to merge a patch to change the way OfflineIMAP works. Be care that such a change requires an option to not break current users expectations.

I have to say that while I'm not going to do such job myself, I'm fine to help anybody with the OfflineIMAP source code, git and whatever can help to do the task, like I always did. BUT I'm not going to do your home work. If someone come with a specific question I would be very happy to answer.

Consider starting with the online development documentation.

Last but not least, I expect such development related work to happen in the mailing list. Mails are much faster and easier to work with for such task.

Regards

[spaetz]

Hi, I checked out the links to the threads you provided:

• issue Criteria for deleting messages? #107 wants "deleted" mails to correspond to a "\Trash" label, something which is absolutely non-standard. I have no clue why Gmail decides to remove messages from folders that have a deleted flag, but given their weird way of exposing tags via IMAP folders all bets are off for Gmail anyway.
• as for https://groups.google.com/forum/#!topic/mu-discuss/z4w_Gy1w6vg
  The only one who voices any kind of concern are the two of you who are also very vocal here. It is kind of an exaggeration to use this thread as a proof that it has also confused lots of other people.
• as for: https://groups.google.com/forum/#!topic/mu-discuss/y6T-MpWKXh4
  This is an issue that the users local IMAP server web interface has been configured to not display any message with a Trash flag, according to the user, I don't see any Maildir involved at all (the user uses IMAP<->IMAP)

OfflineImap has been doing this since 2001, and not once has this behaviour been questioned before. And many advanced IMAP experts have been involved in its development (not counting me). So excuse me if I say, that your interpretation of the flags might not be universally be shared. You solved your issue by simply "MOVING" mails to the Trash folder, which is exactly what you want to do.

As for the literary analysis digresion (I love digressions), I disagree with the metaphors being used. "Moving a mail to the Trash folder, is throwing it in the attic (or bin if you want)", it is recoverable if you want. Putting on a "T" flag means, it is outside your house in the container. The IMAP "deleted" flag is nothing different. Actually, my client shows deleted messages on the server just fine, just crossed through and ready for undeletion. Calling that throwing in the fireplace is not the correct metaphor.

Altogether, I am happy that you found the solution of actually just moving a mail to the trash folder via a key binding, this is what is the right thing to do. Servers auto-expunging deleted mail are a tad overzealous.

All this will probably not make you happy, but there is, in my opinion no bug to fix in this case.

[jgoerzen]

OfflineIMAP is correct here. The bug is in mu4e and I have submitted it there: djcb/mu#1136

[nicolas33]

Hi John!

How are you? Thanks for your hint and report.