New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

offlineimap erroneously maps T => \Deleted #142

Closed
rnkn opened this Issue Dec 4, 2014 · 15 comments

Comments

Projects
None yet
6 participants
@rnkn

rnkn commented Dec 4, 2014

I'm using FastMail + offlineimap + mu4e.

The default FastMail Trash folder behaviour is to auto-purge messages in the folder after 60 days and immediately purge all messages with the \Deleted IMAP flag.

The mu4e "trash" behaviour is to move to the Trash folder and apply the Maildir T (trashed) flag. The "delete" behaviour is to permanently delete the file.

This pretty standard behaviour is obviously to allow a buffer of time between "trashing" an email and having it erased.

However, offlineimap is mapping T => \Deleted which causes emails to be immediately deleted. The T flag is not equivalent to the \Deleted IMAP flag (see http://cr.yp.to/proto/maildir.html).

Both FastMail staff and mu/mu4e creator @djcb (see https://groups.google.com/forum/#!topic/mu-discuss/m4ORymDlf0E) have confirmed that this is erroneous behaviour.

Can you please stop mapping T => \Deleted?

@spaetz

This comment has been minimized.

Show comment
Hide comment
@spaetz

spaetz Dec 5, 2014

Member

I just read the thread you linked above, and nobody seems to be saying that the Maildir "T" flag does not correspond to the IMAP \deleted flag. This behavior has been ingrained in offlineimap since the dawn of time. And the "trashed" Flag and the "\deleted" flag are very much equivalent from everything I have seen so far.

If Fastmail "auto-expunges" deleted/trashed emails then that is their servers policy, but why would you set the "trashed" flag in the first place? This is the first time, I have seen a complaint about this behavior .

Member

spaetz commented Dec 5, 2014

I just read the thread you linked above, and nobody seems to be saying that the Maildir "T" flag does not correspond to the IMAP \deleted flag. This behavior has been ingrained in offlineimap since the dawn of time. And the "trashed" Flag and the "\deleted" flag are very much equivalent from everything I have seen so far.

If Fastmail "auto-expunges" deleted/trashed emails then that is their servers policy, but why would you set the "trashed" flag in the first place? This is the first time, I have seen a complaint about this behavior .

@rnkn

This comment has been minimized.

Show comment
Hide comment
@rnkn

rnkn Dec 5, 2014

It was my take that @djcb was saying that T and \Deleted are not equivalent in this message: https://groups.google.com/d/msg/mu-discuss/m4ORymDlf0E/4HPCFmAl49MJ. I may have misunderstood. Anyway, from the info I've found:

Maildir T flag, from http://cr.yp.to/proto/maildir.html:

Flag "T" (trashed): the user has moved this message to the trash; the trash will be emptied by a later user action.

IMAP \Deleted flag, from https://tools.ietf.org/html/rfc3501#section-2.3.2:

\Deleted
Message is "deleted" for removal by later EXPUNGE

Is EXPUNGE a "user action"? This may be a semantic issue. I read the above as distinct: an EXPUNGE can be initiated by a user action, but every EXPUNGE is not a user action, which is probably most true when an email provider purges \Deleted messages automatically (as with FastMail and I think Gmail).

Also, it seems like a message flagged T ought to be in the Trash folder, whereas a message flagged \Deleted can be anywhere. (Maybe not relevant?)

In my own searching, I found some other confused souls...

Please let me know if you'd prefer using the mailing list.

rnkn commented Dec 5, 2014

It was my take that @djcb was saying that T and \Deleted are not equivalent in this message: https://groups.google.com/d/msg/mu-discuss/m4ORymDlf0E/4HPCFmAl49MJ. I may have misunderstood. Anyway, from the info I've found:

Maildir T flag, from http://cr.yp.to/proto/maildir.html:

Flag "T" (trashed): the user has moved this message to the trash; the trash will be emptied by a later user action.

IMAP \Deleted flag, from https://tools.ietf.org/html/rfc3501#section-2.3.2:

\Deleted
Message is "deleted" for removal by later EXPUNGE

Is EXPUNGE a "user action"? This may be a semantic issue. I read the above as distinct: an EXPUNGE can be initiated by a user action, but every EXPUNGE is not a user action, which is probably most true when an email provider purges \Deleted messages automatically (as with FastMail and I think Gmail).

Also, it seems like a message flagged T ought to be in the Trash folder, whereas a message flagged \Deleted can be anywhere. (Maybe not relevant?)

In my own searching, I found some other confused souls...

Please let me know if you'd prefer using the mailing list.

@chris001

This comment has been minimized.

Show comment
Hide comment
@chris001

chris001 Dec 5, 2014

Member

If I may add a comment on this. To add understanding to this issue.

From my understanding of the spec - and use of the mail client Mulberry - a very nice, compact binary client that respects the IMAP standards very well...

The Maildir flag "T" means "soft" - the user has requested the mailreader app to move message to Trash folder. Similar to the Mac OS desktop Trash can, or Windows Recycle Bin. The user might change their mind, and un-Trash the message. There is a generous margin of time for the user to un-Trash the message... days... months.. years... unlimited... anyone at any time with access to the desktop can manually "empty the trash". If not, maybe it depends on the service level agreement and/or system administrator need's to recover disk space.

The IMAP flag "\deleted" means "hard" Trash - user has deleted a message from inside the Trash folder. On Mulberry and similar mail clients, you can see a horizontal line thru the "\deleted" message - crossing it out - when the app's configuration option is set to show messages that are "pending Expunge". That message is headed for certain deletion. Either instantly, upon user action, (menu option - "Expunge mailbox now") or at a future scheduled garbage collection, during a cron job, nightly, weekly, or at the manual action of the email system admin.

So I agree, that Maildir "T" flag and the IMAP "\deleted" flag, correspond to different user intentions, have different meanings, and should be acted on differently by offlineimap.

Let "T" messages stay in the Trash folder. Only expunge the messages with the "\deleted" IMAP flag, with a configurable delay, such as, 30 days after the date the message was marked "\deleted".

Member

chris001 commented Dec 5, 2014

If I may add a comment on this. To add understanding to this issue.

From my understanding of the spec - and use of the mail client Mulberry - a very nice, compact binary client that respects the IMAP standards very well...

The Maildir flag "T" means "soft" - the user has requested the mailreader app to move message to Trash folder. Similar to the Mac OS desktop Trash can, or Windows Recycle Bin. The user might change their mind, and un-Trash the message. There is a generous margin of time for the user to un-Trash the message... days... months.. years... unlimited... anyone at any time with access to the desktop can manually "empty the trash". If not, maybe it depends on the service level agreement and/or system administrator need's to recover disk space.

The IMAP flag "\deleted" means "hard" Trash - user has deleted a message from inside the Trash folder. On Mulberry and similar mail clients, you can see a horizontal line thru the "\deleted" message - crossing it out - when the app's configuration option is set to show messages that are "pending Expunge". That message is headed for certain deletion. Either instantly, upon user action, (menu option - "Expunge mailbox now") or at a future scheduled garbage collection, during a cron job, nightly, weekly, or at the manual action of the email system admin.

So I agree, that Maildir "T" flag and the IMAP "\deleted" flag, correspond to different user intentions, have different meanings, and should be acted on differently by offlineimap.

Let "T" messages stay in the Trash folder. Only expunge the messages with the "\deleted" IMAP flag, with a configurable delay, such as, 30 days after the date the message was marked "\deleted".

@rnkn

This comment has been minimized.

Show comment
Hide comment
@rnkn

rnkn Dec 6, 2014

@chris001 put it beautifully.

If I may digress in an unwarranted literary analysis... The use of metaphor in computational language helps the layperson understand very quickly what to expect from certain processes, e.g. in this case, the recipient of some physical mail may decide to place the mail in an archive, or throw it in the trash, or maybe take the drastic step of burning it the fireplace. However, if, after having thrown the mail in the trash, the recipient then changes his/her mind, the piece of mail still exists in the trash can (and can be retrieved if one doesn't mind fishing through last night's dinner scraps). By contrast, the mail thrown into the fireplace has since become ash.

Thus, to treat "trash" and "delete" as equivalent, i.e. by erasing email that is sent to the Trash folder, we break this metaphor and change the trash can into a magical vortex in which mail disappears into the ether ;)

rnkn commented Dec 6, 2014

@chris001 put it beautifully.

If I may digress in an unwarranted literary analysis... The use of metaphor in computational language helps the layperson understand very quickly what to expect from certain processes, e.g. in this case, the recipient of some physical mail may decide to place the mail in an archive, or throw it in the trash, or maybe take the drastic step of burning it the fireplace. However, if, after having thrown the mail in the trash, the recipient then changes his/her mind, the piece of mail still exists in the trash can (and can be retrieved if one doesn't mind fishing through last night's dinner scraps). By contrast, the mail thrown into the fireplace has since become ash.

Thus, to treat "trash" and "delete" as equivalent, i.e. by erasing email that is sent to the Trash folder, we break this metaphor and change the trash can into a magical vortex in which mail disappears into the ether ;)

@xmaillard

This comment has been minimized.

Show comment
Hide comment
@xmaillard

xmaillard Dec 6, 2014

Paul Rankin writes:

Thus, to treat "trash" and "delete" as equivalent, i.e. by erasing
email that is sent to the Trash folder, we break this metaphor and
change the trash can into a magical vortex in which mail disappears
into the ether ;)

Beautifully said !

Sent with my mu4e

xmaillard commented Dec 6, 2014

Paul Rankin writes:

Thus, to treat "trash" and "delete" as equivalent, i.e. by erasing
email that is sent to the Trash folder, we break this metaphor and
change the trash can into a magical vortex in which mail disappears
into the ether ;)

Beautifully said !

Sent with my mu4e

@spaetz

This comment has been minimized.

Show comment
Hide comment
@spaetz

spaetz Dec 9, 2014

Member

The thing is, that fastmail seems to have decided to burn your mails. When I mark my mails as deleted, they get the "\delete" flag on the IMAP server, which is my equivalent of putting them in an archive (or not having them shown by default anymore). Your use case seem to simply move mails to the "trash" folder, why would you use the "T" flag for that. Simply map your "d" key to "move to Trash", or is that not what you ultimately want?

Anyway, I am not making a call here of what will or will not be implemented, just saying that this behavior has been part of OfflineImap since (as far as I can tell) 2001 without significant complaints. If that were completely faulty behavior some of our IMAP experts would have had complained long ago :-).

Member

spaetz commented Dec 9, 2014

The thing is, that fastmail seems to have decided to burn your mails. When I mark my mails as deleted, they get the "\delete" flag on the IMAP server, which is my equivalent of putting them in an archive (or not having them shown by default anymore). Your use case seem to simply move mails to the "trash" folder, why would you use the "T" flag for that. Simply map your "d" key to "move to Trash", or is that not what you ultimately want?

Anyway, I am not making a call here of what will or will not be implemented, just saying that this behavior has been part of OfflineImap since (as far as I can tell) 2001 without significant complaints. If that were completely faulty behavior some of our IMAP experts would have had complained long ago :-).

@rnkn

This comment has been minimized.

Show comment
Hide comment
@rnkn

rnkn Dec 9, 2014

When I mark my mails as deleted, they get the "\delete" flag on the IMAP server, which is my equivalent of putting them in an archive

This seems ideal, however this treatment of the \Deleted flag is not how many common email providers seem to operate. Here's FastMail's rationale behind their expunging practice https://www.fastmail.com/help/clients/deleteissue.html

Whether that's the right or wrong way to do things, that's the way they operate, so I fear the issue will keep cropping up.

Your use case seem to simply move mails to the "trash" folder, why would you use the "T" flag for that.

I agree. Applying the "T" flag is superfluous. However it is the default behaviour of mu4e, and the author appears to believe this way is also the right way to do things.

Simply map your "d" key to "move to Trash"

This is what I've done, and it works well.

this behavior has been part of OfflineImap since (as far as I can tell) 2001 without significant complaints.

I don't think that's true. The threads linked above show people experiencing the same frustration.

Given that there seem to be two opposed schools of thought on what the right way is, maybe allowing for both is in everyone's best interests?

rnkn commented Dec 9, 2014

When I mark my mails as deleted, they get the "\delete" flag on the IMAP server, which is my equivalent of putting them in an archive

This seems ideal, however this treatment of the \Deleted flag is not how many common email providers seem to operate. Here's FastMail's rationale behind their expunging practice https://www.fastmail.com/help/clients/deleteissue.html

Whether that's the right or wrong way to do things, that's the way they operate, so I fear the issue will keep cropping up.

Your use case seem to simply move mails to the "trash" folder, why would you use the "T" flag for that.

I agree. Applying the "T" flag is superfluous. However it is the default behaviour of mu4e, and the author appears to believe this way is also the right way to do things.

Simply map your "d" key to "move to Trash"

This is what I've done, and it works well.

this behavior has been part of OfflineImap since (as far as I can tell) 2001 without significant complaints.

I don't think that's true. The threads linked above show people experiencing the same frustration.

Given that there seem to be two opposed schools of thought on what the right way is, maybe allowing for both is in everyone's best interests?

@nicolas33

This comment has been minimized.

Show comment
Hide comment
@nicolas33

nicolas33 Jan 9, 2015

Member

Given that there seem to be two opposed schools of thought on what the right way is, maybe allowing for both is in everyone's best interests?

Don't know but since you could fix your issue, I'm closing.

Member

nicolas33 commented Jan 9, 2015

Given that there seem to be two opposed schools of thought on what the right way is, maybe allowing for both is in everyone's best interests?

Don't know but since you could fix your issue, I'm closing.

@nicolas33 nicolas33 closed this Jan 9, 2015

@xmaillard

This comment has been minimized.

Show comment
Hide comment
@xmaillard

xmaillard Jan 10, 2015

I do not call that a "fix". Don't you think you should take the time to analyze whether it is a good proposition or not ? We are a few expecting such modification.

If you are not willing to do it, could you at least explain where we could try by ourself in the code ?

Regards

xmaillard commented Jan 10, 2015

I do not call that a "fix". Don't you think you should take the time to analyze whether it is a good proposition or not ? We are a few expecting such modification.

If you are not willing to do it, could you at least explain where we could try by ourself in the code ?

Regards

@nicolas33

This comment has been minimized.

Show comment
Hide comment
@nicolas33

nicolas33 Jan 10, 2015

Member

Well, the "workaround" of re-assigning the "d" key binding can fix the operations to what you expect.

Other than that, I'm closing the issue mostly because I don't want do debate of what means a non-standard flag (then not well-defined by definition according to software programming).

Though and to make it clear, I'm willing to merge a patch to change the way OfflineIMAP works. Be care that such a change requires an option to not break current users expectations.

I have to say that while I'm not going to do such job myself, I'm fine to help anybody with the OfflineIMAP source code, git and whatever can help to do the task, like I always did. BUT I'm not going to do your home work. If someone come with a specific question I would be very happy to answer.

Consider starting with the online development documentation.

Last but not least, I expect such development related work to happen in the mailing list. Mails are much faster and easier to work with for such task.

Regards

Member

nicolas33 commented Jan 10, 2015

Well, the "workaround" of re-assigning the "d" key binding can fix the operations to what you expect.

Other than that, I'm closing the issue mostly because I don't want do debate of what means a non-standard flag (then not well-defined by definition according to software programming).

Though and to make it clear, I'm willing to merge a patch to change the way OfflineIMAP works. Be care that such a change requires an option to not break current users expectations.

I have to say that while I'm not going to do such job myself, I'm fine to help anybody with the OfflineIMAP source code, git and whatever can help to do the task, like I always did. BUT I'm not going to do your home work. If someone come with a specific question I would be very happy to answer.

Consider starting with the online development documentation.

Last but not least, I expect such development related work to happen in the mailing list. Mails are much faster and easier to work with for such task.

Regards

@spaetz

This comment has been minimized.

Show comment
Hide comment
@spaetz

spaetz Jan 26, 2015

Member

Hi, I checked out the links to the threads you provided:

  • issue #107 wants "deleted" mails to correspond to a "\Trash" label, something which is absolutely non-standard. I have no clue why Gmail decides to remove messages from folders that have a deleted flag, but given their weird way of exposing tags via IMAP folders all bets are off for Gmail anyway.
  • as for https://groups.google.com/forum/#!topic/mu-discuss/z4w_Gy1w6vg
    The only one who voices any kind of concern are the two of you who are also very vocal here. It is kind of an exaggeration to use this thread as a proof that it has also confused lots of other people.
  • as for: https://groups.google.com/forum/#!topic/mu-discuss/y6T-MpWKXh4
    This is an issue that the users local IMAP server web interface has been configured to not display any message with a Trash flag, according to the user, I don't see any Maildir involved at all (the user uses IMAP<->IMAP)

OfflineImap has been doing this since 2001, and not once has this behaviour been questioned before. And many advanced IMAP experts have been involved in its development (not counting me). So excuse me if I say, that your interpretation of the flags might not be universally be shared. You solved your issue by simply "MOVING" mails to the Trash folder, which is exactly what you want to do.

As for the literary analysis digresion (I love digressions), I disagree with the metaphors being used. "Moving a mail to the Trash folder, is throwing it in the attic (or bin if you want)", it is recoverable if you want. Putting on a "T" flag means, it is outside your house in the container. The IMAP "deleted" flag is nothing different. Actually, my client shows deleted messages on the server just fine, just crossed through and ready for undeletion. Calling that throwing in the fireplace is not the correct metaphor.

Altogether, I am happy that you found the solution of actually just moving a mail to the trash folder via a key binding, this is what is the right thing to do. Servers auto-expunging deleted mail are a tad overzealous.

All this will probably not make you happy, but there is, in my opinion no bug to fix in this case.

Member

spaetz commented Jan 26, 2015

Hi, I checked out the links to the threads you provided:

  • issue #107 wants "deleted" mails to correspond to a "\Trash" label, something which is absolutely non-standard. I have no clue why Gmail decides to remove messages from folders that have a deleted flag, but given their weird way of exposing tags via IMAP folders all bets are off for Gmail anyway.
  • as for https://groups.google.com/forum/#!topic/mu-discuss/z4w_Gy1w6vg
    The only one who voices any kind of concern are the two of you who are also very vocal here. It is kind of an exaggeration to use this thread as a proof that it has also confused lots of other people.
  • as for: https://groups.google.com/forum/#!topic/mu-discuss/y6T-MpWKXh4
    This is an issue that the users local IMAP server web interface has been configured to not display any message with a Trash flag, according to the user, I don't see any Maildir involved at all (the user uses IMAP<->IMAP)

OfflineImap has been doing this since 2001, and not once has this behaviour been questioned before. And many advanced IMAP experts have been involved in its development (not counting me). So excuse me if I say, that your interpretation of the flags might not be universally be shared. You solved your issue by simply "MOVING" mails to the Trash folder, which is exactly what you want to do.

As for the literary analysis digresion (I love digressions), I disagree with the metaphors being used. "Moving a mail to the Trash folder, is throwing it in the attic (or bin if you want)", it is recoverable if you want. Putting on a "T" flag means, it is outside your house in the container. The IMAP "deleted" flag is nothing different. Actually, my client shows deleted messages on the server just fine, just crossed through and ready for undeletion. Calling that throwing in the fireplace is not the correct metaphor.

Altogether, I am happy that you found the solution of actually just moving a mail to the trash folder via a key binding, this is what is the right thing to do. Servers auto-expunging deleted mail are a tad overzealous.

All this will probably not make you happy, but there is, in my opinion no bug to fix in this case.

@jgoerzen

This comment has been minimized.

Show comment
Hide comment
@jgoerzen

jgoerzen Oct 13, 2017

Contributor

OfflineIMAP is correct here. The bug is in mu4e and I have submitted it there: djcb/mu#1136

Contributor

jgoerzen commented Oct 13, 2017

OfflineIMAP is correct here. The bug is in mu4e and I have submitted it there: djcb/mu#1136

@nicolas33

This comment has been minimized.

Show comment
Hide comment
@nicolas33

nicolas33 Oct 13, 2017

Member

Hi John!

How are you? Thanks for your hint and report.

Member

nicolas33 commented Oct 13, 2017

Hi John!

How are you? Thanks for your hint and report.

@jgoerzen

This comment has been minimized.

Show comment
Hide comment
@jgoerzen

jgoerzen Oct 13, 2017

Contributor

Hi Nicolas! Good to be chatting with you again (even in a Github issue box ) I'm well. OfflineIMAP is slowly starting to creep back into my life, due partly to its support for GMail OAuth2 and partly to the increasing quality of Emacs mail clients and (IMHO) decreasing quality of Thunderbird. It is pretty amazing to see how just about every Emacs mail client has instructions for setup with OfflineIMAP.

Thanks for all your work maintaining and enhancing it! Really nice to see what it's become.

Contributor

jgoerzen commented Oct 13, 2017

Hi Nicolas! Good to be chatting with you again (even in a Github issue box ) I'm well. OfflineIMAP is slowly starting to creep back into my life, due partly to its support for GMail OAuth2 and partly to the increasing quality of Emacs mail clients and (IMHO) decreasing quality of Thunderbird. It is pretty amazing to see how just about every Emacs mail client has instructions for setup with OfflineIMAP.

Thanks for all your work maintaining and enhancing it! Really nice to see what it's become.

@nicolas33

This comment has been minimized.

Show comment
Hide comment
@nicolas33

nicolas33 Oct 13, 2017

Member

Thank you. Happy to see you here again. I agree contributors are doing a wonderful job.

Member

nicolas33 commented Oct 13, 2017

Thank you. Happy to see you here again. I agree contributors are doing a wonderful job.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment