Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Data loss) Issue with symlink causes entire remote IMAP store to be purged/deleted. #426

Open
IAXES opened this issue Jan 22, 2017 · 9 comments
Assignees

Comments

@IAXES
Copy link

IAXES commented Jan 22, 2017

General informations

  • system/distribution (with version): Ubuntu 4.4.0-59-generic x86_64
  • offlineimap version (offlineimap -V): 6.6.1
  • Python version:2.7.12
  • server name or domain:
  • CLI options: none

Configuration file offlineimaprc

REMOVE PRIVATE DATA.

pythonfile (if any)

REMOVE PRIVATE DATA.

Logs, error

REMOVE PRIVATE DATA.

Steps to reproduce the error

  • Trivial rc file setup (https://github.com/connermcd/dotfiles/blob/master/.offlineimaprc.template).
  • Create a symlink to ~/Maildir.
  • Sync emails to local store via offlineimap (works).
  • Invalidate the symlink to ~/Maildir (ie: have it point to an empty folder).
  • Run offlineimap.
  • Result: offlineimap assumes the local store is empty, rather than invalid, and purges all e-mails from the remote store (not fun).
  • Suggestions: a validation/sanity check routine should be executed to prevent this.
  • Additional notes: a placeholder file could be put in place, somewhat similar to how Subversion creates .svn folders, or something more involved (ie: tracking DB) could be implemented. I realize that this is not a trivial change.
@nicolas33
Copy link
Member

Deleting the whole content of a folder should only happen while deleting the local folder. We already handle it right when the content of folder is cleared (resync) but not when the parent folder is likely deleted.

@nicolas33
Copy link
Member

I did not read the report right the first time. ~/.offlineimap is the metadata path. Do you pretend that clearing the metadata path makes offlineimap delete the remote emails?

@chris001
Copy link
Member

@dev0null In case you didn't know, you can edit your initial report and fix the mistake, just click on the "pencil" icon on the top right, near the "reaction" icon.

@chris001
Copy link
Member

chris001 commented Jan 22, 2017

This issue brings attention to the sync algorithm. Originally it was designed so the local is always the master, for deleted messages, in the sync process, with no chance to undo a real delete. When the user deletes the local copy of the email message, on the next sync it'll really delete the remote copy of the email message from the remote server also.

Should the local Maildir (or local IMAP) always be the absolute master in the sync?

What algorithm does apple use to sync iphone/ipad emails (equivalent to our local maildir or local imap) to icloud (equivalent to the remote mail server) with hardly any reports of total loss of all email?

Same for google android and gmail, what's their sync algorithm handle local/remote emails, when one side is missing a small or large number of messages? It doesn't instantly delete the missing message from the remote side permanently.

A simple way to handle this: don't really delete mail, just move it to a holding area/recycle bin/soft-delete folder tree, for X days, default 30 or 60, 90, 180, 365 days, after which time, the sync algorithm can be configured to purge/delete it permanently.

@IAXES
Copy link
Author

IAXES commented Jan 23, 2017

@chris001 Updated bug report. Purged now-outdated comments.

@nicolas33
Copy link
Member

Please no! Don't edit the comments afterwards except for minor fixes to avoid breaking the next comments.

AFAICT, we must only clear a folder when the local directory is still there and empty. We fail at this.

@nicolas33
Copy link
Member

nicolas33 commented Jan 23, 2017

Originally it was designed so the local is always the master,

No, there is no concept of master. "Sync" in offlineimap means propagate changes to the other side. On conflicts the result is "keep the data".

for deleted messages, in the sync process, with no chance to undo a real delete. When the user deletes the local copy of the email message, on the next sync it'll really delete the remote copy of the email message from the remote server also.

Yes.

What algorithm does apple use to sync iphone/ipad emails (equivalent to our local maildir or local imap) to icloud (equivalent to the remote mail server) with hardly any reports of total loss of all email?

I don't use above tools. I'd say they don't sync but use standard IMAP (perhaps with disconnected mode).

A simple way to handle this: don't really delete mail, just move it to a holding area/recycle bin/soft-delete folder tree, for X days, default 30 or 60, 90, 180, 365 days, after which time, the sync algorithm can be configured to purge/delete it permanently.

This could be a new (optional) feature. You might want to open a new issue about that.

nicolas33 added a commit that referenced this issue Jan 25, 2017
We currently propagate the deletions if:

- the content of a folder was deleted;
- the parent directory of the folder was deleted.

The second case in incorrect and unexpected. This requires further work.

Github-ref: #426
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
@nicolas33
Copy link
Member

This behaviour is now documented.

@mowny
Copy link

mowny commented Jan 25, 2017

@chris001 Clearing the messages on an iphone also clears them from the iCloud account. I was called to fix such a problem last year and luckily there was a notebook with pop3 so only the 3 weeks from the last pop3 download up to the deletion where gone. Was told there were several warnings, though, which he chose to ignore.

Suggestions:

  • require "--force" or something when there are more than N messages to delete
  • require some switch to delete anything at all, just add (to) folders otherwise

(Right now I only have jessie's 6.3.4-1 here so I can't check whether something like that is already implemented in 7.0.12)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants