Can't copy external mail older than maxage to local maildir (related to 681e27) #581

ianbrody · 2018-11-01T22:48:24Z

General information

system/distribution (with version): Arch linux
offlineimap version (offlineimap -V): 7.2.1
Python version: 3.7.1
server name or domain: gmail
CLI options: -o

Configuration file offlineimaprc

[general]
accounts = test
fsync = False

[Account test]
localrepository = Local-test
remoterepository = Remote-test
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
maxage=5

[Repository Remote-test]
type = IMAP
maxconnections = 3
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
remoteuser = <...>
remotehost = imap.gmail.com
remoteport = 993
ssl = yes
ssl_version = tls1_2
trashfolder = '[Gmail].Trash'
remotepass = <...>
folderfilter = lambda folder: folder in ['INBOX']

[Repository Local-test]
type = Maildir
localfolders = <...>
status_backend = plain
trashfolder = '[Gmail].Trash'

Steps to reproduce the error

Copy an email that's older than maxage in another account, to the local maildir tracked by offlineimap
Run offlineimap once; this copies the new email to the remote account
Run offlineimap again; this deletes the local copy (which I don't want) and keeps the remote copy.

Desired behavior: offlineimap shouldn't be looking at this email at all, because it's older than maxage.

If I downgrade to 7.1.4, this behavior still happens, and if I revert commit 681e27 there, the problem goes away (nothing is copied, and nothing is deleted). I believe this commit is at fault in the more recent versions, but it seems something else has changed around it, and I can't revert cleanly.

Log messages

Running with -d imap,maildir,thread I get

Folder INBOX [acc: test]:
 [thread]: Register new thread 'Folder INBOX [acc: test]' (account 'test')
<...>
Account sync test:
Folder INBOX [acc: test]:
 Syncing INBOX: IMAP -> Maildir
 Loading message list for IMAP[INBOX]
 [imap]: calling imaplib2 fetch command: '5:9,11:12' (FLAGS UID INTERNALDATE)
 [imap]: __options2hash returning: {'INTERNALDATE': '"01-Nov-2018 21:56:23 +0000"', 'FLAGS': '(Old)', 'UID': '6500'}
 [imap]: __options2hash returning: {'INTERNALDATE': '"01-Nov-2018 21:56:26 +0000"', 'FLAGS': '(Old)', 'UID': '6501'}
 [imap]: __options2hash returning: {'INTERNALDATE': '"01-Nov-2018 21:56:29 +0000"', 'FLAGS': '(\\Seen)', 'UID': '6502'}
 [imap]: __options2hash returning: {'INTERNALDATE': '"01-Nov-2018 21:56:30 +0000"', 'FLAGS': '(Old)', 'UID': '6503'}
 [imap]: __options2hash returning: {'INTERNALDATE': '"01-Nov-2018 22:03:50 +0000"', 'FLAGS': '(\\Seen)', 'UID': '6505'}
 [imap]: __options2hash returning: {'INTERNALDATE': '"01-Nov-2018 21:57:36 +0000"', 'FLAGS': '(Old)', 'UID': '6507'}
 [imap]: __options2hash returning: {'INTERNALDATE': '"01-Nov-2018 20:13:30 +0000"', 'FLAGS': '(\\Seen)', 'UID': '6508'}
 Message list for IMAP[INBOX] loaded: 7 messages
 Loading message list for Maildir[INBOX]
 Message list for Maildir[INBOX] loaded: 8 messages
 Syncing messages IMAP[INBOX] -> Maildir[INBOX]
 Syncing messages Maildir[INBOX] -> IMAP[INBOX]
 Copy message UID -1 (1/1) Local-test:INBOX -> Remote-test:INBOX
 [imap]: Write mail 'INBOX:-1' with flags set(['S'])
 []: deletemessageheaders: called to delete []
 []: getmessageheader: called to get message-id
 []: getmessageheader: eoh = 5051
 []: getmessageheader: headers = '<...>'
 [imap]: savemessage: date: "21-Aug-2010 00:04:35 -0500", content: <...>

on the first run, and

Folder INBOX [acc: test]:
 Syncing INBOX: IMAP -> Maildir
 Loading message list for IMAP[INBOX]
 [imap]: calling imaplib2 fetch command: '5:9,11:12' (FLAGS UID INTERNALDATE)
 [imap]: __options2hash returning: {'INTERNALDATE': '"01-Nov-2018 21:56:23 +0000"', 'FLAGS': '(Old)', 'UID': '6500'}
 [imap]: __options2hash returning: {'INTERNALDATE': '"01-Nov-2018 21:56:26 +0000"', 'FLAGS': '(Old)', 'UID': '6501'}
 [imap]: __options2hash returning: {'INTERNALDATE': '"01-Nov-2018 21:56:29 +0000"', 'FLAGS': '(\\Seen)', 'UID': '6502'}
 [imap]: __options2hash returning: {'INTERNALDATE': '"01-Nov-2018 21:56:30 +0000"', 'FLAGS': '(Old)', 'UID': '6503'}
 [imap]: __options2hash returning: {'INTERNALDATE': '"01-Nov-2018 22:03:50 +0000"', 'FLAGS': '(\\Seen)', 'UID': '6505'}
 [imap]: __options2hash returning: {'INTERNALDATE': '"01-Nov-2018 21:57:36 +0000"', 'FLAGS': '(Old)', 'UID': '6507'}
 [imap]: __options2hash returning: {'INTERNALDATE': '"01-Nov-2018 20:13:30 +0000"', 'FLAGS': '(\\Seen)', 'UID': '6508'}
 Message list for IMAP[INBOX] loaded: 7 messages
 Loading message list for Maildir[INBOX]
 Message list for Maildir[INBOX] loaded: 8 messages
 Syncing messages IMAP[INBOX] -> Maildir[INBOX]
 Deleting 1 messages (6515) in Maildir[INBOX]
 Syncing messages Maildir[INBOX] -> IMAP[INBOX]
[thread]: Unregister thread 'Folder INBOX [acc: test]'

on the second run. Note that 21-Aug-2010 00:04:35 -0500 is the date of the message copied.

The text was updated successfully, but these errors were encountered:

nicolas33 · 2018-12-01T23:40:16Z

I'd say that's what maxage is all about : "ignore emails older than maxage". I understand that this is not what you want in this case. Having exceptions for maxage is implemented.

ianbrody · 2018-12-02T19:39:43Z

The problem is that it isn't ignoring the local email-- it's deleting it. I think the right behavior would be for it to ignore the email on step 2. But if it doesn't do that (and instead treats this as "new" on the local side) then it should treat both local and remote copies as "new". Instead, it is treating one as new and one as old, which is causing the local one to be deleted. As it stands, I can't copy old emails to my inbox because offlineimap deletes them.

nicolas33 · 2018-12-02T22:29:32Z

I think this issue could be related to 0d6a9a4. What is the version of offlineimap with the expected behaviour?

ianbrody · 2018-12-03T02:45:33Z

I think it's actually 681e27. It's wrong in version 7.1.4 (which looks like is before the commit you mentioned was included) as well as in every subsequent release, but fixed if you revert 681e27 in version 7.1.4. (I can't test with that commit reverted in a later release because stuff seems to have changed around it and it no longer reverts cleanly.)

nicolas33 · 2018-12-03T07:43:58Z

It can't be 681e27 because this commit doesn't touch code and is not a merge. Reverting this commit won't change anything.

Does it work as expected with v7.1.3?

ianbrody · 2018-12-03T17:46:52Z

Oops, my bad, I think my notes got messed up. I actually meant that I thought f37b97c was the problem. v7.1.3 has the bug. v7.1.5 with 0d6a9a4 reverted also has the bug. v7.1.4 with f37b97c reverted does not have the bug.

nicolas33 · 2018-12-03T21:05:55Z

Your tests look right. I wonder we should revert this f37b97c commit because correctly copying emails between folders is more important than changing maxage as reported in #384. In this case I think there will be no other choice than enabling filename_use_mail_timestamp when maxage is set.

Github-ref: OfflineIMAP#581 Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>

nicolas33 · 2018-12-09T01:41:06Z

Could you please test this patch : nicolas33@db2547d

nicolas33 · 2018-12-09T01:44:45Z

Notice: the expected behavior with this patch is to upload the copied email even when older than maxage because it's considered new in the maildir. However, there should be no more deletion once the folder is synced.

ianbrody · 2018-12-10T22:15:27Z

That fixes the issue for me! (tested on version 7.2.1)

Github-ref: #581 Tested-by: https://github.com/ianbrody Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>

nicolas33 · 2018-12-11T08:33:08Z

Thank you much. Applied.

nicolas33 added feature need contributor! labels Dec 1, 2018

nicolas33 mentioned this issue Dec 3, 2018

maxage is respected only once #384

Closed

nicolas33 added a commit to nicolas33/offlineimap that referenced this issue Dec 9, 2018

maxage: always compute the remote cache list from min_uid

db2547d

Github-ref: OfflineIMAP#581 Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>

nicolas33 added a commit that referenced this issue Dec 11, 2018

maxage: always compute the remote cache list for min_uid

a51064e

Github-ref: #581 Tested-by: https://github.com/ianbrody Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>

nicolas33 closed this as completed Dec 11, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't copy external mail older than maxage to local maildir (related to 681e27) #581

Can't copy external mail older than maxage to local maildir (related to 681e27) #581

ianbrody commented Nov 1, 2018 •

edited

Loading

nicolas33 commented Dec 1, 2018

ianbrody commented Dec 2, 2018

nicolas33 commented Dec 2, 2018

ianbrody commented Dec 3, 2018

nicolas33 commented Dec 3, 2018

ianbrody commented Dec 3, 2018

nicolas33 commented Dec 3, 2018

nicolas33 commented Dec 9, 2018

nicolas33 commented Dec 9, 2018

ianbrody commented Dec 10, 2018

nicolas33 commented Dec 11, 2018

Can't copy external mail older than maxage to local maildir (related to 681e27) #581

Can't copy external mail older than maxage to local maildir (related to 681e27) #581

Comments

ianbrody commented Nov 1, 2018 • edited Loading

General information

Configuration file offlineimaprc

Steps to reproduce the error

Log messages

nicolas33 commented Dec 1, 2018

ianbrody commented Dec 2, 2018

nicolas33 commented Dec 2, 2018

ianbrody commented Dec 3, 2018

nicolas33 commented Dec 3, 2018

ianbrody commented Dec 3, 2018

nicolas33 commented Dec 3, 2018

nicolas33 commented Dec 9, 2018

nicolas33 commented Dec 9, 2018

ianbrody commented Dec 10, 2018

nicolas33 commented Dec 11, 2018

ianbrody commented Nov 1, 2018 •

edited

Loading