--dry-run modifies state, changes behavior of next execution (changes "remote delete" operation into "copy from remote").

[fidergo-stephane-gourichon]

General informations

• system/distribution (with version): Debian 7.11
• offlineimap version (offlineimap -V): installed as user from git tag 7.0.4, says: "offlineimap v7.0.4, imaplib2 v2.55 (bundled)"
• Python version: 2.7.3
• server name or domain: private
• CLI options: --dry-run

Configuration file offlineimaprc

[general]
accounts = ovh
pythonfile = ~/.offlineimap/helpers.py

[Account ovh]
localrepository = ovh-local
remoterepository = ovh-remote

[Repository ovh-local]
type = Maildir
localfolders = /home/mylogin/Maildir
nametrans = oimaptransfolder_sgff_back
folderfilter = lambda folder: re.search('^.sg_ff', folder)

[Repository ovh-remote]
type = IMAP
remoteport = 993
ssl = yes
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
remotehost = ssl0.ovh.net
remoteuser = (removed)
remotepass = (removed)
nametrans = oimaptransfolder_sgff

pythonfile (if any)

import re
def oimaptransfolder_sgff(foldername):
    foldername = re.sub("^INBOX\.", ".sg_ff.", foldername)
    if(foldername == "INBOX"):
        retval = ".sg_ff"
    else:
        retval = foldername
    retval = re.sub("/", ".", retval)
    return retval

def oimaptransfolder_sgff_back(foldername):
    if(foldername == ".sg_ff"):
        retval = "INBOX"
    else:
        retval = re.sub("^.sg_ff\.", "INBOX.", foldername)
    return retval

Logs, error

Offlineimap reproducibly copies messages that were successfully synced then deleted on destination.

With a dry-run this is correct:

Folder .sg_ff [remote name INBOX] [acc: ovh]:
 [DRYRUN] Deleting 1707 messages (7536:7537,...) in IMAP[INBOX]

After a dry-run this is incorrect:

Copy message UID 7536 (1/1707) ovh-remote:INBOX -> ovh-local
 Copy message UID 7537 (2/1707) ovh-remote:INBOX -> ovh-local
...

Overall approach

The goal is to safely lighten a remote IMAP box.
The approach is to reliably sync two ways remote IMAP with local Maildir, then in local Maildir move selected messages out of the synced area so that they get deleted on remote server.

Sanity checks performed

Setup above configures so that remote IMAP tree gets synced to subdir sg_ff in local tree.
Things work well.
Checked addition and removal of one mail, flag propagation, etc.
Checked with --info for correct mappings, they are correct.

Steps to reproduce the error

-run offlineimap to get everything in sync.
-move some mail out of sg_ff to some archive (that offlineimap does not process thanks to folderfilter).
-perform offlineimap --dry-run as a sanity check. Output is correct (see above [DRYRUN] Deleting 1707 messages)
-perform offlineimap to get the 1707 correctly deleted on remote server. Output and action are wrong: (see above [Copy message UID 7536 (1/1707))

Steps to get correct behavior

Actually tested: repeat steps above, but don't run the --dry-run, only the real run.
Behavior is then correct.

Alternative way to showcase error

Run with --dry-run not once but twice: second result is not same as first.

Whatever the configuration, etc, running the same command with --dry-run twice in a row (assuming no local or remote activity) should always yield the same result, right? That's the point of a dry run. Observed result means that --dry-run actually changes offlineimap's internal state.

Thank you for your attention.

[fidergo-stephane-gourichon]

I'm aware of --dry-run can drive you nuts · Issue #52 · OfflineIMAP/offlineimap which is a different issue. In that issue, offlineimap was never without --dry-run. In this issue (370) the state before the --dry-run is fully synchronized without problem.

Also, I upgraded from offlineimap version 6.3.4 shipped with Debian precisely because that release does not feature a --dry-run flag, and I wanted to double check remote delete scenarios. Well, they seem to work only when --dry-run is not used.

Thank you for your attention.

[nicolas33]

Could you please test above version?

[fidergo-stephane-gourichon]

Tried commit 25c82fd . Summary: does not fix problem.

Running several dry runs in a row has correct behavior, fixes "Alternative way to showcase error".

 [DRYRUN] Deleting 30 messages (19093,19115:19116,19119,19123:19127,19130,19161:19166,19181:19182,19187,19192:19194,19316,19586,19588:19589,19750:19752,19768) in IMAP[INBOX]

Running a normal run after that still copies mails from remote server, so "Steps to reproduce the error" still reproduce the error.

 Copy message UID 19119 (1/30) ovh-remote:INBOX -> ovh-local

[nicolas33]

Steps to reproduce the error

-run offlineimap to get everything in sync.
-move some mail out of sg_ff to some archive (that offlineimap does not process thanks to folderfilter).

How do you move the mails out? Is the maildir directory removed?

Please, read http://www.offlineimap.org/doc/FAQ.html#may-i-delete-local-folders. I wonder you did not notice this change coming with the v7.0.0 series.

-perform offlineimap --dry-run as a sanity check. Output is correct (see above [DRYRUN] Deleting 1707 messages)
-perform offlineimap to get the 1707 correctly deleted on remote server. Output and action are wrong: (see above [Copy message UID 7536 (1/1707))

[fidergo-stephane-gourichon]

Please, read http://www.offlineimap.org/doc/FAQ.html#may-i-delete-local-folders. I wonder you did not notice this change coming with the v7.0.0 series.

I had read that before and it's not that case.

How do you move the mails out? Is the maildir directory removed?

No, no maildir directory is removed.

The files are just moved from one .sourcefolder/cur to .destinationfolder/cur by regular mv command from the shell (which is atomic because on same filesystem, faithful to the Maildir philosophy).
dovecot IMAP server sees the change just fine
offlineimap acknowledges and does the right thing just fine.
Only a dry run causes bad behavior.

Since everything works just fine when not doing a dry run, there should be no gotcha specific to IMAP server used or whatever.

Have you tried to reproduce the steps are reported?

Do you want me to perform a diff of ~/.offlineimap before and after dry-run (including per-folder sqlite databases)?

[nicolas33]

No, no maildir directory is removed.

I assume you meant "is NOT removed".

Since everything works just fine when not doing a dry run, there should be no gotcha specific to IMAP server used or whatever.

Ok. So, another change is likely made into the local cache while it should not.

Have you tried to reproduce the steps are reported?

No. I'm relying on your reports that are quite good, BTW.

Do you want me to perform a diff of ~/.offlineimap before and after dry-run (including per-folder sqlite databases)?

That would be wonderful to find the offending code.

[fidergo-stephane-gourichon]

Do you want me to perform a diff of ~/.offlineimap before and after dry-run (including per-folder sqlite databases)?

That would be wonderful to find the offending code.

Here you are.

The dry run showed this:

 [DRYRUN] Deleting 6 messages (21992,21994:21995,21997:21998,22009) in IMAP[INBOX]

The dry run changed only one file: the sqlite database of changed folder:

for A in offlineimap_backup_*/ ; do echo "${A}" ; FILE="${A}/Account-ovh/LocalStatus-sqlite/.sg_ff" ; sqlite3 "$FILE" .dump >"${FILE}.dump" ; done

diff -u */*/*/.*.dump


--- offlineimap_backup_2016y08m09d-21h34m27s/Account-ovh/LocalStatus-sqlite/.sg_ff.dump 2016-08-09 21:37:26.670146172 +0200
+++ offlineimap_backup_2016y08m09d-21h35m13s/Account-ovh/LocalStatus-sqlite/.sg_ff.dump 2016-08-09 21:37:26.690146073 +0200
@@ -3059,13 +3059,8 @@
 INSERT INTO "status" VALUES(21982,'',0,'');
 INSERT INTO "status" VALUES(21984,'FRS',0,'');
 INSERT INTO "status" VALUES(21986,'S',0,'');
-INSERT INTO "status" VALUES(21992,'S',0,'');
 INSERT INTO "status" VALUES(21993,'S',0,'');
-INSERT INTO "status" VALUES(21994,'S',0,'');
-INSERT INTO "status" VALUES(21995,'S',0,'');
 INSERT INTO "status" VALUES(21996,'FS',0,'');
-INSERT INTO "status" VALUES(21997,'S',0,'');
-INSERT INTO "status" VALUES(21998,'S',0,'');
 INSERT INTO "status" VALUES(21999,'S',0,'');
 INSERT INTO "status" VALUES(22000,'',0,'');
 INSERT INTO "status" VALUES(22001,'S',0,'');
@@ -3076,7 +3071,6 @@
 INSERT INTO "status" VALUES(22006,'',0,'');
 INSERT INTO "status" VALUES(22007,'S',0,'');
 INSERT INTO "status" VALUES(22008,'S',0,'');
-INSERT INTO "status" VALUES(22009,'S',0,'');
 INSERT INTO "status" VALUES(22010,'S',0,'');
 INSERT INTO "status" VALUES(22011,'',0,'');
 INSERT INTO "status" VALUES(22012,'',0,'');

Hope this helps.

[nicolas33]

Did you got those results with the patched offlineimap?

[fidergo-stephane-gourichon]

Good question! I had fetched the correct commit from nicolas33 repo but not checked out.
With that version behavior was found to be correct.

# HEAD detached at 25c82fd

This line

[DRYRUN] Deleting 9 messages (19093,19316,19586,19588:19589,19750:19752,19768) in IMAP[INBOX]

becomes

Deleting 9 messages (19093,19316,19586,19588:19589,19750:19752,19768) in IMAP[INBOX]

I also tested several dry runs in a row and it was also consistent (the log wasn't exactly the same, I guess because of non-deterministic thread execution order, but the simulated commands were the same).

Summary: behavior observed here with 25c82fd is correct with our test scenario.

Should I continue to use that commit from now on, or do you recommend something else (like switching to a release, backporting that patch alone, whatever)?

Thank you for your attention.

[nicolas33]

I won't backport the patch because it's not so critical.
I've just released v7.0.5 with this fix.