offlineimap partial download

[jungle-boogie]

General informations

• offlineimap version (offlineimap -V): offlineimap v7.0.0, imaplib2 v2.55 (bundled)
• Python version: Python 2.7.12 (default, Jul 3 2016, 01:17:27) [GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
• server name or domain: home.example.net
• CLI options: none

Configuration file offlineimaprc

[general]    
accounts = Work                   
maxsyncaccounts = 1
ui = TTYUI

[mbnames]                     
enabled = yes        
filename = ~/.mutt/mailboxes/
header = "mailboxes "                                               
peritem = "+%(accountname)s/%(foldername)s"                                                           
sep = "                                                               
footer = "\n"

[Account Work]                                                            
localrepository = work-local                                               
remoterepository = work-remote                                              
status-backend = sqlite                                                
autorefresh = 2                                                       
quick = 10

[Repository work-local]                                                         
type = Maildir                                                             
localfolders = ~/mail/work

[Repository work-remote]                                                        
type = IMAP                                                              
remotehost = mail.host.com                                                   
ssl = no
emotehost = 192.168.0.21                                                       
remoteport = 1143                                                     
remoteuser = CORPORATE\jungle                                              
#remoteuser = jungle@xxx.com                                   
remotepass  = xxx                                                
realdelete=yes                                                       
maxconnections = 1                                                      
folderfilter = lambda foldername: not re.search('Location|Junk|Unsent|Sync|Work|Trash|Calendar|AllItems|Deferred|Exchange|Finder|Freebusy|Mailbox|My|News|People|Recoverable|Reminders|RSS|Schedule|Sharing|Shortcuts|Spooler|System|Tracked|V ews', foldername)                                                           
nametrans = lambda folder: re.sub('INBOX', 'Work.Inbox', folder)

pythonfile (if any)

none

Error

Initial sync with offlineimap + davmail downloads about 1300 emails in sent, but then goes into its periodic check leaving behind the inbox and 12 separate directories.

Folder Sent [acc: Work]:
 Syncing Sent: IMAP -> Maildir
 Skipping Sent (not changed)
Account sync Work:
 *** Finished account 'Work' in 0:05
 Next refresh in 2.0 minutes
 Next refresh in 1.0 minutes

Will offlineimap eventually download the full mailbox or did I make a mistake in my configuration file?

[nicolas33]

Please, check the non-synced folders are not in the folderfilter option. Stupid check but things like that happen to all of us.

I think you're missing the reversed nametrans rule nametrans = lambda folder: re.sub('Work.Inbox', 'INBOX', folder) in the local section.

Initial sync with offlineimap + davmail downloads about 1300 emails in sent, but then goes into its periodic check leaving behind the inbox and 12 separate directories.

Is the "initial' sync ok in the sense "on each new program start" or "first of all the syncs ever"?

[jungle-boogie]

I think you're missing the reversed nametrans rule

Any way to have offlineimap completely ignore and download everything?

Is the "initial' sync ok in the sense "on each new program start" or "first of all the syncs ever"?

First ever sync on this mailbox.

[nicolas33]

Any way to have offlineimap completely ignore and download everything?

Not "completely". Offlineimap is a syncer. The reversed nametrans rule is not required if you disable the creation of folders on the remote.

First ever sync on this mailbox.

Ok. Now, we'll try to figure where the issue comes from.

Is this reproducible with v6.7.0.2?
Does disabling autorefresh make a proper sync?

[nicolas33]

To be sure: "partial" download means:

• some folders seems fully ignored and missing in the output of synced folder;
• offlineimap pretends to sync them but the content is not actually synced;
• some new emails are downloaded but not others.

[jungle-boogie]

Is this reproducible with v6.7.0.2?

I downloaded that version and installed but /usr/local/bin/offlineimap --version always reports 7.0.0.0:

% /usr/local/bin/offlineimap --version
7.0.0

"partial" download means:

ONLY sent directory is downloaded and all others are ignored.

If you can provide a new .offlineimaprc that would download are directories, I'll try it out.

[nicolas33]

I downloaded that version and installed but /usr/local/bin/offlineimap --version always reports 7.0.0.0:

Uninstall all versions. To test different versions, there's no need to install. Download and ./offlineimap.py. This will be the best configuration for now.

ONLY sent directory is downloaded and all others are ignored.

What's the output of ./offlineimap.py --info?

[jungle-boogie]

Download and ./offlineimap.py. That's a good tip! Thanks

OfflineIMAP 6.7.0.2
  Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
Remote repository 'work-remote': type 'IMAP'
Host: 192.168.0.21 Port: 1143 SSL: False
Establishing connection to 192.168.0.21:1143
Server welcome string: * OK [CAPABILITY IMAP4REV1 AUTH=LOGIN MOVE] IMAP4rev1 DavMail 4.7.2-2427 server ready
Server capabilities: ('IMAP4REV1', 'AUTH=LOGIN', 'MOVE')

Folderlist:
 Drafts
 INBOX
 INBOX/0
 INBOX/1
 INBOX/2
 INBOX/3
 INBOX/4
 INBOX/5                                         
 INBOX/6
 INBOX/7
 INBOX/8
 INBOX/9
 INBOX/10
 INBOX/11
 Junk
 Sent
 Sync Issues
 Sync Issues/Conflicts
 Sync Issues/Local Failures
 Sync Issues/Server Failures
 Trash
 Unsent Messages
 Working Set

Local repository 'work-local': type 'Maildir'
Folderlist:
 Drafts
 Work.Inbox.5
 Work.Inbox.4
 Work.Inbox.10
 Work.Inbox.7
 Sent
 Work.Inbox.6
 Work.Inbox.11
 Work.Inbox.12
 Work.Inbox.3
 Work.Inbox.2
 Work.Inbox.9
 Work.Inbox.1
 Work.Inbox.8
 Work.Inbox
 INBOX.3
 Sync Issues.Server Failures
 Sync Issues
 INBOX.6
 INBOX.4
 Trash

[nicolas33]

I can't dig into this with metasyntactic data. I'm fine with a private mail.

[jungle-boogie]

Email sent.

[nicolas33]

I'm not sure how this was fixed but it looks like it is. If not please let me know.

[jungle-boogie]

yes, this is better! Thanks.

[jpalus]

For me the issue is still not fixed. First of all offlineimap does not print actual error with -d all, only blank line. Error is reported with warn level:

                    self.ui.warn('', "Ignoring folder %s due to unsupported "
                        "'%s' character serving as local separator."%
                        (remotefolder.getvisiblename(), localrepos.getsep()))
                    continue # Ignore unsupported folder name.

The error seems to appear because remotefolder.getvisiblename() looks to be already processed by nametrans. Relevant configuration:

sep = .
...
nametrans = lambda dir: '' if dir == 'INBOX' else '.' + re.sub(r'\[Gmail\]/Sent Mail','Sent',dir

basically nametrans adds dot at the beginning of folder name so ie remote folder pld/devel-en ends up as .pld.devel-en. However getvisiblename() returns something in between: .pld/devel-en.

[nicolas33]

@jpalus Please, open a new issue.

[nicolas33]

@jpalus Please, open a new issue.

My bad. Don't. It's the correct issue.

[nicolas33]

@jpalus Could you test above version, please?

[jpalus]

Works fine now. Thanks!

[nicolas33]

@jpalus Thank you for the report and the analysis.
Fix applied into the next branch. I'll release a v7.0.3 soon.