APPEND issue...

[mobamoba]

I opened this issue which we closed recently because nothing had happened with it in a while (#172). Well, it's back, and I'm 99.9999% sure I know the source of the problem (though not the solution): large file attachments. I sync from an Exchange server to Gmail and both of them have an attachment limit of 25mb. Syncing a 20mb will trigger the error. I haven't tested to see if there's a max file size offlineimap can sync without triggering the error, but if I had to take a guess, it's any file size that takes less than 60 seconds to sync because when the syncing goes on longer than 60 seconds, offlineimap (I think) thinks it's timing out.

Does this make sense? And, in a practical sense, is there a way to either: (a) tell offlinimap to ignore attachments above a certain size, (b) manually mark a message as "synced" so that offlineimap will ignore that particular message in the future, or (c) something else so that offlineimap doesn't crash when it hits very large attachments that take an unusually long time to either upload or download to or from the syncing servers?

[nicolas33]

Could you please provide your configuration file? Remove private data!

[mobamoba]

[general]
accounts = Exchange, SLI, TOP, JMI, Shopping, Stuff
maxsyncaccounts = 7
ignore-readonly = no
socktimeout = 60

[Account Exchange]
localrepository = Local
remoterepository = Remote
autorefresh = -1
maxconnections = 2
status_backend = sqlite

[Repository Local]
type = IMAP
remotehost = www.xxx.com
ssl = yes
cert_fingerprint = xxx
remotepass = xxx
remoteuser = xxx@xxx.com
nametrans = lambda sentname: re.sub ('Sent Items', 'Sent Current',
sentname)

[Repository Remote]
type = Gmail
remotepass = xxx
remoteuser = xxx@xxx.com
sslcacertfile = /home/Matt/ca-certificates.crt
nametrans = lambda sentname: re.sub ('Sent Current', 'Sent Items',
sentname)
folderfilter = lambda foldername: foldername in [
'INBOX', 'Saved Current', 'Sent Current', 'Drafts', 'Trash', '[Gmail]/Spam',
'SMS', 'SMSArchived', 'Call log', 'Call log Archived']
keepalive = 60

[Account SLI]
localrepository = Local-SLI
remoterepository = Remote-SLI
autorefresh = 20
maxconnections = 1
status_backend = sqlite

[Repository Local-SLI]
type = IMAP
remotehost = www.xxx.com
ssl = yes
cert_fingerprint = xxx
remotepass = xxx
remoteuser = xxx@xxx.com

[Repository Remote-SLI]
type = Gmail
remotepass = xxx
remoteuser = xxx@xxx.com
sslcacertfile = /home/Matt/ca-certificates.crt
folderfilter = lambda foldername: foldername in [
'SLISent']
nametrans = lambda sentname: re.sub ('SLISent', 'Sent SLI',
sentname)
readonly = True

[Account TOP]
localrepository = Local-TOP
remoterepository = Remote-TOP
autorefresh = 20
maxconnections = 1
status_backend = sqlite

[Repository Local-TOP]
type = IMAP
remotehost = www.xxx.com
ssl = yes
cert_fingerprint = xxx
remotepass = xxx
remoteuser = xxx@xxx.com

[Repository Remote-TOP]
type = Gmail
remotepass = xxx
remoteuser = xxx@xxx.com
sslcacertfile = /home/Matt/ca-certificates.crt
folderfilter = lambda foldername: foldername in [
'TOPSent']
nametrans = lambda sentname: re.sub ('TOPSent', 'Sent TOP',
sentname)
readonly = True

[Account JMI]
localrepository = Local-JMI
remoterepository = Remote-JMI
autorefresh = 20
maxconnections = 1
status_backend = sqlite

[Repository Local-JMI]
type = IMAP
remotehost = www.xxx.com
ssl = yes
cert_fingerprint = xxx
remotepass = xxx
remoteuser = xxx@xxx.com

[Repository Remote-JMI]
type = Gmail
remotepass = xxx
remoteuser = xxx@xxx.com
sslcacertfile = /home/Matt/ca-certificates.crt
folderfilter = lambda foldername: foldername in [
'JMISent']
nametrans = lambda sentname: re.sub ('JMISent', 'Sent JMI',
sentname)
readonly = True

[Account Shopping]
localrepository = Local-Shopping
remoterepository = Remote-Shopping
autorefresh = 20
maxconnections = 1
status_backend = sqlite

[Repository Local-Shopping]
type = IMAP
remotehost = www.xxx.com
ssl = yes
cert_fingerprint = xxx
remotepass = xxx
remoteuser = xxx@xxx.com

[Repository Remote-Shopping]
type = Gmail
remotepass = xxx
remoteuser = xxx@xxx.com
sslcacertfile = /home/Matt/ca-certificates.crt
folderfilter = lambda foldername: foldername in [
'ShoppingSent']
nametrans = lambda sentname: re.sub ('ShoppingSent', 'Sent Shopping',
sentname)
readonly = True

[Account Stuff]
localrepository = Local-Stuff
remoterepository = Remote-Stuff
autorefresh = 20
maxconnections = 1
status_backend = sqlite

[Repository Local-Stuff]
type = IMAP
remotehost = www.xxx.com
ssl = yes
cert_fingerprint = xxx
remotepass = xxx
remoteuser = xxx@xxx.com

[Repository Remote-Stuff]
type = Gmail
remotepass = xxx
remoteuser = xxx@xxx.com
sslcacertfile = /home/Matt/ca-certificates.crt
folderfilter = lambda foldername: foldername in [
'StuffSent']
nametrans = lambda sentname: re.sub ('StuffSent', 'Sent Stuff',
sentname)
readonly = True

[nicolas33]

I wonder if there's a bug around maxconnections. Try with 1 everywhere, please.

[mobamoba]

I tried and that didn't solve the problem.

It's definitely an issue with large files. In the past, it was an email with a 23mb attachment; this time it's an email with an 20mb attachment. Offlineimap keeps duplicating the file as it fails to copy it. For instance, at this point, I have 5 copies of that 20mb file in one of my folders. The duplicates are 100% from offlineimap as I (obviously) only sent the file once. Specifically: I sent the file from my regular server; offlineimap synced it to Gmail; as of now, there are 5 copies of the file on my server and 1 copy in Gmail. The duplicates that offlineimap created and the fact that there are 5 copies on one server and only 1 on the other must be confusing the program in some way and causing it to throw the error, right?

As I noted, I've had this problem with other large files. Unfortunately, the only solution is to delete the files from both servers, which, frankly, isn't much of a solution really.

[hlopko]

Hello,

I too experience this issue and can provide information if needed. I see similar behavior, but I think it doesn't depend on the file size but on the uploading time. I had bad connection yesterday, so the bug triggered even on 2mb attachment. Now I see the message copied 119 times :) What is more funny is that the messages are fully copied, it's only the offlineimap who thinks it didn't succeed in uploading. Relevant part from the logfile (with extra linebreaks to fit into the column:

WARNING: ERROR: Saving msg (<20151203110123.GB8421@zemetras>) in 
    folder 'INBOX.Sent Items', repository 'fastmailRemote' failed (abort). 
    Server responded: command: APPEND => no response after 120.0 secs
Message content was: Date: Thu, 3 Dec 2015 12:01:23 +0100
From: Marcel Hlopko <marcel@hlopko.com>
To: Tisk =?iso-8859-1?Q?svatebn=EDch_ozn=E1men=ED?= 
    <fotosvatebka@fotos...--END PGP SIGNATURE-----

--uXxzq0nDebZQVNAZ--

2015-12-03 12:21:22 WARNING: 
Traceback:
  File "/usr/lib/python2.7/site-packages/offlineimap/folder/Base.py", line 775, 
    in copymessageto 
      new_uid = dstfolder.savemessage(uid, message, flags, rtime)
  File "/usr/lib/python2.7/site-packages/offlineimap/folder/IMAP.py", line 583, 
    in savemessage
      imaputil.flagsmaildir2imap(flags), date, content)
  File "/usr/lib/python2.7/site-packages/offlineimap/imaplib2.py", line 638, 
    in append
      return self._simple_command(name, mailbox, flags, date_time, **kw)
  File "/usr/lib/python2.7/site-packages/offlineimap/imaplib2.py", line 1633, 
    in _simple_command
      return self._command_complete(self._command(name, *args), kw)
  File "/usr/lib/python2.7/site-packages/offlineimap/imaplib2.py", line 1393, 
    in _command_complete
      typ, dat = rqb.get_response('command: %s => %%s' % rqb.name)
  File "/usr/lib/python2.7/site-packages/offlineimap/imaplib2.py", line 188, 
    in get_response
      raise typ(exc_fmt % str(val))

Thank you for taking the time for this issue and all your work on offlineimap!

[nicolas33]

There was a fix about APPEND, very recently. Please try git next branch (not even latest release candidate, it's not there).

Needing logs with imap debug logs enabled.

[nicolas33]

I've just released v6.6.0 so you could use this. :-)

[hlopko]

I tried v6.6.0, but it still crashes. I tried sending an email with 18mb attachment and increasing timeout. But still no luck. Detailed log can be found here:

https://www.dropbox.com/s/dcjoo7yxzary1kk/append_issue.log?dl=0

[nicolas33]

OMG. 93Mo is a lot. If the issue is reproducible, please try a new run until there's almost only the issue left is the logs.

[nicolas33]

Quite old. Please, open a new issue if still valid.

[Profpatsch]

Still happening. In my case I had a message that was 40MB and I wanted to move it from my inbox folder to the archive folder. After some time I always get:

 Establishing connection to SNIP:993 (MailRemote)
 ERROR: Saving msg (<SNIP>) in folder 'INBOX.Archive', repository 'MailRemote' failed (abort). Server responded: command: APPEND => no response after 120.0 secs
Message content was: Received: from [SNIP_IP] (helo=SNIP)
	by SNIP with esmtp (Exim 4.68)
	(envelope-from <SNIP>)...ZWYKNjEwMzA3MQolJUVPRgo=

Looks to me like the transfer takes >120 sec (quite an arbitrary time) and offlineimap doesn’t tell the server it’s still transferring, so the server aborts.

[nicolas33]

@Profpatsch Please, open a new issue.