"cannot reduce request any further"

[pazz]

I set up alot to use the maildir syncronized by gmailieer as send_box. When I send a mail (and store and tag it "sent") and then syncronize I get the following error.

[~/mail/account.patricktotzke.gmail.com] ~/projects/gmailieer/gmi sync                                                    
receiving metadata:   0%|                                                                                               | 0/9 [00:00<?, ?it/s]reducing batch request size to: 25
reducing batch request size to: 12
reducing batch request size to: 6
Traceback (most recent call last):
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 282, in get_messages
    batch.execute (http = self.http)
  File "/usr/lib/python3/dist-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/googleapiclient/http.py", line 1451, in execute
    self._callback(request_id, response, exception)
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 258, in _cb
    raise Remote.BatchException(excep)
lieer.remote.BatchException: <HttpError 400 when requesting https://www.googleapis.com/gmail/v1/users/patricktotzke%40gmail.com/messages/1511291035.M158427P31771Q1.piu?format=minimal&alt=json returned "Invalid id value">

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pazz/projects/gmailieer/gmi", line 8, in <module>
    g.main ()
  File "/home/pazz/projects/gmailieer/lieer/gmailieer.py", line 143, in main
    args.func (args)
  File "/home/pazz/projects/gmailieer/lieer/gmailieer.py", line 193, in sync
    self.push (args, True)
  File "/home/pazz/projects/gmailieer/lieer/gmailieer.py", line 238, in push
    self.remote.get_messages (gids, _got_msg, 'minimal')
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 109, in func_wrap
    return func (self, *args, **kwargs)
  File "/home/pazz/projects/gmailieer/lieer/remote.py", line 305, in get_messages
    raise Remote.BatchException ("cannot reduce request any further")
lieer.remote.BatchException: cannot reduce request any further
[1]    431 abort      ~/projects/gmailieer/gmi sync

[gauteh]

Patrick Totzke writes on november 21, 2017 20:07:

I set up alot to use the maildir syncronized by gmailieer as `send_box`. When I send a mail (and store and tag it "sent") and then syncronize I get the following error.

Hi, gmailieer does not push any new messages (only tag changes on existing messages). Additionally, the 'sent' tag cannot be set manually. It is added based on from address and a few other ways (see GMail API docs). If I send a message using a GMail account it is added automatically by GMail to sent items, and is fetched (with the sent tag) on the next sync. Are you using a differnet account? If you want to have a duplicate copy of the sent message it would probably be better to store it outside of the gmaileer root. As for the backtrace; I guess we should better handle the "invalid id value" - but for now it is not supported to add messages manually to the gmailieer root. It probably stems from trying to update the tags on a non-exisiting message. Note that the file-name is matched to the google ID for the message (which you cannot really know before after the message has been sent). Google ID != message id.

…

File "/home/pazz/projects/gmailieer/lieer/remote.py", line 258, in _cb raise Remote.BatchException(excep) lieer.remote.BatchException: <HttpError 400 when requesting https://www.googleapis.com/gmail/v1/users/patricktotzke%40gmail.com/messages/1511291035.M158427P31771Q1.piu?format=minimal&alt=json returned "Invalid id value">

[pazz]

Hi and thanks a lot for the explanation. Maybe you'd want to add a note to the docs to mention that one cannot write to the synced maildir, to prevent others from running into the same issue.

[gauteh]

Patrick Totzke writes on november 22, 2017 9:37:

Hi and thanks a lot for the explanation. Maybe you'd want to add a note to the docs to mention that one cannot write to the synced maildir, to prevent others from running into the same issue.

Yes, is this a deal-breaker in anyway? It is possible to insert messages, but with the current design I think the easiest might be through a `gmi insert` command. Or even a `gmi send` which acts like a MTA.

[pazz]

To be honest, I'm reverting to imap for the moment, because my sync happens in a cronjob and I cannot be bothered to check the logs for unexpected failures.

I think that your restrictions are reasonable. Most of those are out of your hand anyway. You just need to make it explicit so that users understand the deal. What could help would be if you get some users share their GUI setup files, say for mutt, notmuch/emacs or alot.

I would advise to keep it simple instead of adding insert/send commands. The power of notmuch and friends in my opinion lies in the UNIX philosophy, that you have many small parts that do very simple jobs very well. For me ideally, this project would only sync labels between notmuch and gmail, and leave downloading messages to some other tool, say via imap. That might be impossible due to technical restrictions, but you'll know better than I do. I would happily accept the slowdown in synchronization this might introduce, for the added benefit of having stable imap sync independently in case this tagsync tool fails. But that's just my 2¢..

[gauteh]

Yeah, agreed. The main purpose for me was something simple that fetches the emails quickly and syncs the tags. It is easier to keep track of both together. I'm not sure IMAP actually transfers emails any faster than gmi does. Anyway, thanks for your comments.

Btw, it says in the tag-line that gmailieer only fetches emails.

[julian-klode]

I was considering writing something that takes email in an outbox tag or something and sends them via the API asynchronously (well, I'd run it when gmi is pushing). But not clear plans yet.

[gauteh]

Julian Andres Klode writes on november 22, 2017 14:28:

I was considering writing something that takes email in an outbox tag or something and sends them via the API asynchronously (well, I'd run it when gmi is pushing). But not clear plans yet.

In what scenarios would this be used, since normally the message is already added by GMail itself? I currently see two; * Backing up mail from other accounts * Writing a MTA using the GMail API (emplacing sendmail)

[julian-klode]

Yeah, writing an MTA.

[gauteh]

Julian Andres Klode writes on november 22, 2017 15:33:

Yeah, writing an MTA.

Ok! I assume you know about: https://developers.google.com/gmail/api/v1/reference/users/messages/send

[gauteh]

See #131