Lots of 400 errors with no further details

[autarch]

I just imported about 15 years of mail history using this script, except I only imported about half of it because I got so many 400 errors. Here's an example:

2017-07-10T01:19:04 (+0000) 7625 ERROR process_mbox_files (import-mailbox-to-gmail.py:257) Failed to import mbox message
Traceback (most recent call last):
  File "./import-mailbox-to-gmail.py", line 250, in process_mbox_files
    media_body=media).execute(num_retries=args.num_retries)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/googleapiclient/http.py", line 840, in execute
    raise HttpError(resp, content, uri=self.uri)
HttpError: <HttpError 400 when requesting https://www.googleapis.com/upload/gmail/v1/users/autarch%40urth.org/messages/import?internalDateSource=dateHeader&neverMarkSpam=true&uploadType=multipart&processForCalendar=false&fields=id&alt=json returned "Bad Request">

This doesn't really tell me anything useful.

Was this a transient error? Should I retry my entire import? Is something broken in the mbox file? What's going on.

Overall the google importing experience has been disappointing. I wrote a bit about this at https://blog.urth.org/2017/07/04/going-full-gmail/ if you're interested in some of the other problems I had.

[eesheesh]

Hi Dave,

First, sorry to hear your experience wasn't smooth. I'm aware that there are issues with the data migration tool in the Admin console, particularly around reporting errors - they aren't always very helpful. The good news is that as a G Suite customer you can contact support (via the 'Support' icon in the Admin console) and get help with this. :)

Regarding the API error - my first guess would be a malformed message, but it's also possible it's an issue with the tool's parsing (it uses the Python mailbox library), or the API returning a vague error message (e.g. "Bad Request" instead of saying what's bad about it).

If you can, please create a text mbox file with just one of the affected messages and try to import it. If it succeeds, then this was probably some issue with the request rate. In this case, it's an issue with the API returning the wrong response, because the response should be a status 5xx/429 or a 403 with userRateLimitExceeded/rateLimitExceeded.

If it fails with the same error, and you don't see anything clearly wrong with the message in the mbox, I'd suggest you raise the HTTP logging level to 4 (add --httplib2debuglevel=4) so the full requests and responses are logged, and then contact G Suite support with the log so it can be investigated. You can also post it here but I assume you wouldn't want to post private emails on GitHub. :)

BTW, if you need to re-import some of the data, currently the script would just blindly re-import all of it, but Gmail will de-dupe the data, so although this would still take a lot of time, at least you wouldn't get duplicate messages.

I hope this helps - let me know in any case, so I can follow up internally.

[eesheesh]

Side note: If you want, you can also give got-your-back import a try, although it would probably be affected by API issues the same way.

[eesheesh]

Closing as there was no response.