Google now enforces a maximum number of files per folder

[giomasce]

Looking at https://stackoverflow.com/a/62520242/807307 and outgoing links, it seems that now Google is enforcing a maximum of 500k files per single folder. Large annexes might hit this limit, so it might be sensible to reconsider the decision of avoiding subdirs.

[Lykos153]

Well that's unfortunate. Thank's for bringing this up! I'll think of something...

[froseph]

I have just hit this limit recently since i've been using git annex as an everything bin :/

[Lykos153]

Alright. Man, I use annex for everything, too, and I thought I had lots of data. 😆 Can you please post the output you get after reaching the limit?

[froseph]

Depending on version I get either

(to google...)
Traceback (most recent call last):
  File "/Users/froseph/Library/Python/3.8/bin/git-annex-remote-googledrive", line 8, in <module>
    sys.exit(main())
  File "/Users/froseph/Library/Python/3.8/lib/python/site-packages/git_annex_remote_googledrive/run.py", line 180, in main
    master.Listen()
  File "/Users/froseph/Library/Python/3.8/lib/python/site-packages/annexremote/annexremote.py", line 860, in Listen
    reply = self.protocol.command(line)
  File "/Users/froseph/Library/Python/3.8/lib/python/site-packages/annexremote/annexremote.py", line 551, in command
    reply = method(parts[1])
  File "/Users/froseph/Library/Python/3.8/lib/python/site-packages/annexremote/annexremote.py", line 600, in do_TRANSFER
    func(key, file_)
  File "/Users/froseph/Library/Python/3.8/lib/python/site-packages/git_annex_remote_googledrive/google_remote.py", line 69, in wrapper
    return f(self, *args, **kwargs)
  File "/Users/froseph/Library/Python/3.8/lib/python/site-packages/tenacity/__init__.py", line 329, in wrapped_f
    return self.call(f, *args, **kw)
  File "/Users/froseph/Library/Python/3.8/lib/python/site-packages/tenacity/__init__.py", line 409, in call
    do = self.iter(retry_state=retry_state)
  File "/Users/froseph/Library/Python/3.8/lib/python/site-packages/tenacity/__init__.py", line 368, in iter
    raise retry_exc.reraise()
  File "/Users/froseph/Library/Python/3.8/lib/python/site-packages/tenacity/__init__.py", line 186, in reraise
    raise self.last_attempt.result()
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/Users/froseph/Library/Python/3.8/lib/python/site-packages/tenacity/__init__.py", line 412, in call
    result = fn(*args, **kwargs)
  File "/Users/froseph/Library/Python/3.8/lib/python/site-packages/git_annex_remote_googledrive/google_remote.py", line 98, in wrapper
    return f(self, *args, **kwargs)
  File "/Users/froseph/Library/Python/3.8/lib/python/site-packages/git_annex_remote_googledrive/google_remote.py", line 292, in transfer_store
    self.root.new_key(key).upload(
  File "/Users/froseph/Library/Python/3.8/lib/python/site-packages/git_annex_remote_googledrive/keys.py", line 123, in upload
    self.file.upload(local_filename,
  File "/Users/froseph/Library/Python/3.8/lib/python/site-packages/drivelib/drive.py", line 365, in upload
    status, response = request.next_chunk()
  File "/Users/froseph/Library/Python/3.8/lib/python/site-packages/drivelib/drive.py", line 478, in next_chunk
    content_length = min(self.media_body.size()-self.resumable_progress, self.media_body.chunksize())
  File "/Users/froseph/Library/Python/3.8/lib/python/site-packages/drivelib/drive.py", line 444, in resumable_progress
    raise HttpError(status, resp)
googleapiclient.errors.HttpError: <HttpError 403 "The limit for this folder's number of children (files and folders) has been exceeded.">

  external special remote protocol error, unexpectedly received "<EOF>" (unable to parse command)

  unable to use special remote due to protocol error
failed
git-annex: copy: 1 failed

or in older versions of git-annex-remote-googledrive:

(('Failed to connect with Google. Please check your internet connection.', AttributeError("'Http' object has no attribute 'redirect_codes'"))) failed
git-annex: copy: 1 failed

[Lykos153]

I finally finished my work on this and made a release candidate:
https://github.com/Lykos153/git-annex-remote-googledrive/releases/tag/v1.3.0-rc.1
https://pypi.org/project/git-annex-remote-googledrive/1.3.0rc1/
It now supports multiple directory layouts and can migrate automatically between them. I'm going to test them for a while before I release them. If you want @froseph , you can check them out already. You'll just need to perform the steps described in https://github.com/Lykos153/git-annex-remote-googledrive/tree/dev/layouts#fix-full-folder or enable the remote with git annex enableremote <remote_name> layout=<layout_choice> auto_fix_full=yes.