Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue syncing large google drive directories #18

Closed
desyncr opened this issue Apr 19, 2020 · 10 comments
Closed

Issue syncing large google drive directories #18

desyncr opened this issue Apr 19, 2020 · 10 comments

Comments

@desyncr
Copy link
Contributor

desyncr commented Apr 19, 2020

Before anything: thank you for the work invested in this project!

Now on the issue: I have used the google drive integration without problem for months. Recently I started to experience issues syncing.

Taking a look at the sync process I noticed two issues:

  • Google drive detects multiple downloads on a row and complains as follows:
Your computer or network may be sending automated queries.
To protect our users, we can't process your request right now

The sync process does not detect this error and saves it as the book contents. Further sync wont work as the file appears downloaded and the remoteSize is not obtained and defaults to 1 (the content previously saved size is 1k).

  • The sync process seems to take longer than it's expected and is stopped before it finished. I noticed this looking at the logs and seeing that the lasts files wont get synced (it processes items in alphabetical ascending order).

I have logs if that helps. Some details to reproduce: 70 files, averaging 1.5Mb each.

My intention is to use this shared directory for permanent storage for my books, rather than to use it as a temporary place for syncing.
@bagcivan
Copy link

bagcivan commented May 5, 2020

Yes lately it is not working for google drive. @fsantini can you help us

@fsantini
Copy link
Owner

fsantini commented May 6, 2020

I'll try to have a look over the weekend!

@desyncr
Copy link
Contributor Author

desyncr commented May 7, 2020

Thanks @fsantini let me know if I can help with anything. I was thinking of having a sort of cache (but that sounds like more problems).

@desyncr
Copy link
Contributor Author

desyncr commented May 9, 2020 •
edited
Loading

@fsantini Something I noticed that's happening (that's not related to large directories) is that Google Drive is returning the following:

<HTML>
<HEAD>
<TITLE>Moved Temporarily</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000">
<H1>Moved Temporarily</H1>
The document has moved <A HREF="...">here</A>.
</BODY>
</HTML>

I'm testing a quickfix using curl -L. Nevermind. It actually does includes -L.

@mmmx10
Copy link

mmmx10 commented May 26, 2020

If thi can help...

I actually can get files from dropbox and not from drive, on Libra.

Looking at the get.log files it seems kobocloud can't get the filelist inside the folder of drive, so the getting line always write the folder shared link (in the log from dropbox I find filenames after the shared link)

@desyncr
Copy link
Contributor Author

desyncr commented May 26, 2020

I tried with Dropbox and had no issues. Do you have many ebooks there? Can you share the log (remove any personal information)?

@mmmx10
Copy link

mmmx10 commented May 28, 2020

I don't have problems with dropbox but with gdrive
this is the log for drive
get-drivekoe.log
this the log for dropbox
get-example.log

In the first one you don't have filenames of files, only the hash of the folder...

@desyncr
Copy link
Contributor Author

desyncr commented May 31, 2020 •
edited
Loading

@mmmx10 The issue is that you got an extra character on the URL:

Screen Shot 2020-05-31 at 15 16 27

Remove that ^M (carry return) and it should work. You may have to change the line-ending in whatever text editor you're using.

See this and this. Also this.

@tpet93
Copy link

tpet93 commented Jun 18, 2020

I found issues whith syncing when i had square brackets "[" in the filenames on Gdrive

@desyncr
Copy link
Contributor Author

desyncr commented Jun 18, 2020

Yeah, I think I've seen similar issues before. I'll try it out later. Thanks for the report.

@VPeeters VPeeters closed this as completed Jun 4, 2021
@VPeeters VPeeters reopened this Jun 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@groentebroer @desyncr @fsantini @bagcivan @mmmx10 @tpet93 @VPeeters