onedrive: Error reading destination directory with >100000 files

[gualdo0]

Output of rclone version

rclone v1.44-022-ge606d456-fix-onedrive-list-beta

• os/arch: linux/amd64
• go version: go1.11.1

Describe the issue

Hello.
The issue I have with rclone, when trying to sync a local folder with many files (>100000) and a remote folder on OneDrive, is described here:
https://forum.rclone.org/t/error-reading-destination-directory/7316
As you will see, there is apparently a timeout, so the operation at microsoft took longer than some timeout - which looks like 20s (thanks to Nick for looking at this).
Any help would be greatly appreciated.

[ncw]

@Cnly have you got time to look through the forum thread and give us your opinion? It looks like to me that this directory with a large number of files takes more than 20s for the initial listing so times out.

I wonder if we can set the timeout any higher at onedrive?

[ncw]

Note that there is a thread here #2723 (comment) which summarised says 100,000 files is too many, but 50,000 files is OK.

I'll update the docs with that in a moment

[Cnly]

Thanks for sorting that out... I think I should also open an issue in the OneDrive API repo to get some clarification. In that case, it'd be the best if we have a Request ID. @gualdo0 I saw your earlier forum post but the Request ID part is removed. Do you mind providing some log with that information so we can open an issue here (https://github.com/OneDrive/onedrive-api-docs/issues)?

[ncw]

...can we move discussion of the listing number problem back into #2707 where it belongs?

[gualdo0]

@Cnly sorry what is the Request ID? If you mean the post in the rclone forum, the link is
https://forum.rclone.org/t/error-reading-destination-directory/7316
so is the request ID #7316?

[Cnly]

Sorry for the delay! I looked at the forum post again and it seems @gualdo0 you had sent your log privately to @ncw. If @gualdo0 you agree, I'd like to ask @ncw to help provide a snippet of your original log containing the Request ID.

[gualdo0]

Oh yes, of course, please @ncw provide that log file if it is useful to better describe the issue. Regards.

…

________________________________ De: Alex Chen <notifications@github.com> Enviado: miércoles, 28 de noviembre de 2018 11:00 Para: ncw/rclone Cc: González Alfonso Eduardo; Mention Asunto: Re: [ncw/rclone] onedrive: Error reading destination directory with >100000 files (#2707) Sorry for the delay! I looked at the forum post again and it seems @gualdo0<https://github.com/gualdo0> you had sent your log privately to @ncw<https://github.com/ncw>. If @gualdo0<https://github.com/gualdo0> you agree, I'd like to ask @ncw<https://github.com/ncw> to help provide a snippet of your original log containing the Request ID. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#2707 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AqdiHzTblkpWueI1NwI0aF_Zeq4UfWxwks5uzl6ogaJpZM4X9ZNi>.

[ncw]

I've forwarded the log files to @Cnly via email :-)

[Cnly]

Thanks! I've opened an upstream issue.

[Cnly]

@gualdo0 There's an update from the upstream. I see in your previous log that the error happens multiple times. But does it still happen now? If it does, can you reproduce that and share your logs again? Some more recent logs may help solve this better. If you don't want to post the logs publicly, you can also send them to me at minecnly@gmail.com. Thanks!

[ivandeex]

@gualdo0 Does the problem reproduce with rclone 1.54?

[ivandeex]

No answer since 2018. Closing.