New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

onedrive: Error reading destination directory with >100000 files #2707

Open
gualdo0 opened this Issue Oct 27, 2018 · 10 comments

Comments

Projects
None yet
3 participants
@gualdo0

gualdo0 commented Oct 27, 2018

Output of rclone version

rclone v1.44-022-ge606d456-fix-onedrive-list-beta

  • os/arch: linux/amd64
  • go version: go1.11.1

Describe the issue

Hello.
The issue I have with rclone, when trying to sync a local folder with many files (>100000) and a remote folder on OneDrive, is described here:
https://forum.rclone.org/t/error-reading-destination-directory/7316
As you will see, there is apparently a timeout, so the operation at microsoft took longer than some timeout - which looks like 20s (thanks to Nick for looking at this).
Any help would be greatly appreciated.

@ncw

This comment has been minimized.

Owner

ncw commented Oct 28, 2018

@Cnly have you got time to look through the forum thread and give us your opinion? It looks like to me that this directory with a large number of files takes more than 20s for the initial listing so times out.

I wonder if we can set the timeout any higher at onedrive?

@ncw ncw added this to the v1.45 milestone Oct 28, 2018

@ncw ncw changed the title from Error reading destination directory to onedrive: Error reading destination directory with >100000 files Oct 28, 2018

@ncw

This comment has been minimized.

Owner

ncw commented Nov 11, 2018

Note that there is a thread here #2723 (comment) which summarised says 100,000 files is too many, but 50,000 files is OK.

I'll update the docs with that in a moment

ncw added a commit that referenced this issue Nov 11, 2018

@Cnly

This comment has been minimized.

Collaborator

Cnly commented Nov 11, 2018

Thanks for sorting that out... I think I should also open an issue in the OneDrive API repo to get some clarification. In that case, it'd be the best if we have a Request ID. @gualdo0 I saw your earlier forum post but the Request ID part is removed. Do you mind providing some log with that information so we can open an issue here (https://github.com/OneDrive/onedrive-api-docs/issues)?

@ncw

This comment has been minimized.

Owner

ncw commented Nov 11, 2018

...can we move discussion of the listing number problem back into #2707 where it belongs?

ncw added a commit that referenced this issue Nov 11, 2018

@gualdo0

This comment has been minimized.

gualdo0 commented Nov 12, 2018

@Cnly sorry what is the Request ID? If you mean the post in the rclone forum, the link is
https://forum.rclone.org/t/error-reading-destination-directory/7316
so is the request ID #7316?

@ncw ncw modified the milestones: v1.45, v1.46 Nov 24, 2018

poundifdef added a commit to poundifdef/rclone that referenced this issue Nov 27, 2018

@Cnly

This comment has been minimized.

Collaborator

Cnly commented Nov 28, 2018

Sorry for the delay! I looked at the forum post again and it seems @gualdo0 you had sent your log privately to @ncw. If @gualdo0 you agree, I'd like to ask @ncw to help provide a snippet of your original log containing the Request ID.

@gualdo0

This comment has been minimized.

gualdo0 commented Nov 28, 2018

@ncw

This comment has been minimized.

Owner

ncw commented Nov 28, 2018

I've forwarded the log files to @Cnly via email :-)

@Cnly

This comment has been minimized.

Collaborator

Cnly commented Nov 28, 2018

Thanks! I've opened an upstream issue.

@Cnly

This comment has been minimized.

Collaborator

Cnly commented Dec 14, 2018

@gualdo0 There's an update from the upstream. I see in your previous log that the error happens multiple times. But does it still happen now? If it does, can you reproduce that and share your logs again? Some more recent logs may help solve this better. If you don't want to post the logs publicly, you can also send them to me at minecnly@gmail.com. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment