Onedrive: Error on writing data in serve mode with webdav #3131
Labels
Milestone
Comments
|
That looks like a bug! In the webdav serving or the onedrive backend I'm not sure. Can you run the |
|
Got now a log. First, before I post it: Is it necessary, anonymize anything? I think not because I see there no oauth2 credentials in the log file. |
|
@prodigy7 By default rclone replaces the auth header in your requests with Some other things that you might want to redact are your organization name and your email address that may be returned by the server. If you tried listing files, there may also be some direct download links (which should have expired given the time passed). And there will be your Drive ID, etc. which just look like some random strings and should be safe. |
|
I think, this last lines of the logfile should contain the related part which shows the error: It looks like rclone get/use not a valid upload handle? |
|
Thanks! I took a look at your logs. My guess is that you're hitting some random bugs on the remote side. Can you retry these two actions with rclone v1.47? In that version a workaround was added which has been able to mitigate some similar problems. If you're still seeing them, feel free to post some new logs. :) |
|
No problem, updated to v1.47 and start a new try. Will report! :-) Thanks until then! |
|
Howdy! Next error appears fast then expected ;-) Here the new log: |
|
@prodigy7 Thanks. Does this happen if you don't use WebDAV and instead just |
|
@Cnly Unfortunately I can't reproduce the scenario directly if I don't use the WebDav frontend. The backup software I can use on my NAS does only support webdav as "official frontend". What I do is only schedule the backup software and it runs on specific time. I don't rename/change or do anything else on onedrive storage. |
This looks like the problem transaction... Can you post more of the headers for the REQUEST? In particular Content-Range (or Range) and Content-Length |
|
Does this contains enough informations? |
|
So what it looks like happened is that we tried to put a 10MB chunk and got a 500 error for some reason. However when we retried it we got a "416 Requested Range Not Satisfiable" which probably means the server got the chunk already. This is covered in the docs
So we could assume if we get a 416 on a retry that the server got it already and to skip the chunk. Perhaps better would be to get the upload status to work out where we are. Our hands are somewhat tied here - if the server is missing stuff that isn't in the current chunk we'll have to abort. What do you think @Cnly ? |
|
Just to let you know, there are more people having this problem: I try to upload around 200GB in files of around of 8GB to OneDrive(Private) and with somes files I get the fragmentOverlap Errors. I do not use webdav, but do a direct copy including remote encryption. Thanks. |
ncw
added a commit
that referenced
this issue
Jan 25, 2020
Before this change, when uploading multipart files, onedrive would sometimes return an unexpected 416 error and rclone would abort the transfer. This is usually after a 500 error which caused rclone to do a retry. This change checks the upload position on a 416 error and if the segment has already been uploaded correctly (the most likely scenario) it will skip it and go onto the next one. On any other case rclone will abort the transfer. See: https://forum.rclone.org/t/fragment-overlap-error-with-encrypted-onedrive/14001 Fixes #3131
|
I had a go at fixing this properly - if you could run the transfer in debug (-vv) and watch out for https://beta.rclone.org/branch/v1.50.2-191-gca8bd807-fix-3131-onedrive-416-beta/ (uploaded in 15-30 mins) |
ncw
added a commit
that referenced
this issue
Jan 27, 2020
Before this change, when uploading multipart files, onedrive would sometimes return an unexpected 416 error and rclone would abort the transfer. This is usually after a 500 error which caused rclone to do a retry. This change checks the upload position on a 416 error and if the segment has already been uploaded correctly (the most likely scenario) it will skip it and go onto the next one. On any other case rclone will abort the transfer. See: https://forum.rclone.org/t/fragment-overlap-error-with-encrypted-onedrive/14001 Fixes #3131
|
That had a bug in, here is v2 https://beta.rclone.org/branch/v1.50.2-195-gd4091ccf-fix-3131-onedrive-416-beta/ (uploaded in 15-30 mins) |
ncw
added a commit
that referenced
this issue
Jan 28, 2020
Before this change, when uploading multipart files, onedrive would sometimes return an unexpected 416 error and rclone would abort the transfer. This is usually after a 500 error which caused rclone to do a retry. This change checks the upload position on a 416 error and works how much of the current chunk to skip, then retries (or skips) the current chunk as appropriate. If the position is before the current chunk or after the current chunk then rclone will abort the transfer. See: https://forum.rclone.org/t/fragment-overlap-error-with-encrypted-onedrive/14001 Fixes #3131
ncw
added a commit
that referenced
this issue
Jan 29, 2020
Before this change, when uploading multipart files, onedrive would sometimes return an unexpected 416 error and rclone would abort the transfer. This is usually after a 500 error which caused rclone to do a retry. This change checks the upload position on a 416 error and works how much of the current chunk to skip, then retries (or skips) the current chunk as appropriate. If the position is before the current chunk or after the current chunk then rclone will abort the transfer. See: https://forum.rclone.org/t/fragment-overlap-error-with-encrypted-onedrive/14001 Fixes #3131
ncw
added a commit
that referenced
this issue
Jan 31, 2020
Before this change, when uploading multipart files, onedrive would sometimes return an unexpected 416 error and rclone would abort the transfer. This is usually after a 500 error which caused rclone to do a retry. This change checks the upload position on a 416 error and works how much of the current chunk to skip, then retries (or skips) the current chunk as appropriate. If the position is before the current chunk or after the current chunk then rclone will abort the transfer. See: https://forum.rclone.org/t/fragment-overlap-error-with-encrypted-onedrive/14001 Fixes #3131
|
@ncw, I just wanted to thank you so much for your great work! You make this world a better place! rclone is absolutely great. And I am so sorry, that I did not yet have the time to test your release. At that time I got the error, I was only testing rclone and deleted everything as onedrive was to unreliable. IBut I plan to recreate the setup and test in the next days. Thanks again for your creat work and support! |
|
I've had a willing tester for this patch, and I'm confident it is fixed now. It missed 1.50.0 but I'll release it in 1.50.1 at some point. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What is the problem you are having with rclone?
Because my NAS (Synology) does not support OneDrive for storing backups (HyperBackup), i want use rclone as a "proxy" for storing backup data because the backup software does support webdav.
For that I start serve webdav with:
So far, i works for a time but then I got a error from rclone:
What is your rclone version (output from
rclone version)rclone v1.46
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Synology OS (Linux 4.4.59+), but I can reproduce this at Debian 9/x64.
Which cloud storage system are you using? (eg Google Drive)
Onerive
The command you were trying to run (eg
rclone copy /tmp remote:tmp)rclone serve webdav --addr 127.0.0.1:8090 onedrive:
A log from the command with the
-vvflag (eg output fromrclone -vv copy /tmp remote:tmp)The text was updated successfully, but these errors were encountered: