Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failure copying large file to OneDrive for Business #3229

Open
iJaffa opened this issue May 28, 2019 · 5 comments

Comments

@iJaffa
Copy link

@iJaffa iJaffa commented May 28, 2019

What is the problem you are having with rclone?

Intermittently unable to copy large files (4GB+) to OneDrive for Business

What is your rclone version (output from rclone version)

First noticed on rClone v1.45.
Able to replicate on 1.47 beta as of 28 May 2019.

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 19.04
64-bit

Which cloud storage system are you using? (eg Google Drive)

OneDrive for Business

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount onedrive:remote local --allow-non-empty --allow-other --vfs-read-chunk-size 10M --vfs-cache-mode writes

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

2019/05/28 12:10:30 ERROR : filepath/filename.partial~: Failed to copy: serviceNotAvailable: Service unavailable
2019/05/28 12:10:30 ERROR : filepath/filename.partial~(0xc42048f2c0): failed to transfer file from cache to remote: serviceNotAvailable: Service unavailable
2019/05/28 12:10:30 ERROR : filepath/filename.partial~(0xc42048f2c0): RWFileHandle.Flush error: failed to transfer file from cache to remote: serviceNotAvailable: Service unavailable
2019/05/28 12:10:51 ERROR : Media/TV Shows/: Dir.Stat error: couldn't list files: serviceNotAvailable: Service unavailable
2019/05/28 12:11:14 ERROR : Media/TV Shows/: Dir.Stat error: couldn't list files: serviceNotAvailable: Service unavailable
2019/05/28 12:13:49 NOTICE: filepath/filename.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
2019/05/28 12:14:57 NOTICE: filepath/filename.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
2019/05/28 12:15:03 ERROR : filepath/filename.partial~: Failed to copy: nameAlreadyExists: An upload session already exists for this file (is it a OneNote file?)
2019/05/28 12:15:03 ERROR : filepath/filename.partial~(0xc4202d3c80): failed to transfer file from cache to remote: nameAlreadyExists: An upload session already exists for this file (is it a OneNote file?)
2019/05/28 12:15:03 ERROR : filepath/filename.partial~(0xc4202d3c80): RWFileHandle.Flush error: failed to transfer file from cache to remote: nameAlreadyExists: An upload session already exists for this file (is it a OneNote file?)

@ncw

This comment has been minimized.

Copy link
Collaborator

@ncw ncw commented May 28, 2019

serviceNotAvailable: Service unavailable looks like the remote endpoint just gave up the ghost...

Any ideas @Cnly ?

@Cnly

This comment has been minimized.

Copy link
Collaborator

@Cnly Cnly commented May 31, 2019

Yeah, this looks like the remote's fault. @iJaffa are you able to reproduce the problem on only a specific file, or any files? Recently I uploaded more than 20 4GB+ files to OneDrive for Business in a row and although there were errors, they're ultimately handled by rclone's retries.

@iJaffa

This comment has been minimized.

Copy link
Author

@iJaffa iJaffa commented May 31, 2019

It seems to occur to any files. Like you have found, retrying seems to be the answer. Since rclone mount does 3 retries by default I would have thought this would be fine, but it appears to not be on some occasions.

@iJaffa

This comment has been minimized.

Copy link
Author

@iJaffa iJaffa commented Jun 4, 2019

I have been able to reproduce the issue again. journalctl code is below. Using Fedora 30 (64-bit) this time with the latest stable rclone.

rclone v1.47.0
- os/arch: linux/amd64
- go version: go1.12.1

The rclone mount command is the same as before, and the file size is reported as 5.0GB (5264328263) in Fedora.

Jun 04 12:38:26 rclone[728]: 2019/06/04 12:38:26 ERROR : One drive root '': Statfs failed: about failed: serviceNotAvailable: Service unavailable
Jun 04 12:38:45 rclone[728]: 2019/06/04 12:38:45 ERROR : filepath/: Dir.Stat error: couldn't list files: serviceNotAvailable: Service unavailable
Jun 04 12:39:05 rclone[728]: 2019/06/04 12:39:05 ERROR : filepath/: Dir.Stat error: couldn't list files: serviceNotAvailable: Service unavailable
Jun 04 12:39:28 rclone[728]: 2019/06/04 12:39:28 ERROR : filepath/: Dir.Stat error: couldn't list files: serviceNotAvailable: Service unavailable
Jun 04 12:39:48 rclone[728]: 2019/06/04 12:39:48 ERROR : filepath/: Dir.Stat error: couldn't list files: serviceNotAvailable: Service unavailable
Jun 04 12:40:09 rclone[728]: 2019/06/04 12:40:09 ERROR : filepath/: Dir.Stat error: couldn't list files: serviceNotAvailable: Service unavailable
Jun 04 12:40:37 rclone[728]: 2019/06/04 12:40:37 ERROR : filepath/: Dir.Stat error: couldn't list files: serviceNotAvailable: Service unavailable
Jun 04 12:40:57 rclone[728]: 2019/06/04 12:40:57 ERROR : filepath/: Dir.Stat error: couldn't list files: serviceNotAvailable: Service unavailable
Jun 04 12:41:17 rclone[728]: 2019/06/04 12:41:17 ERROR : filepath/: Dir.Stat error: couldn't list files: serviceNotAvailable: Service unavailable
Jun 04 12:41:43 rclone[728]: 2019/06/04 12:41:43 ERROR : filepath/: Dir.Stat error: couldn't list files: serviceNotAvailable: Service unavailable
Jun 04 12:44:26 rclone[728]: 2019/06/04 12:44:26 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 12:47:17 rclone[728]: 2019/06/04 12:47:17 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 12:47:58 rclone[728]: 2019/06/04 12:47:58 ERROR : filepath/filepath.partial~: Failed to copy: serviceNotAvailable: Service unavailable
Jun 04 12:47:58 rclone[728]: 2019/06/04 12:47:58 ERROR : filepath/filepath.partial~(0xc0003e4660): failed to transfer file from cache to remote: serviceNotAvailable: Service unavailable
Jun 04 12:47:58 rclone[728]: 2019/06/04 12:47:58 ERROR : filepath/filepath.partial~(0xc0003e4660): RWFileHandle.Flush error: failed to transfer file from cache to remote: serviceNotAvailable: Service unavailable
Jun 04 12:51:25 rclone[728]: 2019/06/04 12:51:25 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 12:55:01 rclone[728]: 2019/06/04 12:55:01 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 12:56:17 rclone[728]: 2019/06/04 12:56:17 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 12:56:37 rclone[728]: 2019/06/04 12:56:37 ERROR : filepath/filepath.partial~: Failed to copy: serviceNotAvailable: Service unavailable
Jun 04 12:56:37 rclone[728]: 2019/06/04 12:56:37 ERROR : filepath/filepath.partial~(0xc0006d1a40): failed to transfer file from cache to remote: serviceNotAvailable: Service unavailable
Jun 04 12:56:37 rclone[728]: 2019/06/04 12:56:37 ERROR : filepath/filepath.partial~(0xc0006d1a40): RWFileHandle.Flush error: failed to transfer file from cache to remote: serviceNotAvailable: Service unavailable
Jun 04 12:57:23 rclone[728]: 2019/06/04 12:57:23 ERROR : filepath/: Dir.Stat error: couldn't list files: serviceNotAvailable: Service unavailable
Jun 04 13:02:42 rclone[728]: 2019/06/04 13:02:42 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 13:05:01 rclone[728]: 2019/06/04 13:05:01 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 13:05:49 rclone[728]: 2019/06/04 13:05:49 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 13:11:41 rclone[728]: 2019/06/04 13:11:41 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 13:18:05 rclone[728]: 2019/06/04 13:18:05 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 13:18:05 rclone[728]: 2019/06/04 13:18:05 ERROR : filepath/filepath.partial~: Failed to copy: serviceNotAvailable: Service unavailable
Jun 04 13:18:05 rclone[728]: 2019/06/04 13:18:05 ERROR : filepath/filepath.partial~(0xc0021072c0): failed to transfer file from cache to remote: serviceNotAvailable: Service unavailable
Jun 04 13:18:05 rclone[728]: 2019/06/04 13:18:05 ERROR : filepath/filepath.partial~(0xc0021072c0): RWFileHandle.Flush error: failed to transfer file from cache to remote: serviceNotAvailable: Service unavailable
Jun 04 13:28:46 rclone[728]: 2019/06/04 13:28:46 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 13:32:10 rclone[728]: 2019/06/04 13:32:10 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 13:42:44 rclone[728]: 2019/06/04 13:42:44 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 13:46:11 rclone[728]: 2019/06/04 13:46:11 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 13:46:32 rclone[728]: 2019/06/04 13:46:32 ERROR : filepath/filepath.partial~: Failed to copy: serviceNotAvailable: Service unavailable
Jun 04 13:46:32 rclone[728]: 2019/06/04 13:46:32 ERROR : filepath/filepath.partial~(0xc0003e42a0): failed to transfer file from cache to remote: serviceNotAvailable: Service unavailable
Jun 04 13:46:32 rclone[728]: 2019/06/04 13:46:32 ERROR : filepath/filepath.partial~(0xc0003e42a0): RWFileHandle.Flush error: failed to transfer file from cache to remote: serviceNotAvailable: Service unavailable
Jun 04 13:52:15 rclone[728]: 2019/06/04 13:52:15 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 13:55:54 rclone[728]: 2019/06/04 13:55:54 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 13:57:37 rclone[728]: 2019/06/04 13:57:37 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 13:58:40 rclone[728]: 2019/06/04 13:58:40 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 13:58:40 rclone[728]: 2019/06/04 13:58:40 ERROR : filepath/filepath.partial~: Failed to copy: serviceNotAvailable: Service unavailable
Jun 04 13:58:40 rclone[728]: 2019/06/04 13:58:40 ERROR : filepath/filepath.partial~(0xc002b49680): failed to transfer file from cache to remote: serviceNotAvailable: Service unavailable
Jun 04 13:58:40 rclone[728]: 2019/06/04 13:58:40 ERROR : filepath/filepath.partial~(0xc002b49680): RWFileHandle.Flush error: failed to transfer file from cache to remote: serviceNotAvailable: Service unavailable
Jun 04 14:02:10 rclone[728]: 2019/06/04 14:02:10 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 14:06:59 rclone[728]: 2019/06/04 14:06:59 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 14:08:53 rclone[728]: 2019/06/04 14:08:53 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 14:08:53 rclone[728]: 2019/06/04 14:08:53 ERROR : filepath/filepath.partial~: Failed to copy: serviceNotAvailable: Service unavailable
Jun 04 14:08:53 rclone[728]: 2019/06/04 14:08:53 ERROR : filepath/filepath.partial~(0xc0021072c0): failed to transfer file from cache to remote: serviceNotAvailable: Service unavailable
Jun 04 14:08:53 rclone[728]: 2019/06/04 14:08:53 ERROR : filepath/filepath.partial~(0xc0021072c0): RWFileHandle.Flush error: failed to transfer file from cache to remote: serviceNotAvailable: Service unavailable
Jun 04 14:09:17 rclone[728]: 2019/06/04 14:09:17 ERROR : filepath/: Dir.Stat error: couldn't list files: serviceNotAvailable: Service unavailable
Jun 04 14:09:35 rclone[728]: 2019/06/04 14:09:35 ERROR : filepath/: Dir.Stat error: couldn't list files: serviceNotAvailable: Service unavailable
Jun 04 14:16:40 rclone[728]: 2019/06/04 14:16:40 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 14:26:44 rclone[728]: 2019/06/04 14:26:44 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 14:27:49 rclone[728]: 2019/06/04 14:27:49 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 14:28:09 rclone[728]: 2019/06/04 14:28:09 ERROR : filepath/filepath.partial~: Failed to copy: serviceNotAvailable: Service unavailable
Jun 04 14:28:09 rclone[728]: 2019/06/04 14:28:09 ERROR : filepath/filepath.partial~(0xc000201440): failed to transfer file from cache to remote: serviceNotAvailable: Service unavailable
Jun 04 14:28:09 rclone[728]: 2019/06/04 14:28:09 ERROR : filepath/filepath.partial~(0xc000201440): RWFileHandle.Flush error: failed to transfer file from cache to remote: serviceNotAvailable: Service unavailable
Jun 04 14:35:44 rclone[728]: 2019/06/04 14:35:44 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 14:44:00 rclone[728]: 2019/06/04 14:44:00 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 14:44:45 rclone[728]: 2019/06/04 14:44:45 NOTICE: filepath/filepath.partial~: Failed to cancel multipart upload: serviceNotAvailable: Service unavailable
Jun 04 14:46:26 rclone[728]: 2019/06/04 14:46:26 ERROR : filepath/filepath.partial~: Failed to copy: serviceNotAvailable: Service unavailable
Jun 04 14:46:26 rclone[728]: 2019/06/04 14:46:26 ERROR : filepath/filepath.partial~(0xc004fa35c0): failed to transfer file from cache to remote: serviceNotAvailable: Service unavailable
Jun 04 14:46:26 rclone[728]: 2019/06/04 14:46:26 ERROR : filepath/filepath.partial~(0xc004fa35c0): RWFileHandle.Flush error: failed to transfer file from cache to remote: serviceNotAvailable: Service unavailable
Jun 04 14:46:49 rclone[728]: 2019/06/04 14:46:49 ERROR : filepath/: Dir.Stat error: couldn't list files: serviceNotAvailable: Service unavailable
Jun 04 14:51:39 rclone[728]: 2019/06/04 14:51:39 ERROR : filepath/filepath.partial~: Failed to copy: serviceNotAvailable: Service unavailable
Jun 04 14:51:39 rclone[728]: 2019/06/04 14:51:39 ERROR : filepath/filepath.partial~(0xc004ea04e0): failed to transfer file from cache to remote: serviceNotAvailable: Service unavailable
Jun 04 14:51:39 rclone[728]: 2019/06/04 14:51:39 ERROR : filepath/filepath.partial~(0xc004ea04e0): RWFileHandle.Flush error: failed to transfer file from cache to remote: serviceNotAvailable: Service unavailable
@iJaffa

This comment has been minimized.

Copy link
Author

@iJaffa iJaffa commented Jun 18, 2019

It seems Tuesdays might be the day where the most issues occur with OneDrive for Business. Got this just now:

Jun 18 11:10:16 rclone[20026]: 2019/06/18 11:10:16 ERROR : filepath/: Dir.Stat error: couldn't list files: serviceNotAvailable: Service unavailable
Jun 18 11:15:17 rclone[20026]: 2019/06/18 11:15:17 ERROR : filepath/: Dir.Stat error: couldn't list files: serviceNotAvailable: Service unavailable
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.