Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rclone Keeps Failing to Copy File with "corrupted on transfer" (Google Drive) #981

Closed
reubendb opened this issue Jan 3, 2017 · 20 comments

Comments

Projects
None yet
8 participants
@reubendb
Copy link

commented Jan 3, 2017

What is your rclone version (eg output from rclone -V)

rclone 1.35

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux AMD64

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -v copy "GDrive:/Google Photos/VID_20161129_141853510.mp4" /tmp/

A log from the command with the -v flag (eg output from rclone -v copy /tmp remote:tmp)

2017/01/02 21:41:47 rclone: Version "v1.35" starting with parameters ["rclone" "-v" "copy" "GDrive:/Google Photos/VID_20161129_141853510.mp4" "/tmp/"]
2017/01/02 21:41:47 Local file system at /tmp: Modify window is 1ms
2017/01/02 21:41:48 Local file system at /tmp: Waiting for checks to finish
2017/01/02 21:41:48 Local file system at /tmp: Waiting for transfers to finish
2017/01/02 21:41:53 VID_20161129_141853510.mp4: corrupted on transfer: MD5 hash differ "53a0e5a5db26301f97d390a51155cae7" vs "cd6c3d2dc4e91101b5000357dfd96e73"
2017/01/02 21:41:53 VID_20161129_141853510.mp4: Removing failed copy
2017/01/02 21:41:53 Attempt 1/3 failed with 1 errors and: corrupted on transfer: MD5 hash differ "53a0e5a5db26301f97d390a51155cae7" vs "cd6c3d2dc4e91101b5000357dfd96e73"

(Two more attempts failed with the same error)

Hello,
First of all, thank you for this wonderful program.

I am trying to download all of my files in Google Photos to my local machine using 'rclone copy', with the destination is a completely empty (local) directory initially. Out of about ~3500 files, 90 of them says that they are "corrupted on transfer: MD5 hash differ". These are mostly .mp4 files, but there are also a handful of JPEG files.

I then tried to manually download a few of the corrupted files one at a time (as shown in the form above) and I kept getting the same error. I checked the using the web interface and there is no duplicate of the file.

I then downloaded the file using the web interface and manually run md5sum on it and got: cd6c3d2dc4e91101b5000357dfd96e73
I then also tried using [https://github.com/prasmussen/gdrive] to download the entire 'Google Photos' folder and was able to do so. The md5sum of the downloaded file (via gdrive) is also the same as the version downloaded via web interface.

What can I do to help troubleshoot this issue? It is also curious that from multiple attempts rclone always end up with the same corrupted MD5 hash.

The files in my Google Photos are mostly from automatic upload from an Android phone and I wanted to use rclone as an automated way to also keep a backup of those on my local machine.

Thanks.

@ncw

This comment has been minimized.

Copy link
Owner

commented Jan 3, 2017

I just had a look at the code. The corrupted hashes are in order src, dst so from

2017/01/02 21:41:53 Attempt 1/3 failed with 1 errors and: corrupted on transfer: MD5 hash differ "53a0e5a5db26301f97d390a51155cae7" vs "cd6c3d2dc4e91101b5000357dfd96e73"

"53a0e5a5db26301f97d390a51155cae7" is the hash that google thinks it is

"cd6c3d2dc4e91101b5000357dfd96e73" is the hash that we get when downloading - which agrees the web interface.

So for some reason google has the checksum wrong on these objects.

--ignore-checksum which I'm implementing in #793 would help you here.

@reubendb

This comment has been minimized.

Copy link
Author

commented Jan 5, 2017

Thanks, I will give that a try once implemented. I wonder if the issue is on Google's side for not keeping the correct hash. #863 seems to have similar issue except the destination is remote, I wonder if it's related (a symptom of the same problem somehow).

@ncw

This comment has been minimized.

Copy link
Owner

commented Jan 5, 2017

#863 is the same problem.

I'll try to get #793 done soon!

@ncw

This comment has been minimized.

Copy link
Owner

commented Jan 5, 2017

Can you try this which implements the --ignore-checksum command?

http://pub.rclone.org/v1.35-15-g82742f5-ignore-checksum%CE%B2/

@ncw

This comment has been minimized.

Copy link
Owner

commented Feb 3, 2017

I've merged this to master for the 1.36 release

Here is the beta - http://beta.rclone.org/v1.35-62-g9d331ce/ (uploaded in 15-30 mins)

@ncw ncw added the enhancement label Feb 3, 2017

@ncw ncw added this to the v1.36 milestone Feb 3, 2017

@ncw ncw closed this in 9d331ce Feb 3, 2017

@BenoitDuffez

This comment has been minimized.

Copy link

commented May 10, 2017

Thanks for this workaround, it's awesome.
Would it be possible though to touch the files remotely, or do something that will make Google Drive recompute the hash?

@jtenniswood

This comment has been minimized.

Copy link

commented May 10, 2017

I've run into the same problem, tried the --ignore-checksum switch, it does make it upload but the file is corrupt when I try to download it again.

In my case, the same script and setup on two machines running the same OS and rclone release, one machine throws errors and the others work perfectly.

Not sure what to try next, any ideas?

@BenoitDuffez

This comment has been minimized.

Copy link

commented May 10, 2017

OK then, maybe it's worth reopening?
For me it's a download that is failing. Debian 8.
I also tried to change some metadata on the web interface of drive hoping this would change the remote checksum but it didn't.

@jtenniswood

This comment has been minimized.

Copy link

commented May 10, 2017

Sorry, turns out it was just me being incredibly dim, I had left a cron job running every minute to test this backup, but the new job overwrote the old backup, which rclone was trying to update, which caused the errors. Doh!

@BenoitDuffez

This comment has been minimized.

Copy link

commented May 15, 2017

Would it be possible to disable checksum on specific files?

@vitor-alves

This comment has been minimized.

Copy link

commented Aug 15, 2017

+1
Ignoring the checksum in specific folders/files would be perfect. In my machine it seems that only a single file is having this problem.

@djsmiley2k

This comment has been minimized.

Copy link

commented Mar 3, 2018

I'm seeing this problem, going to try the ignore checksum fix... However I've noticed for all the photos I've had it having an issue with, they seem to be duplicates which google has 'fiddled' with - i.e. it's added effects, animations, rotations etc.

Maybe google isn't recomputing the hashes after it makes these changes?

@ncw

This comment has been minimized.

Copy link
Owner

commented Mar 5, 2018

@djsmiley2k this is an old issue now! If you can reproduce the problem then open a new issue and I'll investigate - thanks!

@srd424

This comment has been minimized.

Copy link

commented Nov 10, 2018

I know this is closed, but just wanted to drop a comment in case this rings any bells with anyone..
I'm seeing this, but only on .MOV files. Makes me wonder if Google transcode these files to reduce quality or something, or to create multiple versions for different bandwidth, and then the API gets confused?

@srd424

This comment has been minimized.

Copy link

commented Nov 10, 2018

Suspicion confirmed: looking at one of these movies that raised a checksum error, it turns out my copy was one I had accidentally downloaded from someone else's shared album. I had previously backed up that shared album using the "download all" feature in Photos. The version from the previous backup had a much higher bitrate, and a checksum that matched the one that rclone was expecting. So something at Google is transcoding movies (fair enough) but then getting confused about the hash, or exposing the wrong one, or something..

@djsmiley2k

This comment has been minimized.

Copy link

commented Nov 10, 2018

@pongnguy

This comment has been minimized.

Copy link

commented Nov 26, 2018

I recently downloaded rclone and am very new to it. In running some test cases, I successfully copied a local folder to Amazon S3 using the --ignore-checksum flag. However I am getting "Failed to copy: corrupted on transfer: MD5 crypted hash differ" errors when I try to copy the same files to a crypt remote. Not sure what else to do!

@ncw

This comment has been minimized.

Copy link
Owner

commented Nov 26, 2018

I recently downloaded rclone and am very new to it. In running some test cases, I successfully copied a local folder to Amazon S3 using the --ignore-checksum flag. However I am getting "Failed to copy: corrupted on transfer: MD5 crypted hash differ" errors when I try to copy the same files to a crypt remote. Not sure what else to do!

Can you post the command line for which you are getting the error please?

@pongnguy

This comment has been minimized.

Copy link

commented Nov 27, 2018

Sure. I am trying to upload from a local directory (i.e. 'Vacation Pictures') to an encrypted remote (i.e. amazon_s3_crypt). The encrypted remote has a ":" in its location (i.e. I made a separate bucket for the encrypted files).

Using this command:

rclone sync ./'Vacation Pictures'/ "amazon_s3_crypt:Vacation Pictures/" --ignore-checksum

Here is a section from the output of rclone config for reference:

Name Type
==== ====
amazon_s3_crypt crypt
amazon_s3_dropbox_personal s3

@ncw

This comment has been minimized.

Copy link
Owner

commented Nov 27, 2018

@pongnguy thanks. I think this is a different issue to this one. What you are doing shouldn't require --ignore-checksum - why are you using that?

The message Failed to copy: corrupted on transfer: MD5 crypted hash differ is from the crypt layer stating that the hash for the upload wasn't what it expected.

It would probably be best if you please make a new issue on github - can you fill out the issue template details with rclone version, os version, logs etc and make a way for me to reproduce the issue ideally - thanks :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.