Rclone sync from local to GoogleDrive hanging at 100%

[DurvalMenezes]

What is the problem you are having with rclone?

rclone sync from local to GoogleDrive hanging at 100%: everything runs perfectly in the beginning of the sync, but sometimes, when it reaches 100% it just hangs there for hours and hours. The only way out is killing the process and then running it again with exactly the same command; so far the second execution always finishes OK (it's a very intermittent error).

PS: this is also being discussed on this forum topic: https://forum.rclone.org/t/rclone-sync-from-local-to-googledrive-hanging-at-100/18622

What is your rclone version (output from rclone version)

rclone v1.50.2
os/arch: darwin/amd64
go version: go1.13.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Mac OSX High Sierra

Which cloud storage system are you using? (eg Google Drive)

Google Drive, with a legit (ie, >5 users) GSuite Unlimited account

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -v --checkers=12 --transfers=8 --multi-thread-cutoff=1k
--multi-thread-streams=8 --low-level-retries=1000 --user-agent REDACTED
--rc --bwlimit "09:00,4m 21:00,off sat-09:01,off sun-09:01,off"
--fast-list sync /REDACTED/REDACTED/REDACTED GDRIVE_REMOTE:REMOTEDIR

The rclone config contents with secrets removed.

[GDRIVE_REMOTE]
type = drive
client_id = REDACTED.apps.googleusercontent.com
client_secret = REDACTED
service_account_file =
token = {"access_token":"REDACTED","token_type":"Bearer","refresh_token":"1/REDACTED","expiry":"2020-08-20T11:00:27.698728-03:00"}
root_folder_id = REDACTED

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

rclone_-vv_100percent_hang_bug.out.txt

[maggie44]

Looks like it is under a lot of investigation in the forum thread already, but here are some system details where I am experiencing the same issues in the latest versions just in case of any use.

What is your rclone version (output from rclone version):

rclone v1.53.1

• os/arch: linux/amd64
• go version: go1.15

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20.04.1

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync /dir host:dir/ --exclude=folder1/** --create-empty-src-dirs --config=/home/rclone.conf -v

[ncw]

@Maggie0002 can you do a run with -vv and attach the log file to the issue please? I'm not sure that is the same issue.

[maggie44]

It occurred on two systems, a Raspberry Pi running Buster, and the Ubuntu server. Different content being synced to different parts of the same GDrive.

Both rclone v1.53.1

Both are syncing content that is being used by processes, so may be changes to files during sync.

Issue has been intermittent, both systems created the issue the first time it was run (perhaps because it had a lot of uploading to do and took longer). The Raspberry Pi with less to sync sometimes passes through. The larger Ubuntu system sync is yet to exit cleanly, although as far as I can tell all the content is synced (at least in terms of it is about the right folder size).

Issue looked exactly the same as the logs attached by DurvalMenezes. When I woke up in the morning, it had been sitting there for 7 hours longer than it should, echoing the same transfer report like the one seen in the bottom of DurvalMenezes' logs and without any progress being made.

Will run it on the Ubuntu system where the error is most likely to occur and try and get a log of it hanging. The process takes a long time though, even before the error occurs so could be a while.

[DurvalMenezes]

Hello @ncw,

Just keeping you posted: since I upgraded to v1.52.3, it's been 28 days already (and 20 backups) and the bug hasn't manifested itself yet. At least for my case, things look promising.

Regarding @Maggie0002's situation: perhaps it's a regression on v1.53.x? I don't think it's very probable, but if you can easily downgrade to v1.52.3 and give it a try, I would suggest you do so (contingent on @ncw's recommendation after examining your v1.53.1 -vv log, of course).

Cheers,
-- Durval.

[maggie44]

Thanks for the input @ncw and @DurvalMenezes ,

The sync finished this time on the Ubuntu system. By activating -vv for logging I see some new messages that have identified the problem for me. For around 6 minutes there was no movement on the status, it was stuck on:

Transferred:   	   75.010M / 75.010 MBytes, 100%, 912.203 kBytes/s, ETA 0s
Checks:             60216 / 60216, 100%
Transferred:           44 / 44, 100%
Elapsed time:   3h26m16.8s

But a bunch of messages exist in-between each as a result of the -vv, around 25 a minute:

2020/09/28 13:51:41 DEBUG : Archive/local/2ea30f5056f9ba9asd2039917b06f28f22: Making directory

Presumably this is what was happening when I ran the sync on my two systems yesterday. They were both syncing for the first time and therefore likely doing a lot more of this than it is today. The Ubuntu system took an exceptionally long time to do all these if that is what is going on, but without the logs from that issue I can only assume this is the case.

Is this related to the --create-empty-src-dirs flag I was running or is this some other sync related task?

Either way, perhaps some indicator in the progress (-P) output would be helpful for those running very large syncs to inform that it is still doing its thing effectively?

[DurvalMenezes]

@Maggie0002, thanks for reporting back. So, rclone in your case was not really hanging, but rather creating a rather deep/large directory hierarchy, right?

perhaps some indicator in the progress (-P) output would be helpful for those running very large syncs to inform that it is still doing its thing effectively?

Like, adding a "Directories created: " line to the progress (-P) output?

Cheers,
-- Durval.

[maggie44]

That appears to be the case. Not sure why all the directory creations are left until the end, but yes.

A "Directories created: " would have given me a number I could see rising, so yes would have avoided me falsely assuming a hang. At the least, perhaps moving the 'Making directory' notifications out of the DEBUG level and into INFO.

[DurvalMenezes]

Folks, rclone v1.52.3 went through a full month without any more hangs, so I'm considering this solved by the upgrade from v.1.50.2 (as @animosity22 predicted right at the start -- but in my defense, I must say I went through the changelog at the time, and saw no mention of such a fix, which is clear now was a 'welcome side effect' from other changes done by @ncw.

Anyway, I'm marking this as closed -- thanks @ncw and everyone who chimed in.