Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rclone copy --files-from tag takes some time before staring the migration #3248

Closed
manishgupta24 opened this issue Jun 7, 2019 · 6 comments

Comments

Projects
None yet
2 participants
@manishgupta24
Copy link

commented Jun 7, 2019

Hi

I am trying to upload a folder using copy command and I am passing a list of files using --files-from tag. If I use copy without this tag the migration starts immediately. But when using this tag, Rclone waits for some time before the files start copying to destination.

The wait time depends on the number of files I am trying to migrate with the --files-from tag, from few seconds to more than 15 minutes and above.

I tested it with a folder containing around 5k files and the initial wait time was around 45 minutes.

I looked into the code and this might be happening because Rclone creates a DirTree for the list of files provided with the tag.

rclone/fs/walk/walk.go

Lines 714 to 715 in a28239f

// If --files-from and --no-traverse is set then a DirTree will be
// constructed with just those files in.

My question is why does this take more time as compared to the Folder Walk using os.Walk, since we are passing the absolute path. Is there any way to shorten this wait time or better bypass this?

@ncw

This comment has been minimized.

Copy link
Owner

commented Jun 9, 2019

Which backend are you using? Google drive is particularly slow with --files-from.

Which version of rclone are you using?

With --files-from rclone finds all the files first before starting the transfers as you've noted.

In the latest beta I changed the algorithm so that it will now traverse directories to find the files from list, unless you use --no-traverse when it will use the 1.47 behaviour.

So give the latest beta a go and see if that fixes your problem.

@manishgupta24

This comment has been minimized.

Copy link
Author

commented Jun 10, 2019

My Source is a local Windows Directory and My destination is Google Cloud Storage bucket.
I am using Rclone version 1.45.

In the latest beta I changed the algorithm so that it will now traverse directories to find the files from the list, unless you use --no-traverse when it will use the 1.47 behavior.

How is it different from not passing a list of files using --files-from tag? By default the rclone traverses the directory given as argument with copy command.

@ncw

This comment has been minimized.

Copy link
Owner

commented Jun 10, 2019

How is it different from not passing a list of files using --files-from tag? By default the rclone traverses the directory given as argument with copy command.

You still pass the list of files with --files-from, rclone v1.48 beta just converts that into a filter which turns out to be as efficient as trying to find each file individually (as v1.47 does).

@manishgupta24

This comment has been minimized.

Copy link
Author

commented Jun 14, 2019

rclone v1.48 beta just converts that into a filter ...

Hi, The latest version is rclone v1.47.0-128-g22368b99-beta according to version.txt on Rclone beta. Where can I find the version 1.48 beta ?

@ncw

This comment has been minimized.

Copy link
Owner

commented Jun 14, 2019

rclone v1.48 beta just converts that into a filter ...

Hi, The latest version is rclone v1.47.0-128-g22368b99-beta according to version.txt on Rclone beta. Where can I find the version 1.48 beta ?

That is the 1.48 beta it is 1.47 + a lot but isn't 1.48 yet!

@manishgupta24

This comment has been minimized.

Copy link
Author

commented Jul 8, 2019

Thanks It worked.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.