Feature request: Number of files in queue at core/stats

[alexgit2k]

What is your current rclone version (output from rclone version)?

1.57.0

What problem are you are trying to solve?

I want to know if the sync-job has finished transfering files / how many files still have to be transfered.

How do you think rclone should be changed to solve that?

core/stats (https://rclone.org/rc/#core-stats) shows the currently active file transfers at "transferring", but with the default value of 4 for parallel transfers this shows only the 4 active transfers, but not those which are currently waiting.

I suggets a new parameter like "queue" or "files in queue" which shows the files currently in the queue.

[animosity022]

Hmm, neat idea, but not sure how that works as rclone normally doesn't 'know' how many files are going to be transferred unless you specify a few things to get it all up front to get the full list.

What's the use case for the information and how would it be used?

[alexgit2k]

When starting a sync with rclone at the comand line, rclone successfully exits although the files are not transfered yet.

I want to know if the transfer is finished, so that I can unmount or shutdown the computer and be sure, that the files are successfully backuped.

I would use it in a small user-interface which polls core/stats and shows if there is still activity.

If the number of files is not available, ist there any other information which can be used to check if the files have been transfered?

[animosity022]

How are you starting it? What is the full command you are using and what is the full command you are using to check the completion?

[alexgit2k]

I'm not at my computer until tomorrow, but what I have in my mind is:
rclone mount --vfs-cache-mode=writes --vfs-cache-max-size=100M remote:path/ drive1:
rclone sync --create-empty-dirs drive2:/path drive1:/path

To check I'm polling http://localhost:5572/core/stats

[animosity022]

I'm not sure how the mount comes into play?

Normally, you'd submit a job via the API / remote control commands and just check the status of the job.

If you are running a script/one line command, why poll at all and just watch the job finish.

https://rclone.org/rc/#job-status

I think what you want already exists as it's just a use case question on how to do it.

[alexgit2k]

The mount is needed to mount the encrypted remote cloud-drive as local drive.

Submitting the sync-job via API is a good idea, will try that. Thanks!

If you are running a script/one line command, why poll at all and just watch the job finish.

That does not work, because: When starting a sync with rclone at the comand line, rclone successfully exits although the files are not transfered yet.

[animosity022]

The mount is needed to mount the encrypted remote cloud-drive as local drive.

You don't need a mount to copy from one remote to another.

rclone sync --create-empty-dirs drive2:/path drive1:/path

Is that not the command you are running or something else?

That does not work, because: When starting a sync with rclone at the comand line, rclone successfully exits although the files are not transfered yet.

That's just wrong based on the command you've shared.

Best to take a step back, share full commands and logs if you want figure out what's going on.

[alexgit2k]

I'm syncing a local drive to a remote one. The remote one is mounted, to use vfs-cache-mode=writes. Thats probably the reason, why the sync command exits before files are transfered.

Will share full commands tomorrow.

[animosity022]

There's no reason to mount and sync it.

Just sync remote to remote.

The mount is unneeded overhead.

[alexgit2k]

Thank you very much for your feedback!

As mentioned I was mounting the remote drive to a local drive and then syncing it to this local drive. Therefore rclone successfully exits although the files are not transfered yet, because they were transfered in the background via the cache.

If syncing directly to the remote, it only exits after the sync has been completely executed:

rclone sync --create-empty-src-dirs localdrive:localpath remote:remotepath

or the via API:

rclone --rc-no-auth rcd
rclone rc sync/sync srcFs=localdrive:localpath dstFs=remote:remotepath createEmptySrcDirs=true

or asynchronously via API:

rclone --rc-no-auth rcd
rclone rc sync/sync srcFs=localdrive:localpath dstFs=remote:remotepath createEmptySrcDirs=true _async=true
rclone rc job/status jobid=1

So now I know, when the sync has been successfully completed, but not how much files still have to be transfered. But when using the group-parameter and then core/stats with that group, the remaining files which are scheduled to be transfered can be calculated by substracting "transfers" from "totalTransfers":

rclone --rc-no-auth rcd
rclone rc sync/sync srcFs=localdrive:localpath dstFs=remote:remotepath createEmptySrcDirs=true _async=true _group=mySync
./rclone rc --json '{ "group": "mySync" }' core/stats

The group-filter is required to get the remaining transfers, because without the group-filter "totalTransfers" in core/stats = transfered files ("transfers") + active transfers

So everything is already available in your wonderful rclone 😃 Therefore closing this issue.