Query rclone mount VFS stats via rclone remote control API

[jdr85]

The associated forum post URL from https://forum.rclone.org

https://forum.rclone.org/t/how-to-check-mount-upload-progress-from-scripts/27396

What is your current rclone version (output from rclone version)?

rclone v1.57.0

• os/version: ubuntu 20.04 (64 bit)
• os/kernel: 5.4.0-90-generic (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.17.2
• go/linking: static
• go/tags: none

What problem are you are trying to solve?

I would like to query the VFS cache stats of rclone mount from scripts using rclone's remote control (rc) API. At the moment I have to parse rclone's output or logfile and examine the corresponding INFO messages such as vfs cache: cleaned: objects 2 (was 2) in use 2, to upload 0, uploading 2, total size 76.831Mi (was 76.831Mi)

How do you think rclone should be changed to solve that?

It would be awesome to query these statistics via remote control, such as (arbitrary example, could also look a little bit different):

rclone rc vfscache/stats
{
  "objects": 10,
  "inUse": 5,
  "toUpload": 3,
  "uploading": 2,
  "totalSize": 131072
}

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

[ncw]

Have a go with this

v1.58.0-beta.5892.f9f46777a.fix-5816-vfs-stats on branch fix-5816-vfs-stats (uploaded in 15-30 mins)

It implements vfs/stats the help for which looks like this

vfs/stats: Stats for a VFS. {#vfs-stats}

This returns stats for the selected VFS.

{
    // Status of the disk cache - only present if --vfs-cache-mode > off
    "diskCache": {
        "bytesUsed": 0,
        "erroredFiles": 0,
        "files": 0,
        "hashType": 1,
        "outOfSpace": false,
        "path": "/home/user/.cache/rclone/vfs/local/mnt/a",
        "pathMeta": "/home/user/.cache/rclone/vfsMeta/local/mnt/a",
        "uploadsInProgress": 0,
        "uploadsQueued": 0
    },
    "fs": "/mnt/a",
    "inUse": 1,
    // Status of the in memory metadata cache
    "metadataCache": {
        "dirs": 1,
        "files": 0
    },
    // Options as returned by options/get
    "opt": {
        "CacheMaxAge": 3600000000000,
        // ...
        "WriteWait": 1000000000
    }
}

This command takes an "fs" parameter. If this parameter is not
supplied and if there is only one VFS in use then that VFS will be
used. If there is more than one VFS in use then the "fs" parameter
must be supplied.

[jdr85]

Looks like this was easier for you to implement than for me to open this issue... fantastic, thanks! 😃

Maybe there could be an additional entry like "synced": true/false but I assume that "files": 0 means that the cache is completely synced.

[ncw]

For synced you'd want these both 0

    "uploadsInProgress": 0,
    "uploadsQueued": 0

There will always be files in the cache.

I'd appreciated it if you'd have a play with the beta and see if it works for you as I only wrote the code, I didn't try it in anger ;-)

[jdr85]

Hi ncw,
Yes, of course I already did some basic testing (although also not in anger ;)) for my use case and everything seems to work perfectly. Here's what I basically did and what happened.

(1) Mount "test" remote to /cloud/test; "test" is a simple WebDAV remote (GMX Cloud).
./rclone --rc mount --vfs-cache-mode writes --log-level INFO --cache-dir /daten/_jdr/rclone_vfs_cache --vfs-write-back 60s --vfs-cache-max-age 2m --vfs-cache-max-size 10G --stats 1m --stats-log-level NOTICE --stats-one-line test:Backup/ /cloud/test/

After that, I watched the output of rclone rc vfs/stats fs=test:Backup in different situations.

(2) Before doing anything else (/cloud/test is still empty)

"diskCache": {
                "bytesUsed": 0,
                "erroredFiles": 0,
                "files": 0,
                "hashType": 0,
                "outOfSpace": false,
                "uploadsInProgress": 0,
                "uploadsQueued": 0
        },

(3) Drop a file into /cloud/test

"diskCache": {
                "bytesUsed": 77101072,
                "erroredFiles": 0,
                "files": 1,
                "hashType": 0,
                "outOfSpace": false,
                "uploadsInProgress": 0,
                "uploadsQueued": 1
        },

--> Exactly what I expected, there is 1 new file which uses some space and this file is queued for upload.

(4) Wait until vfs-write-back time has expired (60s here)

"diskCache": {
                "bytesUsed": 77101072,
                "erroredFiles": 0,
                "files": 1,
                "hashType": 0,
                "outOfSpace": false,
                "uploadsInProgress": 1,
                "uploadsQueued": 0
        },

--> As expected: file is now uploading and not just queued any more.

(5) Wait until file upload finished

 "diskCache": {
                "bytesUsed": 77101072,
                "erroredFiles": 0,
                "files": 1,
                "hashType": 0,
                "outOfSpace": false,
                "uploadsInProgress": 0,
                "uploadsQueued": 0
        },

--> Not uploading any more but still in cache as vfs-cache-max-age 2m has not expired yet

(6) Wait for vfs-cache-max-age expiry

"diskCache": {
                "bytesUsed": 0,
                "erroredFiles": 0,
                "files": 0,
                "hashType": 0,
                "outOfSpace": false,
                "uploadsInProgress": 0,
                "uploadsQueued": 0
        },

--> Cache is now empty.

Renames and deletions are immediately propagated to the server. The "files" count will increase accordingly for each file renamed/deleted until the vfs-cache-max-age has expired for the respective file.

I just cannot confirm that there will always be files in the cache. As soon as the vfs-cache-max-age has expired for all files, the cache is empty (which was also what I expected).

Is there anything else I should test? Otherwise I would claim that everything works as expected and this issue can probably be closed.

Btw, thank you very much for this fantastic piece of software and its continuous development!

[ncw]

Thanks for testing.

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.58

[matzzzl]

I have tested rclone v1.58.0-beta.5918.408d9f3e7 (current latest beta) (because the feature discussed in this feature request was just what I was looking for!)

But unfortunately the "--rc" option of rclone does not work together with the "--daemon" option of rclone mount.

If I enter
rclone --rc mount --daemon --vfs-cache-mode writes --vfs-write-back 60s --vfs-cache-max-age 2m --log-level INFO --log-file "/tmp/rclone_mount.log" mega: /mnt
the log file contains the error
Failed to start remote control: start server failed: listen tcp 127.0.0.1:5572: bind: address already in use
and the mount is not available.

If I enter this command without either "--rc" or "--deamon" everything works as expected.

[jdr85]

Hi @matzzzl
I guess this has nothing to do with the new feature but is a known bug tracked in #2618.

It's also described here.