bisync: session name should normalize to non-canonical version for overridden configs

[LivingUserError]

What is the problem you are having with rclone?

When calling bisync with a remote, but a leading / in the remote path like so

rclone bisync remote:/path/with/unneeded/slash /some/local/path --resync

the path will be normalized by Rclone and the command will be executed without throwing any error (or warning).

If you then try to run bisync without --resync however, it will not behave as expected. Running the exact same command without --resync will always yield:

ERROR : Bisync critical error: cannot find prior Path1 or Path2 listings, likely due to critical error on prior run

This suggests that bisync failed previously, when in fact it didn't. It seems to name the listings after the normalized path, while checking for existing listings with the non-normalized path, as provided by the user.

This seems likely, because running the command again, but without the leading slash and --resync, will work:

rclone bisync remote:path/with/unneeded/slash /some/local/path

What is your rclone version (output from rclone version)

rclone v1.64.2

• os/version: arch Soaring (64 bit)
• os/kernel: 6.6.1-zen1-1-zen (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.21.3
• go/linking: dynamic
• go/tags: none

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

Arch Linux, 64 bit

Which cloud storage system are you using? (e.g. Google Drive)

Proton Drive

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

[nielash]

Thank you for reporting this! I am having trouble reproducing it, though. For me, the following paths:

remote:/path/with/unneeded/slash
remote:path/with/unneeded/slash

both result in:

remote_path_with_unneeded_slash

in the listing filename.

Are you using any backend-specific flags that could be overriding your config and adding a suffixed canonical name? (More info)

Another thing that would be helpful is if you could try going into your workdir to see what bisync is actually naming the listing files, and whether there is any difference when you --resync with and without the slash.

For example, try something like the following sequence:

rclone bisync remote:/path/with/unneeded/slash /some/local/path --resync
rclone lsl '~/.cache/rclone/bisync'
rclone bisync remote:path/with/unneeded/slash /some/local/path --resync
rclone lsl '~/.cache/rclone/bisync'

And let me know if there is any difference between the filenames output by the two lsls?

If that doesn't yield clues, I'm also curious if it could be something specific to the encoding settings on Proton Drive. Any chance you could try the same thing with Google Drive and see if you get a different result?

[LivingUserError]

I am using one backend-specific option:

--protondrive-replace-existing-draft=true

My full command looks like this:

rclone bisync proton:sync /home/user/Proton --no-update-modtime --no-traverse --transfers 8 --protondrive-replace-existing-draft=true -v

ls '~/.cache/rclone/bisync' yields:

proton{kpshc}_sync..home_user_Proton.lck
proton{kpshc}_sync..home_user_Proton.path1.lst
proton{kpshc}_sync..home_user_Proton.path2.lst
proton_sync..home_user_Proton.path1.lst-new
proton_sync..home_user_Proton.path2.lst-new

This is after the command has been run with the unneeded slash and --resync first, and without both afterwards. If I find the time I can retry from zero, as you suggested.

I forgot to mention that I run rclone as root btw., maybe that also helps.

[nielash]

Thanks! The {kpshc} in the remote name is the telltale sign of an overridden config. The two that don't have it are curious to me -- are you able to tell from their modtimes whether they were created around the same time as the others? Or are they possibly just leftover from an old run, before you started using --protondrive-replace-existing-draft=true?

Are you using --protondrive-replace-existing-draft=true for both --resync and non-resync?

[LivingUserError]

After some extensive testing, you are probably right. I deleted my previous comment, because I noticed a mistake in the testing - didn't provide anything of value.

I must have added --protondrive-replace-existing-draft=true after the fact and that broke it. Might still want to add a hint that similar listings (i.e. listings form running the command with different options) exist. That was quite the debug session 😅

[nielash]

Might still want to add a hint that similar listings (i.e. listings form running the command with different options) exist. That was quite the debug session 😅

Thanks, and I agree! This is one I have tripped over myself. I added a section to the docs about it in v1.64.0, but I don't think that goes far enough, as users are continuing to stumble over it, and it can actually have serious consequences in the reverse direction (if you remove a backend-specific flag and still have old listings using the alternate name.)

The ultimate fix for this is #5678, but in the meantime I think we should normalize to the non-canonical name for listings -- it should not be possible to have more than one session name for the same pair of bisync paths. I have an idea about how to implement this in a way that is backward-compatible, which I will try out as soon as I have a chance. Thanks again for your work on this!

[LivingUserError]

Thank you for pointing me in the right direction! I could see why you would want to have multiple sessions for accessing the same remote tho - these backend-specific options could do pretty much anything, can't they? Maybe a warning/prompt is really the best one can do here, without limiting functionality.

[nielash]

When I say "session" here I'm talking specifically about the SessionName.

In other words, if you do

rclone bisync a b

followed by

rclone bisync a b --protondrive-replace-existing-draft=true

bisync should understand that this is the same pair of paths, and that therefore the same listing file should be used. (Same as would happen today with a non-backend-specific flag.)

As opposed to:

rclone bisync a c

which is a different session from rclone bisync a b and should use a different listing file.

I don't think what I'm suggesting limits any functionality (it's more of a bug fix, really), but feel free to push back if you think I'm missing something!