bisync: implement sessions

[ivandeex]

What problem are you trying to solve?

We should track critical filtering and change detection flags in sessions
and either add a flag named --use-saved-options (or --continue)...
or by default restore such flags from the session json,
allowing user to change such flags only when --resync.

Also beta bisync introduced the flag --filters-file to track hashsum of filter files.
We should remove it (it duplicates --filter-from) and track filter checksums in the session altogether.

Implementation details

Currently bisync applies its own set of rules to adapt names of source-target listings to local OS.
Better use good old filepath.ToSlash, filepath.FromSlash as well as new encoder.OS from commit 3a2f748.

Prior discussions

#5164 (comment) (Jwink3101)

Unlike mirroring or one-way sync (a la sync), true bidirectional sync cannot be stateless. However, I think where to store the state warrants more discussion. One thing often discussed on the forum and rclone subreddit is that you can pull your config to a VPS and run it there. If you store the state of past syncs on the syncing machine, you now need to keep track of that.

This is actually, in my opinion, one of the fundamental issues with rclone and bi-directional sync. You need a robust way to handle this and you can't just go into a deeper dir. (for example rclone sync A: B: works as does rclone sync A:sub B:sub). This is the primary reason I chose to for my tool to not be a pure CLI one and instead of a sync config file that dictated these things.

#5164 (comment) (ivandeex)

I thought in a different direction - to snapshot the current state of flags at the time of operation into bisync session (extending the idea of hashsumming the filters) and request a resync or at least warn if it changes.

#5164 (comment) (cjnaz)

Hashing the user's filter file in rclonesync addressed a bug where many files were deleted because they did not show up in the current state list and were in the prior history list, so they needed to get deleted on the other side. The rclonesync algorithm evolved, so the specific fail case may not be there now. In any case, damage control must be a prime design requirement. --backup-dir sounds appropriate. Are CLI filter options accepted by bisync?

#5164 (comment) (ivandeex)

The postponed item "Track critical filtering and change detection flags" will:

• snapshot selected settings: backend options, command-line flags, environment, in particular related to change detection and filtering, etc
• store them in a per-session file (together with filters file hash, instead of the per-filter file)
• on the next runs bisync will detect critical changes and maybe request a resync

Additionally, since many settings are pre-saved for user, we can add a shortcut bisync flag --use-saved-options.
User can run just rclone bisync path1 path2 --use-saved-options and all saved settings will apply, keyed by pair path1/path2.

#5164 (comment) (ivandeex)

Naturally, we could add another flag --saved-sessions-only to abort if combination remote1:empty-dir remote2:
has not been used with bisync before and prevent poof or BANG followed by collapse.

Bisync help (long one) has a safety dedicated section and a few notes regarding --dry-run and --verbose scattered around the text.
User data safety is also covered by the TODO item "Honor --interactive and --auto-confirm flags"
(please note that these flags have long been supported by the sync and other destructive commands).

Non-interactive, unattended use in scripts or with cron should be possible with --force and/or --auto-confirm.

#5164 (comment) (ncw) -> (ivandeex)

What is the difference between --filters-file and --filter-from ?

Bisync will track changes in the filters file set by --filters-file and enforce --resync if any.
By the merge time I'm going to extend this idea to other sync-related flags, so we can ensure even more safety.

Probably bisync should set hook on every --filter-from file making this special flag moot.

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

[ivandeex]

The bisync --workdir argument should support both local and remote paths. The latter case will allow to run bisync from multiple workplaces consistently.

[ivandeex]

We already have following config sources (ordered from low to high priority):
defaults, backend config, environment, cmdline.

bisync sessions can be implemented as a new config source bisync session where session settings file can be saved in an ini-file located at ~/.cache/rclone/bisync/SRC~DST/session.ini

Sessions will be created whenever new bisync src-dst combination is used or --resync-ed.

I'm not sure about priority of the session source. I think it should go between backend and environment above.
If profiles will be ever implemented, sessions should have lower priority than profiles.

@albertony thoughts?

[nielash]

Notes to self:

• see discussion here about configurable session name, partcularly as currently some hacky workarounds are required to sync very long paths.
• See also relevant discussion here regarding safety check for overridden configs.