bisync: add modtime support

[ivandeex]

What problem are you are trying to solve?

bisync delta engine and unit tests do not take backend time tolerance into account. this should be fixed.

modtime in unit tests

#5164 (comment) (ivandeex)

the unit test runner is based on mangleListing which is a very far relative of fstest.CompareItems. It already has ability to ignore hashsums on demand. It still needs ability to compare times with per-backend tolerance or even ignore times if unsupported. Having that would let us run tests against backends without time support and probably lift the corresponding ban.

Tentatively this can be implemented as:

1. parse out the time filed and convert to time.Time
2. round time down to doubled tolerance
3. put time field back into string
4. feed the result to diff.

Why not just parse and compare with tolerance?
Because mangleResult is two-fold. Besides comparing it can prepare a test state snapshot for storing in golden directory, which later can provide a bigger picture of what's going on.

#5164 (comment) (ivandeex)

With a feature must come a method of testing it.

Rclone has a comprehensive fstest framework dedicated to peculiarities of the plethora of supported cloud providers. Planning to work on this patch I had the heart to keep the project test engine's nature (while translating it from python to golang and refactoring it to work natively/unattended on Windows and with a wider set of backends) and propose it as a new kid on the set of rclone approaches (see my inline comment above), instead of rewriting it from the grounds up for fstest.

The bisync test suite lacks the ability to compare imprecise/absent timestamps as of the time of this writing (I hope I can say it, absent timestamps should be "comparable" too). It works correctly mostly because it compares logs from a particular filesystem with itself. Unless this ward is fixed, we can't run tests against backends without modtime and consequently can't declare them supported. I have an idea on how to fix that (see another inline comment), but the effort will need time.

modtime in delta engine

Try to extend operations.Equal into operations.Compare with more returned conditions like Newer, Older, ... (related to #4810) for direct use in bisync delta engine

#5164 (comment) (JWink3101)

The fact that this only supports a small subset of remotes is disappointing, esp. some popular ones not supporting mod time. Others, like S3, are way more performant if you disable it.

This is an issue with the overall algorithm of some sync tools but there is no reason you need modtime to decide what to sync. You may need it for conflicts but if you have past state, then you can tell if a changed file-size is a modification on one side only.

#5164 (comment) (ivandeex)

This is an issue with the overall algorithm of some sync tools but there is no reason you need modtime to decide what to sync.

Theoretically, no. The bisync change detection in its current state is all about bit masks (as opposed to "rclonic" Equal). But it needs more testing.

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

[nielash]

Related to this ticket (and also #5676), --check-sync should also support comparing modtime (and eventually, when checksum and size are supported, it should use the same sync key used for the bisync run.)

Currently, --check-sync only compares filenames, not modtime, size, or hash. It can detect whether a file from one Path is missing on the other, but not whether both Paths have different versions of a file. This can cause issues such as this one.

Alternatively, perhaps this would be better handled by outsourcing it to the more robust rclone check, rather than attempting to duplicate the same functionality in --check-sync.

Here's where the trouble is:

rclone/cmd/bisync/operations.go

Lines 474 to 479 in 94a320f

	for _, file := range files1.list {
	if !files2.has(file) {
	b.indent("ERROR", file, "Path1 file not found in Path2")
	ok = false
	}
	}

rclone/cmd/bisync/listing.go

Lines 53 to 58 in 94a320f

	// fileList represents a listing
	type fileList struct {
	list []string
	info map[string]*fileInfo
	hash hash.Type
	}