Skip to content

Commit

Permalink
sync: implement directory sync for mod times and metadata
Browse files Browse the repository at this point in the history 
Directory mod times are synced by default if the backend is capable
and directory metadata is synced if the --metadata flag is provided
and the backend is capable.

This updates the bisync golden tests also which were affected by
--dry-run setting of directory modtimes.

Fixes #6685
  • Loading branch information
@ncw
ncw committed Feb 28, 2024
1 parent 15579c2 commit f5f8678
Show file tree
Hide file tree
Showing 23 changed files with 504 additions and 14 deletions.
7 changes: 7 additions & 0 deletions cmd/bisync/bisync_test.go
View file
Expand Up @@ -86,6 +86,10 @@ var logReplacements = []string{
`^.*?"SlowHashDetected":.*?$`, dropMe,
`^.*? for same-side diffs on .*?$`, dropMe,
`^.*?Downloading hashes.*?$`, dropMe,
// ignore timestamps in directory time updates
`^(INFO : .*?: Made directory with (metadata|modification time)).*$`, `$1`,
// ignore sizes in directory time updates
`^(NOTICE: .*?: Skipped set directory modification time as --dry-run is set).*$`, `$1`,
}

// Some dry-run messages differ depending on the particular remote.
Expand Down Expand Up @@ -121,6 +125,9 @@ var logHoppers = []string{

// order of files re-checked prior to a conflict rename
`ERROR : .*: md5 differ.*`,

// Directory modification time setting can happen in any order
`INFO : .*: (Set directory modification time|Made directory with metadata).*`,
}

// Some log lines can contain Windows path separator that must be
Expand Down
8 changes: 8 additions & 0 deletions cmd/bisync/testdata/test_all_changed/golden/test.log
View file
Expand Up @@ -16,7 +16,11 @@ INFO : Bisyncing with Comparison Settings:
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Copying Path2 files to Path1
INFO : - Path2 Resync is copying files to - Path1
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : - Path1 Resync is copying files to - Path2
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : Resync updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
Expand Down Expand Up @@ -59,6 +63,8 @@ INFO : - Path1 Queue copy to Path2 - {
INFO : - Path1 Queue copy to Path2 - {path2/}file1.txt
INFO : - Path1 Queue copy to Path2 - {path2/}subdir/file20.txt
INFO : - Path1 Do queued copies to - Path2
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : Updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
Expand Down Expand Up @@ -133,6 +139,8 @@ INFO : - Path1 Queue copy to Path2 - {
INFO : - Path1 Queue copy to Path2 - {path2/}file1.txt
INFO : - Path1 Queue copy to Path2 - {path2/}subdir/file20.txt
INFO : - Path1 Do queued copies to - Path2
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : Updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
6 changes: 6 additions & 0 deletions cmd/bisync/testdata/test_basic/golden/test.log
View file
Expand Up @@ -16,7 +16,11 @@ INFO : Bisyncing with Comparison Settings:
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Copying Path2 files to Path1
INFO : - Path2 Resync is copying files to - Path1
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : - Path1 Resync is copying files to - Path2
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : Resync updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
Expand Down Expand Up @@ -53,6 +57,8 @@ INFO : - Path1 Queue copy to Path2 - {
INFO : - Path2 Queue copy to Path1 - {path1/}file1.txt
INFO : - Path2 Do queued copies to - Path1
INFO : - Path1 Do queued copies to - Path2
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : Updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
12 changes: 12 additions & 0 deletions cmd/bisync/testdata/test_check_access/golden/test.log
View file
Expand Up @@ -16,7 +16,11 @@ INFO : Bisyncing with Comparison Settings:
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Copying Path2 files to Path1
INFO : - Path2 Resync is copying files to - Path1
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : - Path1 Resync is copying files to - Path2
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : Resync updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
Expand Down Expand Up @@ -87,7 +91,11 @@ INFO : Bisyncing with Comparison Settings:
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Copying Path2 files to Path1
INFO : - Path2 Resync is copying files to - Path1
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : - Path1 Resync is copying files to - Path2
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : Resync updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
Expand Down Expand Up @@ -180,7 +188,11 @@ INFO : Bisyncing with Comparison Settings:
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Copying Path2 files to Path1
INFO : - Path2 Resync is copying files to - Path1
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : - Path1 Resync is copying files to - Path2
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : Resync updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
Expand Down
24 changes: 24 additions & 0 deletions cmd/bisync/testdata/test_check_access_filters/golden/test.log
View file
Expand Up @@ -21,7 +21,15 @@ INFO : Using filters file {workdir/}exclude-other-filtersfile.txt
INFO : Storing filters file hash to {workdir/}exclude-other-filtersfile.txt.md5
INFO : Copying Path2 files to Path1
INFO : - Path2 Resync is copying files to - Path1
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir/subdirB: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir/subdirB: Set directory modification time (using SetModTime)
INFO : - Path1 Resync is copying files to - Path2
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir/subdirB: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir/subdirB: Set directory modification time (using SetModTime)
INFO : Resync updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
Expand Down Expand Up @@ -136,7 +144,23 @@ INFO : Using filters file {workdir/}include-other-filtersfile.txt
INFO : Storing filters file hash to {workdir/}include-other-filtersfile.txt.md5
INFO : Copying Path2 files to Path1
INFO : - Path2 Resync is copying files to - Path1
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdirX: Set directory modification time (using SetModTime)
INFO : subdirX/subdirX1: Set directory modification time (using SetModTime)
INFO : subdir/subdirB: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdirX: Set directory modification time (using SetModTime)
INFO : subdirX/subdirX1: Set directory modification time (using SetModTime)
INFO : subdir/subdirB: Set directory modification time (using SetModTime)
INFO : - Path1 Resync is copying files to - Path2
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdirX: Set directory modification time (using SetModTime)
INFO : subdir/subdirB: Set directory modification time (using SetModTime)
INFO : subdirX/subdirX1: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdirX: Set directory modification time (using SetModTime)
INFO : subdir/subdirB: Set directory modification time (using SetModTime)
INFO : subdirX/subdirX1: Set directory modification time (using SetModTime)
INFO : Resync updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
Expand Down
8 changes: 8 additions & 0 deletions cmd/bisync/testdata/test_check_filename/golden/test.log
View file
Expand Up @@ -16,7 +16,11 @@ INFO : Bisyncing with Comparison Settings:
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Copying Path2 files to Path1
INFO : - Path2 Resync is copying files to - Path1
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : - Path1 Resync is copying files to - Path2
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : Resync updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
Expand Down Expand Up @@ -90,7 +94,11 @@ INFO : Copying Path2 files to Path1
INFO : Checking access health
INFO : Found 2 matching ".chk_file" files on both paths
INFO : - Path2 Resync is copying files to - Path1
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : - Path1 Resync is copying files to - Path2
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : Resync updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
Expand Down
4 changes: 4 additions & 0 deletions cmd/bisync/testdata/test_createemptysrcdirs/golden/test.log
View file
Expand Up @@ -147,7 +147,11 @@ INFO : Bisyncing with Comparison Settings:
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Copying Path2 files to Path1
INFO : - Path2 Resync is copying files to - Path1
INFO : subdir: Made directory with metadata (mtime=2024-02-27T04:53:52.809861575-05:00)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : - Path1 Resync is copying files to - Path2
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : Resync updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
Expand Down
12 changes: 12 additions & 0 deletions cmd/bisync/testdata/test_extended_char_paths/golden/test.log
View file
Expand Up @@ -84,7 +84,11 @@ INFO : Bisyncing with Comparison Settings:
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Copying Path2 files to Path1
INFO : - Path2 Resync is copying files to - Path1
INFO : 測試_Русский_ _ _ě_áñ: Set directory modification time (using SetModTime)
INFO : 測試_Русский_ _ _ě_áñ: Set directory modification time (using SetModTime)
INFO : - Path1 Resync is copying files to - Path2
INFO : 測試_Русский_ _ _ě_áñ: Set directory modification time (using SetModTime)
INFO : 測試_Русский_ _ _ě_áñ: Set directory modification time (using SetModTime)
INFO : Resync updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
Expand Down Expand Up @@ -129,7 +133,11 @@ INFO : Bisyncing with Comparison Settings:
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Copying Path2 files to Path1
INFO : - Path2 Resync is copying files to - Path1
INFO : 測試_Русский_ _ _ě_áñ: Set directory modification time (using SetModTime)
INFO : 測試_Русский_ _ _ě_áñ: Set directory modification time (using SetModTime)
INFO : - Path1 Resync is copying files to - Path2
INFO : 測試_Русский_ _ _ě_áñ: Set directory modification time (using SetModTime)
INFO : 測試_Русский_ _ _ě_áñ: Set directory modification time (using SetModTime)
INFO : Resync updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
Expand Down Expand Up @@ -174,7 +182,11 @@ INFO : Using filters file {workdir/}測試_filtersfile.txt
INFO : Storing filters file hash to {workdir/}測試_filtersfile.txt.md5
INFO : Copying Path2 files to Path1
INFO : - Path2 Resync is copying files to - Path1
INFO : 測試_Русский_ _ _ě_áñ: Set directory modification time (using SetModTime)
INFO : 測試_Русский_ _ _ě_áñ: Set directory modification time (using SetModTime)
INFO : - Path1 Resync is copying files to - Path2
INFO : 測試_Русский_ _ _ě_áñ: Set directory modification time (using SetModTime)
INFO : 測試_Русский_ _ _ě_áñ: Set directory modification time (using SetModTime)
INFO : Resync updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
Expand Down
12 changes: 12 additions & 0 deletions cmd/bisync/testdata/test_extended_filenames/golden/test.log
View file
Expand Up @@ -25,7 +25,11 @@ INFO : Bisyncing with Comparison Settings:
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Copying Path2 files to Path1
INFO : - Path2 Resync is copying files to - Path1
INFO : subdir_with_ࢺ_: Set directory modification time (using SetModTime)
INFO : subdir_with_ࢺ_: Set directory modification time (using SetModTime)
INFO : - Path1 Resync is copying files to - Path2
INFO : subdir_with_ࢺ_: Set directory modification time (using SetModTime)
INFO : subdir_with_ࢺ_: Set directory modification time (using SetModTime)
INFO : Resync updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
Expand Down Expand Up @@ -112,7 +116,15 @@ INFO : - Path2 Queue copy to Path1 - "
INFO : - Path1 Queue delete - {path1/}subdir_with_ࢺ_/filename_contains_ě_.txt
INFO : - Path2 Queue copy to Path1 - {path1/}subdir_with_ࢺ_/filename_contains_ࢺ_p2s.txt
INFO : - Path2 Do queued copies to - Path1
INFO : subdir with␊white space.txt: Made directory with metadata (mtime=2024-02-27T04:53:52.913860529-05:00)
INFO : subdir_rawchars_␙_�_�: Made directory with metadata (mtime=2024-02-27T04:53:52.913860529-05:00)
INFO : subdir_with_ࢺ_: Set directory modification time (using SetModTime)
INFO : subdir with␊white space.txt: Set directory modification time (using SetModTime)
INFO : subdir_rawchars_␙_�_�: Set directory modification time (using SetModTime)
INFO : subdir_with_ࢺ_: Set directory modification time (using SetModTime)
INFO : - Path1 Do queued copies to - Path2
INFO : subdir_with_ࢺ_: Set directory modification time (using SetModTime)
INFO : subdir_with_ࢺ_: Set directory modification time (using SetModTime)
INFO : Updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
6 changes: 6 additions & 0 deletions cmd/bisync/testdata/test_filters/golden/test.log
View file
Expand Up @@ -20,7 +20,11 @@ INFO : Using filters file {workdir/}filtersfile.flt
INFO : Storing filters file hash to {workdir/}filtersfile.flt.md5
INFO : Copying Path2 files to Path1
INFO : - Path2 Resync is copying files to - Path1
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : - Path1 Resync is copying files to - Path2
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : Resync updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
Expand Down Expand Up @@ -54,6 +58,8 @@ INFO : Path2 checking for diffs
INFO : Applying changes
INFO : - Path1 Queue copy to Path2 - {path2/}subdir/fileZ.txt
INFO : - Path1 Do queued copies to - Path2
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : Updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
12 changes: 12 additions & 0 deletions cmd/bisync/testdata/test_filtersfile_checks/golden/test.log
View file
Expand Up @@ -16,7 +16,11 @@ INFO : Bisyncing with Comparison Settings:
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Copying Path2 files to Path1
INFO : - Path2 Resync is copying files to - Path1
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : - Path1 Resync is copying files to - Path2
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : Resync updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
Expand Down Expand Up @@ -81,7 +85,11 @@ INFO : Using filters file {workdir/}filtersfile.txt
INFO : Storing filters file hash to {workdir/}filtersfile.txt.md5
INFO : Copying Path2 files to Path1
INFO : - Path2 Resync is copying files to - Path1
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : - Path1 Resync is copying files to - Path2
INFO : subdir: Set directory modification time (using SetModTime)
INFO : subdir: Set directory modification time (using SetModTime)
INFO : Resync updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
Expand Down Expand Up @@ -146,7 +154,11 @@ INFO : Using filters file {workdir/}filtersfile.txt
INFO : Skipped storing filters file hash to {workdir/}filtersfile.txt.md5 as --dry-run is set
INFO : Copying Path2 files to Path1
INFO : - Path2 Resync is copying files to - Path1
NOTICE: subdir: Skipped set directory modification time as --dry-run is set (size 4Ki)
NOTICE: {path1String}: Skipped set directory modification time as --dry-run is set
INFO : - Path1 Resync is copying files to - Path2
NOTICE: subdir: Skipped set directory modification time as --dry-run is set (size 4Ki)
NOTICE: {path2String}: Skipped set directory modification time as --dry-run is set
INFO : Resync updating listings
INFO : Bisync successful

Expand Down

0 comments on commit f5f8678

Please sign in to comment.