Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Return status info from movers #538

Closed
JohnStrunk opened this issue Dec 7, 2022 · 5 comments
Closed

Return status info from movers #538

JohnStrunk opened this issue Dec 7, 2022 · 5 comments
Assignees
@tesshuflower
Labels
enhancement New feature or request
Milestone
Version 0.7

Comments

@JohnStrunk
Copy link
Member

Describe the feature you'd like to have.
Today, there is relatively little information about what is happening during the sync process. We publish events for things like snapshots/clones, but once the Job starts, there is no available information until it completes. I'd like to be able to return status updates from the data movers so that the information can be exposed to end users without requiring them to kubectl logs <volsync-mover-job>.

What is the value to the end user? (why is it a priority?)
It would be good for the mover jobs to be able to expose status information so that users can more easily see what the current status of synchronization is. Is it syncing? Is it failing to connect?

How will we know we have a good solution? (acceptance criteria)

  • Movers can send short "status updates" to the controller
  • The controller can publish the status updates to the associated CR's status field (probably via the Synchronizing condition)

Additional context
Updates we might want to publish:

  • (Un)successful connection to a remote (i.e., is the network working)
  • Transfer rates and/or progress information (x% complete; MM:SS remaining)
  • Amount of data transferred
  • Relevant timestamps w/ the above

Potential methods for returning information:

  • Specially formatted log lines that could be scraped by the controller
  • http endpoint that could be read by the operator
@JohnStrunk JohnStrunk added the enhancement New feature or request label Dec 7, 2022
@JohnStrunk JohnStrunk added this to the Version 0.7 milestone Dec 14, 2022
@tesshuflower
Copy link
Contributor

tesshuflower commented Dec 14, 2022 •
edited
Loading

Some info about current logging from movers

Rclone:

Example output from replication source mover (with no changes to rclone settings):

VolSync rclone container version: ACM-0.6.0-ce9a280
getfacl: Removing leading '/' from absolute path names
2022/12/14 20:41:10 DEBUG : Setting --config "/rclone-config/rclone.conf" from environment variable RCLONE_CONFIG="/rclone-config/rclone.conf"
2022/12/14 20:41:10 DEBUG : rclone: Version "v1.59.0-beta.." starting with parameters ["rclone" "sync" "--checksum" "--one-file-system" "--create-empty-src-dirs" "--progress" "--stats-one-line-date" "--stats" "20s" "--transfers" "10" "/data" "rclone-data-mover:rclone-test-0-zx42b" "--log-level" "DEBUG"]
2022/12/14 20:41:10 DEBUG : Creating backend with remote "/data"
2022/12/14 20:41:10 DEBUG : local: detected overridden config - adding "{vECL7}" suffix to name
2022/12/14 20:41:10 DEBUG : Using config file from "/rclone-config/rclone.conf"
2022/12/14 20:41:10 DEBUG : fs cache: renaming cache item "/data" to be canonical "local{vECL7}:/data"
2022/12/14 20:41:10 DEBUG : Creating backend with remote "rclone-data-mover:rclone-test-0-zx42b"

2022-12-14 20:41:15 DEBUG : datafile: md5 = 6137cde4893c59f76f005a8123d8e8e6 OK
2022/12/14 20:41:15 -         580 B / 4.481 MiB, 0%, 0 B/s, ETA -
2022-12-14 20:41:15 DEBUG : datafile: Size and md5 of src and dst objects identical
2022/12/14 20:41:15 -     4.566 KiB / 4.481 MiB, 0%, 0 B/s, ETA -
2022-12-14 20:41:15 DEBUG : datafile: Unchanged skipping
2022/12/14 20:41:15 -     4.566 KiB / 4.481 MiB, 0%, 0 B/s, ETA -
2022-12-14 20:41:15 DEBUG : outfile: md5 = 25cefd52d32a3eb1edf336909001aaa6 OK
2022/12/14 20:41:15 -     4.566 KiB / 4.481 MiB, 0%, 0 B/s, ETA -
2022-12-14 20:41:15 INFO  : outfile: Copied (new)
2022/12/14 20:41:15 -     4.566 KiB / 4.481 MiB, 0%, 0 B/s, ETA -
2022-12-14 20:41:15 DEBUG : S3 bucket rclone-test-0-zx42b: Waiting for checks to finish
2022/12/14 20:41:15 -     4.566 KiB / 4.481 MiB, 0%, 0 B/s, ETA -
2022-12-14 20:41:15 DEBUG : S3 bucket rclone-test-0-zx42b: Waiting for transfers to finish
2022/12/14 20:41:15 -    60.566 KiB / 4.481 MiB, 1%, 0 B/s, ETA -
2022-12-14 20:41:15 DEBUG : TESTDIR2/file3: md5 = 9513a5e720188686c24dba9b99b44198 OK
2022/12/14 20:41:15 -     4.481 MiB / 4.481 MiB, 100%, 0 B/s, ETA -
2022-12-14 20:41:15 INFO  : TESTDIR2/file3: Copied (new)
2022/12/14 20:41:15 -     4.481 MiB / 4.481 MiB, 100%, 0 B/s, ETA -
2022-12-14 20:41:15 DEBUG : TESTDIR2/file2: md5 = 8be2afc0c3d8fea26d7e674ee6e56867 OK
2022/12/14 20:41:15 -     4.481 MiB / 4.481 MiB, 100%, 0 B/s, ETA -
2022-12-14 20:41:15 INFO  : TESTDIR2/file2: Copied (new)
2022/12/14 20:41:15 -     4.481 MiB / 4.481 MiB, 100%, 0 B/s, ETA -
2022-12-14 20:41:15 DEBUG : TESTDIR1/file1: md5 = 67f30a9f1d05a2a348b8152382022eb6 OK
2022/12/14 20:41:15 -     4.481 MiB / 4.481 MiB, 100%, 0 B/s, ETA -
2022-12-14 20:41:15 INFO  : TESTDIR1/file1: Copied (new)
2022/12/14 20:41:15 -     4.481 MiB / 4.481 MiB, 100%, 0 B/s, ETA -
2022-12-14 20:41:15 DEBUG : lost+found: Making directory
2022/12/14 20:41:15 -     4.481 MiB / 4.481 MiB, 100%, 0 B/s, ETA -
2022-12-14 20:41:15 DEBUG : S3 bucket rclone-test-0-zx42b: copied 1 directories
2022/12/14 20:41:15 -     4.481 MiB / 4.481 MiB, 100%, 0 B/s, ETA -
2022-12-14 20:41:15 DEBUG : Waiting for deletions to finish
2022/12/14 20:41:15 -     4.481 MiB / 4.481 MiB, 100%, 0 B/s, ETA -
2022-12-14 20:41:15 INFO  : permissions.facl: Deleted
2022/12/14 20:41:15 -     4.481 MiB / 4.481 MiB, 100%, 0 B/s, ETA -2022/12/14 20:41:15 -     4.481 MiB / 4.481 MiB, 100%, 0 B/s, ETA -
2022/12/14 20:41:15 INFO  : 2022/12/14 20:41:15 -     4.481 MiB / 4.481 MiB, 100%, 0 B/s, ETA -
2022/12/14 20:41:15 DEBUG : 11 go routines active
2022/12/14 20:41:15 DEBUG : Setting --config "/rclone-config/rclone.conf" from environment variable RCLONE_CONFIG="/rclone-config/rclone.conf"
2022/12/14 20:41:15 DEBUG : rclone: Version "v1.59.0-beta.." starting with parameters ["rclone" "copy" "--checksum" "--one-file-system" "--create-empty-src-dirs" "--progress" "--stats-one-line-date" "--stats" "20s" "--transfers" "10" "--include" "permissions.facl" "/tmp" "rclone-data-mover:rclone-test-0-zx42b" "--log-level" "DEBUG"]
2022/12/14 20:41:15 DEBUG : Creating backend with remote "/tmp"
2022/12/14 20:41:15 DEBUG : local: detected overridden config - adding "{vECL7}" suffix to name
2022/12/14 20:41:15 DEBUG : Using config file from "/rclone-config/rclone.conf"
2022/12/14 20:41:15 DEBUG : fs cache: renaming cache item "/tmp" to be canonical "local{vECL7}:/tmp"
2022/12/14 20:41:15 DEBUG : Creating backend with remote "rclone-data-mover:rclone-test-0-zx42b"

2022-12-14 20:41:15 DEBUG : datafile: Excluded
2022/12/14 20:41:15 -           0 B / 0 B, -, 0 B/s, ETA -
2022-12-14 20:41:15 DEBUG : outfile: Excluded
2022/12/14 20:41:15 -           0 B / 0 B, -, 0 B/s, ETA -
2022-12-14 20:41:15 DEBUG : S3 bucket rclone-test-0-zx42b: Waiting for checks to finish
2022/12/14 20:41:15 -           0 B / 869 B, 0%, 0 B/s, ETA - (xfr#0/1)
2022-12-14 20:41:15 DEBUG : S3 bucket rclone-test-0-zx42b: Waiting for transfers to finish
2022/12/14 20:41:15 -           0 B / 869 B, 0%, 0 B/s, ETA - (xfr#0/1)
2022-12-14 20:41:15 DEBUG : permissions.facl: md5 = f6b91300cd0b781be85b8cd45ee1d56c OK
2022/12/14 20:41:15 -         869 B / 869 B, 100%, 0 B/s, ETA -
2022-12-14 20:41:15 INFO  : permissions.facl: Copied (new)
2022/12/14 20:41:15 -         869 B / 869 B, 100%, 0 B/s, ETA -2022/12/14 20:41:15 -         869 B / 869 B, 100%, 0 B/s, ETA -
2022/12/14 20:41:15 INFO  : 2022/12/14 20:41:15 -         869 B / 869 B, 100%, 0 B/s, ETA -
2022/12/14 20:41:15 DEBUG : 7 go routines active
Rclone completed in 5s
  • No summary in the output atm, there are some transfer sizes/speeds for individual files, but interestingly everything I see shows 0 B/s

Here's output from a destination mover job:

VolSync rclone container version: ACM-0.6.0-ce9a280
2022/12/14 20:48:29 DEBUG : Setting --config "/rclone-config/rclone.conf" from environment variable RCLONE_CONFIG="/rclone-config/rclone.conf"
2022/12/14 20:48:29 DEBUG : rclone: Version "v1.59.0-beta.." starting with parameters ["rclone" "sync" "--checksum" "--one-file-system" "--create-empty-src-dirs" "--progress" "--stats-one-line-date" "--stats" "20s" "--transfers" "10" "--exclude" "permissions.facl" "rclone-data-mover:rclone-test-0-zx42b" "/data" "--log-level" "DEBUG"]
2022/12/14 20:48:29 DEBUG : Creating backend with remote "rclone-data-mover:rclone-test-0-zx42b"
2022/12/14 20:48:29 DEBUG : Using config file from "/rclone-config/rclone.conf"
2022/12/14 20:48:29 DEBUG : Creating backend with remote "/data"
2022/12/14 20:48:29 DEBUG : local: detected overridden config - adding "{vECL7}" suffix to name
2022/12/14 20:48:29 DEBUG : fs cache: renaming cache item "/data" to be canonical "local{vECL7}:/data"

2022-12-14 20:48:34 DEBUG : permissions.facl: Excluded
2022/12/14 20:48:34 -           0 B / 0 B, -, 0 B/s, ETA -
2022-12-14 20:48:34 DEBUG : datafile: md5 = 6137cde4893c59f76f005a8123d8e8e6 OK
2022/12/14 20:48:34 -           5 B / 5 B, 100%, 0 B/s, ETA -
2022-12-14 20:48:34 INFO  : datafile: Copied (new)
2022/12/14 20:48:34 -           5 B / 5 B, 100%, 0 B/s, ETA -
2022-12-14 20:48:34 DEBUG : outfile: md5 = 25cefd52d32a3eb1edf336909001aaa6 OK
2022/12/14 20:48:34 -         585 B / 1.403 MiB, 0%, 0 B/s, ETA - (xfr#1/4)
2022-12-14 20:48:34 INFO  : outfile: Copied (new)
2022/12/14 20:48:34 -         585 B / 585 B, 100%, 0 B/s, ETA -
2022-12-14 20:48:34 DEBUG : Local file system at /data: Waiting for checks to finish
2022/12/14 20:48:34 -         585 B / 585 B, 100%, 0 B/s, ETA -
2022-12-14 20:48:34 DEBUG : Local file system at /data: Waiting for transfers to finish
2022/12/14 20:48:34 -         585 B / 585 B, 100%, 0 B/s, ETA -
2022-12-14 20:48:34 DEBUG : TESTDIR2/file2: md5 = 8be2afc0c3d8fea26d7e674ee6e56867 OK
2022/12/14 20:48:34 -     2.753 MiB / 4.481 MiB, 61%, 0 B/s, ETA -
2022-12-14 20:48:34 INFO  : TESTDIR2/file2: Copied (new)
2022/12/14 20:48:34 -     2.753 MiB / 4.481 MiB, 61%, 0 B/s, ETA -
2022-12-14 20:48:34 DEBUG : TESTDIR2/file3: md5 = 9513a5e720188686c24dba9b99b44198 OK
2022/12/14 20:48:34 -     3.660 MiB / 4.481 MiB, 82%, 0 B/s, ETA -
2022-12-14 20:48:34 INFO  : TESTDIR2/file3: Copied (new)
2022/12/14 20:48:34 -     3.660 MiB / 4.481 MiB, 82%, 0 B/s, ETA -
2022-12-14 20:48:34 DEBUG : TESTDIR1/file1: md5 = 67f30a9f1d05a2a348b8152382022eb6 OK
2022/12/14 20:48:34 -     4.481 MiB / 4.481 MiB, 100%, 0 B/s, ETA -
2022-12-14 20:48:34 INFO  : TESTDIR1/file1: Copied (new)
2022/12/14 20:48:34 -     4.481 MiB / 4.481 MiB, 100%, 0 B/s, ETA -
2022-12-14 20:48:34 DEBUG : Waiting for deletions to finish
2022/12/14 20:48:34 -     4.481 MiB / 4.481 MiB, 100%, 0 B/s, ETA -
2022-12-14 20:48:34 INFO  : lost+found: Removing directory
2022/12/14 20:48:34 -     4.481 MiB / 4.481 MiB, 100%, 0 B/s, ETA -
2022-12-14 20:48:34 DEBUG : Local file system at /data: deleted 1 directories
2022/12/14 20:48:34 -     4.481 MiB / 4.481 MiB, 100%, 0 B/s, ETA -2022/12/14 20:48:34 -     4.481 MiB / 4.481 MiB, 100%, 0 B/s, ETA -
2022/12/14 20:48:34 INFO  : 2022/12/14 20:48:34 -     4.481 MiB / 4.481 MiB, 100%, 0 B/s, ETA -
2022/12/14 20:48:34 DEBUG : 12 go routines active
2022/12/14 20:48:34 DEBUG : Setting --config "/rclone-config/rclone.conf" from environment variable RCLONE_CONFIG="/rclone-config/rclone.conf"
2022/12/14 20:48:34 DEBUG : rclone: Version "v1.59.0-beta.." starting with parameters ["rclone" "copy" "--checksum" "--one-file-system" "--create-empty-src-dirs" "--progress" "--stats-one-line-date" "--stats" "20s" "--transfers" "10" "--include" "permissions.facl" "rclone-data-mover:rclone-test-0-zx42b" "/tmp" "--log-level" "DEBUG"]
2022/12/14 20:48:34 DEBUG : Creating backend with remote "rclone-data-mover:rclone-test-0-zx42b"
2022/12/14 20:48:34 DEBUG : Using config file from "/rclone-config/rclone.conf"
2022/12/14 20:48:34 DEBUG : Creating backend with remote "/tmp"
2022/12/14 20:48:34 DEBUG : local: detected overridden config - adding "{vECL7}" suffix to name
2022/12/14 20:48:34 DEBUG : fs cache: renaming cache item "/tmp" to be canonical "local{vECL7}:/tmp"

2022-12-14 20:48:34 DEBUG : datafile: Excluded
2022/12/14 20:48:34 -           0 B / 0 B, -, 0 B/s, ETA -
2022-12-14 20:48:34 DEBUG : outfile: Excluded
2022/12/14 20:48:34 -           0 B / 0 B, -, 0 B/s, ETA -
2022-12-14 20:48:34 DEBUG : TESTDIR2/file2: Excluded
2022/12/14 20:48:34 -           0 B / 0 B, -, 0 B/s, ETA -
2022-12-14 20:48:34 DEBUG : TESTDIR2/file3: Excluded
2022/12/14 20:48:34 -           0 B / 0 B, -, 0 B/s, ETA -
2022-12-14 20:48:34 DEBUG : permissions.facl: md5 = f6b91300cd0b781be85b8cd45ee1d56c OK
2022/12/14 20:48:34 -         869 B / 869 B, 100%, 0 B/s, ETA -
2022-12-14 20:48:34 INFO  : permissions.facl: Copied (new)
2022/12/14 20:48:34 -         869 B / 869 B, 100%, 0 B/s, ETA -
2022-12-14 20:48:34 DEBUG : TESTDIR1/file1: Excluded
2022/12/14 20:48:34 -         869 B / 869 B, 100%, 0 B/s, ETA -
2022-12-14 20:48:34 DEBUG : Local file system at /tmp: Waiting for checks to finish
2022/12/14 20:48:34 -         869 B / 869 B, 100%, 0 B/s, ETA -
2022-12-14 20:48:34 DEBUG : Local file system at /tmp: Waiting for transfers to finish
2022/12/14 20:48:34 -         869 B / 869 B, 100%, 0 B/s, ETA -
2022-12-14 20:48:34 DEBUG : TESTDIR1: Making directory
2022/12/14 20:48:34 -         869 B / 869 B, 100%, 0 B/s, ETA -
2022-12-14 20:48:34 DEBUG : TESTDIR2: Making directory
2022/12/14 20:48:34 -         869 B / 869 B, 100%, 0 B/s, ETA -
2022-12-14 20:48:34 DEBUG : Local file system at /tmp: copied 2 directories
2022/12/14 20:48:34 -         869 B / 869 B, 100%, 0 B/s, ETA -2022/12/14 20:48:34 -         869 B / 869 B, 100%, 0 B/s, ETA -
2022/12/14 20:48:34 INFO  : 2022/12/14 20:48:34 -         869 B / 869 B, 100%, 0 B/s, ETA -
2022/12/14 20:48:34 DEBUG : 10 go routines active
  File: /tmp/permissions.facl
  Size: 869       	Blocks: 8          IO Block: 4096   regular file
Device: 300085h/3145861d	Inode: 3851286     Links: 1
Access: (0644/-rw-r--r--)  Uid: (1000830000/1000830000)   Gid: (1000830000/ UNKNOWN)
Access: 2022-12-14 20:41:10.216955575 +0000
Modify: 2022-12-14 20:41:10.216955575 +0000
Change: 2022-12-14 20:48:34.635775606 +0000
 Birth: -
setfacl: data/lost+found: No such file or directory
setfacl: data/TESTDIR1: Cannot change owner/group: Operation not permitted
setfacl: data/TESTDIR1/file1: Cannot change owner/group: Operation not permitted
setfacl: data/TESTDIR2: Cannot change owner/group: Operation not permitted
setfacl: data/TESTDIR2/file3: Cannot change owner/group: Operation not permitted
setfacl: data/TESTDIR2/file2: Cannot change owner/group: Operation not permitted
setfacl: data/outfile: Cannot change owner/group: Operation not permitted
Rclone completed in 5s

Similar issue where all transfer speeds indicate 0 B/s. At least in both cases there's a summary at the end that shows total time.

Edit: the actual summary is printed out by our wrapper script (active.sh) in the mover itself, not from rclone.

volsync/mover-rclone/active.sh

Line 41 in 5c2c92e

echo "Rclone completed in $(( SECONDS - START_TIME ))s"

Update:

  • Not sure how useful monitoring status of rclone is on the fly - we could potentially grab times but latest will just be for whatever file is being transferred at the time. I do think for speed/transfer info, the summary at the end is perhaps most useful.

    e.g:

    2022/12/15 10:33:54 INFO  : 2022/12/15 10:33:54 -    24.280 MiB / 24.280 MiB, 100%, 1.059 MiB/s, ETA 0s

    Note if we remove the --stats-one-line-date flag we could get an output like this instead at the end:

    2022/12/15 10:42:08 INFO  : 
Transferred:   	  24.280 MiB / 24.280 MiB, 100%, 847.665 KiB/s, ETA 0s
Checks:                 2 / 2, 100%
Transferred:           23 / 23, 100%
Elapsed time:        31.2s

  • One more note: We run an rclone sync and then a separate rclone copy to copy over the permissions.facl file - so we probably want to exclude that from file transfer summaries, but perhaps still catch errors

@JohnStrunk
Copy link
Member Author

Related #554

@tesshuflower
Copy link
Contributor

tesshuflower commented Dec 15, 2022 •
edited
Loading

Some restic info:

Backup:

Starting container
VolSync restic container version: unknown
backup
restic 0.14.0 (v0.3.0-952-g444628f-dirty) compiled with go1.19.3 on linux/amd64
Testing mandatory env variables
== Checking directory for content ===
== Initialize Dir =======
created restic repository f5bccd54c8 at s3:http://minio-api-minio.apps.app-aws-411ga-sno-net2-zp5jq.dev06.red-chesterfield.com/ttest-restic-new

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
=== Starting backup ===
~/DEVFEDORA/tesshuflower/volsync/TESTDATA ~/DEVFEDORA/tesshuflower/volsync/RESTICTESTS
repository f5bccd54 opened (repository version 2) successfully, password is correct
created new cache in /home/tflower/DEVFEDORA/tesshuflower/volsync/RESTICTESTS/CACHE
no parent snapshot found, will read all files

Files:          25 new,     0 changed,     0 unmodified
Dirs:            3 new,     0 changed,     0 unmodified
Added to the repository: 12.941 MiB (12.529 MiB stored)

processed 25 files, 36.658 MiB in 0:12
snapshot 0ff74383 saved
~/DEVFEDORA/tesshuflower/volsync/RESTICTESTS
=== Starting forget ===
Restic completed in 18s
=== Done ===

restore:

Starting container
VolSync restic container version: unknown
restore
restic 0.14.0 (v0.3.0-952-g444628f-dirty) compiled with go1.19.3 on linux/amd64
Testing mandatory env variables
=== Starting restore ===
~/DEVFEDORA/tesshuflower/volsync/RESTICTESTS/RESTOREDIR ~/DEVFEDORA/tesshuflower/volsync/RESTICTESTS
Selected restic snapshot with id: 0ff74383
repository f5bccd54 opened (repository version 2) successfully, password is correct
restoring <Snapshot 0ff74383 of [/home/tflower/DEVFEDORA/tesshuflower/volsync/TESTDATA] at 2022-12-15 11:10:01.858799017 -0500 EST by tflower@volsync> to .
~/DEVFEDORA/tesshuflower/volsync/RESTICTESTS
Restic completed in 9s
=== Done ===
  • For Restic it also looks like trying to extract time/transfer data while in-progress might not be worthwhile - there is a bit of a summary at the end however.

When turning up the verbosity there is an updating line that shows progress of the file in-transfer (similar to --progress with Rclone) and we do get a bit more info in the summary. Here's an example restic backup with --verbose=3 (the max level):

Starting container
VolSync restic container version: unknown
backup
restic 0.14.0 (v0.3.0-952-g444628f-dirty) compiled with go1.19.3 on linux/amd64
Testing mandatory env variables
== Checking directory for content ===
== Initialize Dir =======
created restic repository bdc70339d1 at s3:http://minio-api-minio.apps.app-aws-411ga-sno-net2-zp5jq.dev06.red-chesterfield.com/ttest-restic-new

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
=== Starting backup ===
~/DEVFEDORA/tesshuflower/volsync/RESTICTESTS/RESTOREDIR ~/DEVFEDORA/tesshuflower/volsync/RESTICTESTS
open repository
repository bdc70339 opened (repository version 2) successfully, password is correct
created new cache in /home/tflower/DEVFEDORA/tesshuflower/volsync/RESTICTESTS/CACHE
lock repository
no parent snapshot found, will read all files
load index files
start scan on [.]
start backup on [.]
scan finished in 0.717s: 25 files, 36.658 MiB
new       /DIR2/os-release, saved in 0.015s (712 B added)
new       /DIR1/config-6.0.7-301.fc37.x86_64, saved in 0.031s (245.633 KiB added)
new       /DIR2/, saved in 0.034s (0 B added, 0 B stored, 400 B metadata)
new       /DIR3/README, saved in 0.034s (359 B added)
new       /DIR1/, saved in 0.040s (0 B added, 0 B stored, 417 B metadata)
new       /DIR3/basic.conf, saved in 0.034s (1.221 KiB added)
new       /DIR3/chrony.conf, saved in 0.027s (132 B added)
new       /DIR3/cyrus-sasl.conf, saved in 0.002s (186 B added)
new       /DIR3/dbus.conf, saved in 0.003s (118 B added)
new       /DIR3/dnsmasq.conf, saved in 0.004s (59 B added)
new       /DIR3/flatpak.conf, saved in 0.004s (38 B added)
new       /DIR3/gamemode.conf, saved in 0.003s (15 B added)
new       /DIR3/gdm.conf, saved in 0.010s (46 B added)
new       /DIR3/gnome-initial-setup.conf, saved in 0.029s (174 B added)
new       /DIR3/kubernetes.conf, saved in 0.032s (124 B added)
new       /DIR3/plocate.conf, saved in 0.015s (12 B added)
new       /DIR3/samba.conf, saved in 0.013s (39 B added)
new       /DIR3/systemd-coredump.conf, saved in 0.003s (335 B added)
new       /DIR3/systemd-journal.conf, saved in 0.003s (316 B added)
new       /DIR3/systemd-network.conf, saved in 0.002s (343 B added)
new       /DIR3/systemd-oom.conf, saved in 0.003s (339 B added)
new       /DIR3/systemd-resolve.conf, saved in 0.003s (333 B added)
new       /DIR3/systemd-timesync.conf, saved in 0.003s (344 B added)
new       /DIR3/tcpdump.conf, saved in 0.002s (207 B added)
new       /DIR3/tpm2-tss.conf, saved in 0.002s (128 B added)
new       /DIR3/aaa.txt, saved in 0.225s (12.802 MiB added)
new       /DIR3/, saved in 0.253s (0 B added, 0 B stored, 1.971 KiB metadata)
new       /vmlinuz-0-rescue-b1184baf4f57425dbd7f28a601c3bab7, saved in 0.123s (0 B added)

Files:          25 new,     0 changed,     0 unmodified
Dirs:            3 new,     0 changed,     0 unmodified
Data Blobs:     34 new
Tree Blobs:      4 new
Added to the repository: 13.062 MiB (12.658 MiB stored)

processed 25 files, 36.658 MiB in 0:18
snapshot 06635130 saved
~/DEVFEDORA/tesshuflower/volsync/RESTICTESTS
=== Starting forget ===
Restic completed in 23s
=== Done ===

@tesshuflower
Copy link
Contributor

Example output from rsync:

source:

VolSync rsync container version: v0.6.0+5d8dcef-dirty
Syncing data to 10.96.145.93:22 ...
.d..t...... ./
cd+++++++++ TESTDATA/
<f+++++++++ TESTDATA/vmlinuz-0-rescue-b1184baf4f57425dbd7f28a601c3bab7
cd+++++++++ TESTDATA/DIR1/
<f+++++++++ TESTDATA/DIR1/config-6.0.7-301.fc37.x86_64
cd+++++++++ TESTDATA/DIR2/
<f+++++++++ TESTDATA/DIR2/os-release
cd+++++++++ TESTDATA/DIR3/
<f+++++++++ TESTDATA/DIR3/README
<f+++++++++ TESTDATA/DIR3/aaa.txt
<f+++++++++ TESTDATA/DIR3/basic.conf
<f+++++++++ TESTDATA/DIR3/chrony.conf
<f+++++++++ TESTDATA/DIR3/cyrus-sasl.conf
<f+++++++++ TESTDATA/DIR3/dbus.conf
<f+++++++++ TESTDATA/DIR3/dnsmasq.conf
<f+++++++++ TESTDATA/DIR3/flatpak.conf
<f+++++++++ TESTDATA/DIR3/gamemode.conf
<f+++++++++ TESTDATA/DIR3/gdm.conf
<f+++++++++ TESTDATA/DIR3/gnome-initial-setup.conf
<f+++++++++ TESTDATA/DIR3/kubernetes.conf
<f+++++++++ TESTDATA/DIR3/plocate.conf
<f+++++++++ TESTDATA/DIR3/samba.conf
<f+++++++++ TESTDATA/DIR3/systemd-coredump.conf
<f+++++++++ TESTDATA/DIR3/systemd-journal.conf
<f+++++++++ TESTDATA/DIR3/systemd-network.conf
<f+++++++++ TESTDATA/DIR3/systemd-oom.conf
<f+++++++++ TESTDATA/DIR3/systemd-resolve.conf
<f+++++++++ TESTDATA/DIR3/systemd-timesync.conf
<f+++++++++ TESTDATA/DIR3/tcpdump.conf
<f+++++++++ TESTDATA/DIR3/tpm2-tss.conf

Number of files: 31 (reg: 26, dir: 5)
Number of created files: 29 (reg: 25, dir: 4)
Number of deleted files: 0
Number of regular files transferred: 25
Total file size: 38.44M bytes
Total transferred file size: 38.44M bytes
Literal data: 38.44M bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 37.57M
Total bytes received: 529

sent 37.57M bytes  received 529 bytes  25.05M bytes/sec
total size is 38.44M  speedup is 1.02
Rsync completed in 1s
Synchronization completed successfully. Notifying destination...
Initiating shutdown. Exit code: 0

destination:

VolSync rsync container version: v0.6.0+5d8dcef-dirty
Waiting for connection...
Exiting... Exit code: 0

@tesshuflower tesshuflower self-assigned this Jan 9, 2023
@tesshuflower tesshuflower mentioned this issue Jan 12, 2023
Merged
@tesshuflower tesshuflower mentioned this issue Jan 23, 2023
Merged
@tesshuflower
Copy link
Contributor

Initial implementation is complete - closing this now - will open new issues if fine-tuning is required in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tesshuflower tesshuflower

Labels
enhancement New feature or request
Projects
VolSync project tracking
Archived in project
Milestone
Version 0.7
Development

No branches or pull requests
2 participants
@JohnStrunk @tesshuflower