Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Volume Shadow Copy Service (VSS) on windows #2274

Open
wants to merge 11 commits into
base: master
from

Conversation

@fgma
Copy link

commented May 13, 2019

What is the purpose of this change? What does it change?

Add transparent support for Volume Shadow Copy Service (VSS) on windows.

Use of VSS needs to be activated via the new flag --use-windows-vss for the backup command e.g.

restic backup --verbose --use-windows-vss data

To easily test the new feature I prepared some bat/powershell files that will run restic with the new flag and also get an exclusive lock on a file via the included powershell script to test VSS allows reading the file:
vss_test.zip

Right now it is not a finished pull request. It is missing many details:

  • proper logging
  • code cleanup
  • testing

But basic VSS functionality should work!

Right now I've tested it only on Windows 10 Professional 64 bit.

Was the change discussed in an issue or in the forum before?

closes #340

Checklist

  • I have read the Contribution Guidelines
  • I have added tests for all changes in this PR
  • I have added documentation for the changes (in the manual)
  • There's a new file in changelog/unreleased/ that describes the changes for our users (template here)
  • I have run gofmt on the code in all commits
  • All commit messages are formatted in the same style as the other commits in the repo
  • I'm done, this Pull Request is ready for review

@fgma fgma force-pushed the fgma:master branch from 83d08f3 to e2d5f9b May 15, 2019

@codecov-io

This comment has been minimized.

Copy link

commented May 15, 2019

Codecov Report

Merging #2274 into master will decrease coverage by 4.31%.
The diff coverage is 0%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2274      +/-   ##
==========================================
- Coverage   51.09%   46.78%   -4.32%     
==========================================
  Files         178      180       +2     
  Lines       14546    14617      +71     
==========================================
- Hits         7433     6839     -594     
- Misses       6042     6756     +714     
+ Partials     1071     1022      -49
Impacted Files Coverage Δ
internal/fs/fs_local_vss.go 0% <0%> (ø)
cmd/restic/cmd_backup.go 44.03% <0%> (-0.83%) ⬇️
internal/fs/vss.go 0% <0%> (ø)
internal/backend/b2/b2.go 0% <0%> (-80.69%) ⬇️
internal/backend/swift/swift.go 0% <0%> (-78.83%) ⬇️
internal/backend/gs/gs.go 0% <0%> (-74%) ⬇️
internal/backend/azure/azure.go 0% <0%> (-69.46%) ⬇️
internal/backend/swift/config.go 34.69% <0%> (-57.15%) ⬇️
internal/archiver/blob_saver.go 95.06% <0%> (-4.94%) ⬇️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5bd5db4...feff626. Read the comment docs.

fgma added some commits May 15, 2019

@fgma fgma force-pushed the fgma:master branch from 8cd655a to 7bb4b34 May 16, 2019

@fgma

This comment has been minimized.

Copy link
Author

commented May 17, 2019

Right now this pull request reached a state where I'd like to get some feedback. I'm interested in test results but also feedback concerning the code to finish this pull request.

@fbkopp

This comment has been minimized.

Copy link

commented May 21, 2019

two consecutives run give me error with thunderbird open

`
C:\User\restic>restic -r c:\temp\r backup C:\Users\User\AppData\Roaming\Thunderbird\Profiles\9ceckvqy.default --use-windows-vss
enter password for repository:
repository f82f6e2c opened successfully, password is correct
creating VSS snapshot for [C:]
Fatal: unable to save snapshot: Lstat: GetFileInformationByHandle \?\GLOBALROOT\Device\HarddiskVolumeShadowCopy7: Função incorreta.io/AppData/Roaming/Thunderbird/Profiles/9ceckvqy.default/webappsstore.sqlite
/C/Users/User/AppData/Roaming/Thunderbird/Profiles/9ceckvqy.default/training.dat

C:\User\restic>restic -r c:\temp\r backup C:\Users\User\AppData\Roaming\Thunderbird\Profiles\9ceckvqy.default --use-windows-vss
enter password for repository:
repository f82f6e2c opened successfully, password is correct
creating VSS snapshot for [C:]
[1:16] 12796 files 3.721 GiB, total 12800 files 3.721 GiB, 0 errors
C:\User\restic>ppData/Roaming/Thunderbird/Profiles/9ceckvqy.default/webappsstore.sqlite
/C/Users/User/AppData/Roaming/Thunderbird/Profiles/9ceckvqy.default/webappsstore.sqlite-shm
`
and the snapshot is not saved

@fgma

This comment has been minimized.

Copy link
Author

commented May 22, 2019

@fbkopp Can you try to backup using a relative path to your thunderbird folder?

It seems like there is some strange behavior when using an absolute path. There is a problem when accessing the volume root folder e.g. c:\ inside the VSS snapshot which is only done by restic when an absolute path is given.

@fgma

This comment has been minimized.

Copy link
Author

commented May 22, 2019

@fbkopp I've updated the pull request which hopefully fixes your problem. Please give it a try.

@fbkopp

This comment has been minimized.

Copy link

commented May 23, 2019

@fbkopp Can you try to backup using a relative path to your thunderbird folder?

It seems like there is some strange behavior when using an absolute path. There is a problem when accessing the volume root folder e.g. c:\ inside the VSS snapshot which is only done by restic when an absolute path is given.

Okay, it worked now, but the repository is locked from the backup.



C:\User\restic>rd /q/s c:\Temp\r

C:\User\restic>restic.exe -r c:\Temp\r init
enter password for new repository:
enter password again:
created restic repository c6ab143bf7 at c:\Temp\r

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

C:\User\restic>restic.exe -r c:\Temp\r backup "c:\Users\User\AppData\Roaming\Thunderbird" --use-windows-vss
enter password for repository:
repository c6ab143b opened successfully, password is correct
created new cache in C:\Users\User\AppData\Local\restic
found 1 old cache directories in C:\Users\User\AppData\Local\restic, pass --cleanup-cache to remove them
creating VSS snapshot for [c:\]

Files:       12951 new,     0 changed,     0 unmodified
Dirs:            5 new,     0 changed,     0 unmodified
Added to the repo: 3.669 GiB

processed 12951 files, 3.736 GiB in 1:10
snapshot 7c35f491 saved

C:\User\restic>restic.exe -r c:\Temp\r snapshots
enter password for repository:
repository c6ab143b opened successfully, password is correct
found 1 old cache directories in C:\Users\User\AppData\Local\restic, pass --cleanup-cache to remove them
ID        Time                 Host        Tags        Paths
--------------------------------------------------------------------------------------------------
7c35f491  2019-05-23 11:23:18  User-VM2               c:\Users\User\AppData\Roaming\Thunderbird
--------------------------------------------------------------------------------------------------
1 snapshots

C:\User\restic>cd c:\Users\User\AppData\Roaming\

c:\Users\User\AppData\Roaming>c:\User\restic\restic.exe -r c:\temp\r backup Thunderbird --use-windows-vss
enter password for repository:
repository c6ab143b opened successfully, password is correct
found 1 old cache directories in C:\Users\User\AppData\Local\restic, pass --cleanup-cache to remove them
creating VSS snapshot for [c:\]

Files:       12952 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 3.127 MiB

processed 12952 files, 3.736 GiB in 0:30
snapshot 9c60430a saved

c:\Users\User\AppData\Roaming>c:\User\restic\restic.exe -r c:\temp\r snapshots
enter password for repository:
repository c6ab143b opened successfully, password is correct
found 1 old cache directories in C:\Users\User\AppData\Local\restic, pass --cleanup-cache to remove them
ID        Time                 Host        Tags        Paths
--------------------------------------------------------------------------------------------------
7c35f491  2019-05-23 11:23:18  User-VM2               c:\Users\User\AppData\Roaming\Thunderbird
9c60430a  2019-05-23 11:25:36  User-VM2               c:\Users\User\AppData\Roaming\Thunderbird
--------------------------------------------------------------------------------------------------
2 snapshots

c:\Users\User\AppData\Roaming>

c:\Users\User\AppData\Roaming>c:\User\restic\restic.exe -r c:\temp\r check --read-data
using temporary cache in C:\Users\User\AppData\Local\Temp\restic-check-cache-308975783
enter password for repository:
repository c6ab143b opened successfully, password is correct
created new cache in C:\Users\User\AppData\Local\Temp\restic-check-cache-308975783
create exclusive lock for repository
Fatal: unable to create lock in backend: repository is already locked by PID 2788 on User-VM2 by User-VM2\User (UID 0, GID 0)
lock was created at 2019-05-23 11:25:38 (1h40m28.818369s ago)
storage ID 2548f0d0

@fgma

This comment has been minimized.

Copy link
Author

commented May 23, 2019

@fbkopp I reproduced the issue. The problem is related to runBackup() in cmd/restic/cmd_backup.go:

lock, err := lockRepo(repo)
defer unlockRepo(lock)

For testing I've added a debug message to unlockRepo(). When using VSS snapshots the defer statement seems to be ignored from time to time but it works when not using VSS snapshots. I'm not sure about the actual reason for this behavior.

@fgma

This comment has been minimized.

Copy link
Author

commented May 26, 2019

@fbkopp It took me a while to narrow it down. I hope I fixed it in b7aafd4. Can you please give it a try?

@fgma fgma force-pushed the fgma:master branch from 86c179e to cad6374 May 27, 2019

@fbkopp

This comment has been minimized.

Copy link

commented May 27, 2019

@fbkopp It took me a while to narrow it down. I hope I fixed it in b7aafd4. Can you please give it a try?

@fgma, created a new repository, now worked with the last change.
I'll do some more tests to make sure it's working.

C:\User\restic>restic.exe -r c:\Temp\repo backup "c:\Users\User\AppData\Roaming\Thunderbird" --use-windows-vss
enter password for repository:
repository a6b6d175 opened successfully, password is correct
created new cache in C:\Users\User\AppData\Local\restic
found 1 old cache directories in C:\Users\User\AppData\Local\restic, pass --cleanup-cache to remove them
creating VSS snapshot for [c:\]

Files:       12996 new,     0 changed,     0 unmodified
Dirs:            5 new,     0 changed,     0 unmodified
Added to the repo: 3.637 GiB

processed 12996 files, 3.738 GiB in 1:04
snapshot a6e94dd3 saved

C:\User\restic>restic.exe -r c:\Temp\repo snapshots
enter password for repository:
repository a6b6d175 opened successfully, password is correct
found 1 old cache directories in C:\Users\User\AppData\Local\restic, pass --cleanup-cache to remove them
ID        Time                 Host        Tags        Paths
--------------------------------------------------------------------------------------------------
a6e94dd3  2019-05-27 14:52:44  User-VM2               c:\Users\User\AppData\Roaming\Thunderbird
--------------------------------------------------------------------------------------------------
1 snapshots

C:\User\restic>restic.exe -r c:\Temp\repo check --read-data
using temporary cache in C:\Users\User\AppData\Local\Temp\restic-check-cache-813659931
enter password for repository:
repository a6b6d175 opened successfully, password is correct
created new cache in C:\Users\User\AppData\Local\Temp\restic-check-cache-813659931
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
read all data
[0:19] 100.00%  710 / 710 items
duration: 0:19
no errors were found

C:\User\restic>restic.exe -r c:\Temp\repo backup "c:\Users\User\AppData\Roaming\Thunderbird" --use-windows-vss
enter password for repository:
repository a6b6d175 opened successfully, password is correct
found 1 old cache directories in C:\Users\User\AppData\Local\restic, pass --cleanup-cache to remove them
creating VSS snapshot for [c:\]

Files:           0 new,     7 changed, 12989 unmodified
Dirs:            0 new,     5 changed,     0 unmodified
Added to the repo: 5.322 MiB

processed 12996 files, 3.738 GiB in 0:13
snapshot 5746edec saved

C:\User\restic>restic.exe -r c:\Temp\repo snapshots
enter password for repository:
repository a6b6d175 opened successfully, password is correct
found 1 old cache directories in C:\Users\User\AppData\Local\restic, pass --cleanup-cache to remove them
ID        Time                 Host        Tags        Paths
--------------------------------------------------------------------------------------------------
a6e94dd3  2019-05-27 14:52:44  User-VM2               c:\Users\User\AppData\Roaming\Thunderbird
5746edec  2019-05-27 14:54:51  User-VM2               c:\Users\User\AppData\Roaming\Thunderbird
--------------------------------------------------------------------------------------------------
2 snapshots

C:\User\restic>restic.exe -r c:\Temp\repo check --read-data
using temporary cache in C:\Users\User\AppData\Local\Temp\restic-check-cache-475815227
enter password for repository:
repository a6b6d175 opened successfully, password is correct
created new cache in C:\Users\User\AppData\Local\Temp\restic-check-cache-475815227
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
read all data
[0:16] 100.00%  713 / 713 items
duration: 0:16
no errors were found

C:\User\restic>
@fgma

This comment has been minimized.

Copy link
Author

commented Jun 2, 2019

@fbkopp Do you have any additional feedback?

@bherila

This comment has been minimized.

Copy link

commented Jun 2, 2019

I will just add that I am super excited for this and hopeful that we can see it merged soon! Thank you for your hard work!

@jryan128

This comment has been minimized.

Copy link

commented Jun 5, 2019

I downloaded your fork and am testing it too with a new restic repo, I'll let you know how it goes.

@jryan128

This comment has been minimized.

Copy link

commented Jun 5, 2019

Tried to use it with an external drive (which I know might not make too much sense) and got this error:

PS C:\Users\XXXX\Desktop\restic-with-vss\cmd\restic> .\restic-vss.exe -r b2:XXXX:XXXX backup --verbose --use-windows-vss "D:\"
open repository
enter password for repository:
repository XXXXX opened successfully, password is correct
lock repository
load index files
start scan on [D:\]
start backup on [D:\]
creating VSS snapshot for [D:\]
successfully created snapshot for [D:\]
scan: Readdirnames D:\ failed: Readdir \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy6: The system cannot find the path specified.
error: Readdirnames D:\ failed: Readdir \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy6: The system cannot find the path specified.
scan finished in 6.276s: 0 files, 0 B
Fatal: unable to save snapshot: snapshot is empty
@jryan128

This comment has been minimized.

Copy link

commented Jun 5, 2019

Putting a path after the D:\ fixed the problem. So:

.\restic-vss.exe -r b2:XXXX:XXXX backup --verbose --use-windows-vss "D:\"
vs
.\restic-vss.exe -r b2:XXXX:XXXX backup --verbose --use-windows-vss "D:\somefolder\"

@fbkopp

This comment has been minimized.

Copy link

commented Jun 8, 2019

Putting a path after the D:\ fixed the problem. So:

.\restic-vss.exe -r b2:XXXX:XXXX backup --verbose --use-windows-vss "D:\"
vs
.\restic-vss.exe -r b2:XXXX:XXXX backup --verbose --use-windows-vss "D:\somefolder\"

Same issue here.

F:\>restic backup d:\ --use-windows-vss
repository 197b524c opened successfully, password is correct
created new cache in C:\Users\Administrator\AppData\Local\restic
creating VSS snapshot for [d:\]
error: Readdirnames d:\ failed: Readdir \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy80: The system cannot find the path specified.
Fatal: unable to save snapshot: snapshot is empty
@fgma

This comment has been minimized.

Copy link
Author

commented Jun 8, 2019

@fbkopp @jryan128

I hope I fixed it in feff626

@fbkopp

This comment has been minimized.

Copy link

commented Jun 10, 2019

@fbkopp @jryan128

I hope I fixed it in feff626

It's working for me.

@adamsewell

This comment has been minimized.

Copy link

commented Aug 1, 2019

I'd love to see this merged! +1

@nixzahlen

This comment has been minimized.

Copy link

commented Aug 3, 2019

Greatly desired / awaited / appreciated feature here.
If it's also implementing the Excluding Files from Shadow Copies, it would be a "dream coming true" ;)

@fgma

This comment has been minimized.

Copy link
Author

commented Aug 7, 2019

@nixzahlen Thanks for your feedback. Excluding files might be a good addition when the basic functionality got merged.

Can anyone give me a hint how to progress with this pull-request?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.