Using Volume Shadow Copy Service (VSS) with rclone

This is an example of how you can get rclone to read from a volume shadow copy.

It is basically a summary of my documentation at github.com/albertony/vss.

Volume Shadow Copy Service (VSS)

From Wikipedia:

Shadow Copy (also known as Volume Snapshot Service, Volume Shadow Copy Service or VSS) is a technology included in Microsoft Windows that can create backup copies or snapshots of computer files or volumes, even when they are in use. It is implemented as a Windows service called the Volume Shadow Copy service. A software VSS provider service is also included as part of Windows to be used by Windows applications. Shadow Copy technology requires either the Windows NTFS or ReFS filesystems in order to create and store shadow copies. Shadow Copies can be created on local and external (removable or network) volumes by any Windows component that uses this technology, such as when creating a scheduled Windows Backup or automatic System Restore point.

From Microsoft in Windows Developer Center:

The Volume Shadow Copy Service (VSS) is a set of COM interfaces that implements a framework to allow volume backups to be performed while applications on a system continue to write to the volumes.

VSS with rclone

The intention of integrating rclone with Microsoft Volume Shadow Copy Service (VSS) is to make the uploading of large file sets to cloud storage more robust. It is, of course, not intended to be used as a backup solution, not any more than rclone itself is! Some of the same challanges apply for syncing/transferring large file sets, as with backup, so with some "edits" we can use the description from Microsoft TechNet Library:

Backing up and restoring Syncing critical business data can be very complex due to the following issues:

• The data usually needs to be backed up synced while the applications that produce the data are still running. This means that some of the data files might be open or they might be in an inconsistent state.
• If the data set is large, it can be difficult to back up sync all of it at one time.

Using VShadow

VShadow is a command-line tool from Microsoft that you can use to query, create and manage many aspects of volume shadow copies. It is not included in Windows, but in Windows SDK which you would normally install as part of Visual Studio, but can also download and install from Microsoft Developer Downloads. This utility can be used to integrate external utilities like rclone with VSS. It is not an entirely straight forward proces, nor a very elegant solution, but it will most likely get you to a solution that you can live with.

At github.com/albertony/shadowrun#using-vshadow I have discussed in detail how to use vshadow, with rclone as a case. Will just refer to the end result here: Let's say that we want to sync a directory C:\Data\ to the cloud. Using rclone you would normally execute rclone sync C:\Data\ remote:Data. To make this command read source files from a VSS snapshot, start by creating a batch script with the following content:

setlocal enabledelayedexpansion
call "%~dp0setvars.cmd" || set exit_code=!errorlevel!&&goto end
mklink /d C:\Snapshot\ %SHADOW_DEVICE_1%\ || set exit_code=!errorlevel!&&goto end
subst T:\ C:\Snapshot\ || set exit_code=!errorlevel!&&goto remove_link
rclone sync T:\Data\ remote:Data
set exit_code=%errorlevel%
subst T: /d
:remove_link
rmdir C:\Snapshot\
:end
del "%~dp0setvars.cmd"
exit /b %exit_code%

Then, if you save this script as a file exec.cmd, execute it via the VShadow tool with the following command from the same directory:

vshadow.exe -nw -script=setvars.cmd -exec=exec.cmd C:

Or, if you want a single-click solution; create a second batch script in the same directory, which executes this command - but with full paths of both file references to avoid issues with changing working directory:

vshadow.exe -nw -script="%~dp0setvars.cmd" -exec="%~dp0exec.cmd" C:

Using ShadowRun

The solution based on Microsoft's VShadow utility works fine, but is not extremely elegant. It suffers from the tool being a showcase for a lot of features around VSS. I decided to create my own variant of VShadow which I called ShadowRun. It focuses on a single very specific feature: The creation of temporary read-only shadow copy and running of a specified utility that can work on this, before everything is automatically cleaned up. (ShadowRun is still very much based on the original source from Microsoft, it is not a complete rewrite; mostly just a few extra features added, and a lot of irrelevant features removed).

As described at github.com/albertony/vss/shadowrun/README.md#using-shadowrun-with-rclone: To use the same example as with VShadow above, where we wanted to sync a directory C:\Data\ to the cloud, and by use of rclone one would normally execute rclone sync C:\Data\ remote:Data.

Now instead of all the steps and batch scripting needed to make this work with VSS using the original vshadow utility, we can be able to do this very easy with shadowrun:

shadowrun.exe -env -mount -exec=C:\Tools\Rclone\rclone.exe C: -- sync %SHADOW_DRIVE_1%\Data\ remote:Data

Not only is it clean and simple, we can also check the exit code which will be the one from rclone.