Repository navigation
Wim: do one thing (backup my C:) and do it right #364
fcorbelli
announced in
Announcements
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
A "side effect" of the not-so-good review is... wim's review 😄
-wim: incremental images of NTFS that read only what changed
Or I hope so...
Available from zpaqfranz 65.21, Windows only.
In short
You make an image of a Windows drive (
ac, ora X: -image) every day, into the same archive. The second and every following run of-wimreads only what changed since the previous one, instead of the whole drive. The backup takes seconds instead of minutes.On my PC (C: with about 200 GB of data, NVMe), the daily image of C: went from about 4 minutes and 17 seconds to about 22 seconds: some 6 GB read instead of 200.
-wim, and stores what the next run needs. From the second one on, only the changes (and the file system metadata) are read.xextracts it,xcrestores it,tchecks it.-wimsaves is time and disk reads, not space.-until Nuses the state of version N. After arepack, everything is read again, once.Using it
Where it works
-wimacts on an NTFS image that zpaqfranz reads from a copy that cannot change while it is read:acand-imageask for one by themselves (zpaqfranz must run as administrator, as for any image);Everything else (FAT32, exFAT, ReFS, a drive that could be neither copied nor locked) is imaged as always, with every block read, and one line says so:
The skip uses the multithreaded way of adding (the default). With
-noturbo,-t1,-verifyor-collisionthe image is made as without-wim.What you see
By default, three lines per image:
Add
-verbosefor the full report: why each part was read, the times of each phase, the threads, the pieces of 64 KB.-wimcheck
The blocks that
-wimwould skip are read anyway and compared with what is in the archive. It is as slow as an image without-wim, and it is the way to verify, on your own machine, that nothing is being missed.-wimcheck OKlines, for the skipped blocks and the skipped pieces.Once in a while (say, once a week) is enough.
The first -wim run on an old archive
With
-wimthe image is cut into fragments in a slightly different way (see The stream). On an archive made without-wim, the first-wimrun therefore stores most of the image again, once. Better: start a new archive for-wim, and keep using-wimon it.How it works
The starting point: what an image of zpaqfranz is
An image made by
-image(and byac) is a VHD of the partition: zpaqfranz reads the$Bitmapof the volume, reads the used clusters, and writes zeros in place of the free ones. The VHD is "dynamic": its data is a sequence of blocks of 2 MB, each one preceded by 512 bytes of bitmap.That stream goes into the archive like any other file: cut into fragments by the content, each fragment identified by its SHA-1, stored only if it is not already there. This is why a second image of an unchanged drive costs almost nothing in space.
In time, however, it costs as much as the first: to know that a block did not change, zpaqfranz had to read it and compute its fragments. 200 GB read to discover that 199 of them are the same as yesterday.
-wimanswers a different question: which blocks of 2 MB might have changed, without reading them? NTFS knows. Everything is written in the$MFT.Reading the $MFT of the shadow copy
After the shadow copy is made,
-wimreads the$MFTof the copy (the same frozen moment the image will be read from), in one pass, with threads that read ahead.For every record in use it takes:
$STANDARD_INFORMATION(creation, modification and MFT change times, attributes; not the access time, which changes on a simple read), the USN (the number of the last record of the change journal for that file), the sequence number;From the runs, every cluster of the volume is assigned to its owner, so every block of 2 MB of the image knows which files (and which attributes) have data in it.
The signature of a block
For every block, a signature of 64 bits: the sum of the XXH64 of everything that describes what is inside it. For every stretch of a file in the block: the state of the file, the record, the type and name of the attribute, VCN, LCN, length, the block. The sum does not depend on the order in which the records are visited.
If a file is written, NTFS updates its modification time and its USN: the signature of every block holding its clusters changes. If a file is moved (defragmentation), its LCN changes. If it is deleted, its clusters become free (zeros in the image) and the signature changes again. A new file appears in the blocks where its clusters are.
A block with the same signature as in the previous run is not read. Its fragments of the previous version go into the image in its place.
What is always read
Some data changes without leaving a trace in the signatures. It is always read:
$MFT,$MFTMirr,$LogFile,$Bitmap...), everything under$Extend(the USN journal,$ObjId,$Reparse...), every attribute that is not$DATA(above all the indexes of the folders), and the clusters marked as used that belong to nobody.hivelist), the log files of the ETW sessions (event tracing), and the files open for writing, found by listing the handles of all processes. Taken twice, just before and just after the shadow copy.$MFTitself is read anyway to compute the signatures: its pieces of 64 KB are compared by content with the previous run, and the ones that did not change are skipped.The stores of the shadow copies (the files in
System Volume Informationthat hold the older copies) are instead left out, written as zeros: they are useless in a restore, they change all the time, and they can be huge. They are found from the$MFTof the copy.Why always read the folder indexes, instead of trusting the USN journal? Because I measured it: NTFS rewrites the index blocks of a folder (to update the size and the dates of the files it lists) without logging it and without a record in the change journal. On my PC, about one run in two had at least one index block changed in this way, in folders whose files had no record in the journal at all. An index that is not read would be an index restored wrong.
Pieces of 64 KB
Reading a whole block of 2 MB because of a few KB of metadata would be a waste: on a system drive, metadata is spread across thousands of blocks.
So a block with metadata is cut into 32 pieces of 64 KB (the first one with the 512 bytes of the bitmap), each with its own signature and its own mark of "metadata". Only the pieces with metadata, or with a new signature, are read. The others take their fragments of the previous version.
A block without metadata that changes is read whole the first time, and cut into pieces too. From its next change on, only its pieces with a new signature are read. This helps a lot with the blocks shared by many small files (logs, mailboxes, caches): one file that changes no longer makes the whole block be read again.
The stream of the image, cut block by block
To reuse the fragments of a block, a block that did not change must give exactly the same fragments as before, wherever it is in the stream. Normally the fragmenter of zpaq cuts by content, and the cut points depend on what comes before.
With
-wim, every element of the stream (512 bytes of bitmap + one block of 2 MB, or one piece of 64 KB) is cut on its own: the fragmenter starts fresh at its beginning, and there is always a cut at its end. A block that did not change has the same fragments, by construction.This is why an archive made with
-wimdiffers from one made without it.Reading fast what must be read
The blocks and pieces to read are known in advance, so 8 threads (each with its own handle to the shadow copy) read ahead up to 16 blocks, in reads of up to 512 KB. A read that fails takes the usual way of the image (with its error handling, zeros,
-rescue).The hash of the image
The SHA-1 (or XXH3, or whatever hash is in use) of the image file needs all its bytes, and the skipped blocks are not read. So an image with skipped blocks is stored with its CRC-32 only: the CRC-32 of every element is kept in the state, and the CRC-32 of the whole image is computed by combining them, without the data.
tchecks it as usual.xandxcaccept an image with no hash.The state of -wim: base + changes
To skip a block, the next run needs, for every block of the previous one: its signature, the numbers of its fragments in the archive, its CRC-32, its hash, and the same for its pieces of 64 KB. This is the state.
Inside the archive
The state is written in the archive, as two files of the image, next to the VHD:
In the state, by block: number, flags (metadata, reusable, cut in pieces, no hash), signature, hash, CRC-32, the number of fragments and their numbers (as differences, in variable length integers); for every piece, the same. In the head: the serial number of the volume, cluster and block sizes, the date, and a hash of all the SHA-1 of all the fragments named.
Before a fragment is reused, zpaqfranz checks that it is in this archive, and the same:
-until N(an archive rolled back) uses the state of version N, not the last one;repackthe fragments are renumbered: the hash of the SHA-1 does not match, and everything is read again, once (44059);trim), is read again.Old development versions wrote the state in a file next to the archive. If there is one, it is read once; from then on, the state lives only in the archive.
Why a base and changes
The state is not small: about 3.5 MB on a test VM with 40 GB of data, about 24 MB on my PC. Written whole in every version, it would cost the archive several MB each time: few records change, but they are spread all over the file, and deduplication by fragments of tens of KB cannot catch scattered changes.
So:
image_X.wimis the base: written byte by byte the same as in the previous version, as long as possible. For zpaq it is the same file: it costs a line in the index, nothing more.image_X.wimdholds the changes: the records of the blocks that are no longer as in the base, plus the blocks that disappeared. Always relative to the base (cumulative), not to the previous version: the two files of the last version are all the next run needs.On the test VM, the state went from about 1.5 MB per version (written whole) to 70-250 KB per version.
How it was tested
During the development,
-wimcheckwas run many times on my PC (in daily use, with a browser, mail, and everything else running), on a Windows 11 VM (GPT), and on a Windows 10 VM (MBR, with a logical partition in an extended one).-wimarchive (with blocks skipped over several versions) was compared with a fresh image made without-wim. The only differences: the about 40 bytes that change in every image anyway (the head and the tail of the VHD, the signature of the MBR).-untilandrepack: a rolled back archive uses the state of its version; a repacked one reads everything once, then skips again.ton every version of every archive.The cases
-wimcheckcaught, and that became rules:NTUSER.DAT.LOGand a log of SleepStudy (ETW): files rewritten in place without any change to their record. Since then, hives, their logs, ETW logs and files open for writing are always read.$MFTof the copy.Where the remaining time goes
On my PC, a run of about 22 seconds:
$MFTand the signatures of ~102,000 blocksac, closing the archive)What is read is now almost all metadata: about 2.5 GB of folder indexes, about 1.5 GB of files always read, a few hundred MB of system files, and only a few hundred MB of really changed data.
The future
I do not rule out trying smarter ways to reduce the data to read to the minimum: the reasons in the USN journal, looking inside the folder indexes, finer pieces. Each of them saves some more seconds.
But first,
-wimhas to be stable in the sense of taking all the data, every time, on every machine: an image that skips one changed block is worse than a slow one. The goal of this first version is that, more than the maximum theoretical efficiency.-wimcheckis there for this: if it ever says something in red, I want to know.All reactions