Implemented backup based on changes recorded in NTFS USN journal

[dgehri]

First implementation of NTFS USN journal optimization.

• Folder renames are correctly handled (old name is removed, new one added)
• I don't actually rely on the USN journal to determine if files have changed. Instead, I use it to filter the source list (reduce it) to the set of possibly modified files and folders, which are then scanned recursively using FilterHandler.EnumerateFilesAndFolders(). That way, there is no risk of incorrectly interpreting the journal entries in complex rename / delete / re-create scenarios. Whenever a file / folder is "touched" in the journal, it will be fully scanned.
• The state of the USN journal is recorded in a new table ChangeJournalData in the local database. This table also records a hash value for each fileset, representing the active source files and filter set. An initial full scan is re-triggered whenever the backup configuration is modified.
• A full scan is also triggered if the USN journal is incomplete (has been overwritten / truncated), and of course in case of errors.
• In case of DB loss, recovery, the USN data is not recovered, and a full scan is triggered to establish a sane baseline for subsequent backups.

TODO:

The USN journal records a limited amount of changes, and if backups are spaced too far apart, full scans are required as the data will be incomplete. This has the following implications:

• Frequent (realtime) backups avoid this problem. If nothing has changed, the fileset will be compacted. Duplicati may need optimizing if this becomes a common scenario (compacting before uploading).
• Frequent backups result in numerous filesets, and this may interfere with retention policy. Maybe for the current day, many more filesets would make sense in the "automatic" retention policy mode, to avoid deleting changing data.
• Less frequent backups with USN support could be made possible by scanning the USN journal at regular intervals, and recording the changes, using a process / thread separate from the backup thread. When the backup is run, this data is then used instead of reading the journal at that time. There is no risk that modifications will be missed during reboots / Duplicati not running, as the USN numbers allow us to ensure that the journal is recorded w/o gaps.

[duplicatibot]

This pull request has been mentioned on Duplicati. There might be relevant details there:

https://forum.duplicati.com/t/real-time-backup/263/38

[piegamesde]

I think you can close the pull request and reopen it again if you don't want it to be merged at the moment.

[dgehri]

The pull request is now in sync with the latest canary.

[kenkendk]

Ok, all looks good.
For future commits, it is much easier to review if refactoring and actual changes are two different commits.

I added some logging that was not previously there, and made some minor fixes to the MD5 utility class:
https://github.com/duplicati/duplicati/tree/feature/usn-support-update

I did have one problem, namely that the EnumerateRecords function crashes the Mono compiler when it tries to build the enumeration class. This is quite strange as enumerator methods are used in many places, but for some reason this particular function is causing problems. It might be related to the type being enumerated over is private. In any case, I rewrote the function to use a manual instantiated IEnumerable<Record> (not as pretty, but at least the compiler does not crash).

Can you review/test my changes and then I will merge them into master, closing this PR in the progress?

[dgehri]

Kenneth, Thanks for reviewing, and yes, I will test your changes right now. I know about the refactoring. The issue was that I needed the Snapshot classes to behave differently, namely, that they Enumeration method takes the list of folders to enumerate, and not the constructor. This resulted in a big code change, and that's when I realized that I can minimize new duplicate code by refactoring. I'll try to keep things separate in the future. Not related to USN, and regarding snapshots, by the way: this takes almost 5' on my PC, and in fact, it would be sufficient to use the VSS only for files that cannot be backed up directly (because they are open). An additional option "on demand" would be nice, where the snapshot service is used only on a per-file basis (not per drive), whenever a file cannot be accessed otherwise. I think that would further speed up things. That said, USN already reduces backup time on my server from roughly 30 minutes to about 2-3, and only a few seconds of this are for enumating files. It's close to real-time. - Daniel ᐧ

…

On Wed, May 9, 2018 at 10:54 AM, Kenneth Skovhede ***@***.***> wrote: Ok, all looks good. For future commits, it is much easier to review if refactoring and actual changes are two different commits. I added some logging that was not previously there, and made some minor fixes to the MD5 utility class: https://github.com/duplicati/duplicati/tree/feature/usn-support-update I did have one problem, namely that the EnumerateRecords function crashes the Mono compiler when it tries to build the enumeration class. This is quite strange as enumerator methods are used in many places, but for some reason this particular function is causing problems. It might be related to the type being enumerated over is private. In any case, I rewrote the function to use a manual instantiated IEnumerable<Record> (not as pretty, but at least the compiler does not crash). Can you review/test my changes and then I will merge them into master, closing this PR in the progress? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#3184 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACLCbrU0safNKAEM7jRUvfZfzJpW9ovXks5twq6_gaJpZM4TchIz> .

[dgehri]

Ken,

Your changes look good, too. I just made a minor change to the Enumerator (ensuring that Current is read-only, and ensuring that m_entryData is > sizeof(long).

I think we are good to go.

[dgehri]

Not sure why the checks failed - this seems to be a n issue with AppVeyor / Travis rather than this commit...

[duplicatibot]

This pull request has been mentioned on Duplicati. There might be relevant details there:

https://forum.duplicati.com/t/will-changed-files-use-macos-fsevents-service/5485/6

[duplicatibot]

This pull request has been mentioned on Duplicati. There might be relevant details there:

https://forum.duplicati.com/t/2-0-4-22-ntfs-usn/7719/2

[duplicatibot]

This pull request has been mentioned on Duplicati. There might be relevant details there:

https://forum.duplicati.com/t/explain-usn-option/150/7