File list verify is unnecessarily slow on some filesystems

[Jojo-1000]

• I have searched open and closed issues for duplicates.
• I have searched the forum for related topics.

---

Environment info

• Duplicati version: current master
• Operating system: Windows 10
• Backend: File

Description

Listing the backend files for exFAT targets (may also apply to other FAT types) takes much longer than required. It is suspected this is due to unnecessary lookups of files by name.

Using DirectoryInfo to directly list the folder contents as FileInfo, this lookup is avoided and the list operation completes much faster (50 seconds instead of 1 hour for 110000 files).

ISystemIO should be updated to implement the more efficient method of listing.

Steps to reproduce

1. Create a folder on exFAT partition with 10000 - 100000 empty files
2. Create new backup with that folder as target
3. Run backup

• Actual result:
  File verify takes a long time (minutes to hours).
• Expected result:
  File verify should not take longer than a minute.

Test code

This code simulates two different methods of file access. It was discovered that accessing LastAccessTime or other metadata is the main reason for the slowdown.

string path = @"F:\test";
int iterations = 1;
var timeInfo = TimeSpan.Zero;
var timeListNames = TimeSpan.Zero;
var timeLookupNames = TimeSpan.Zero;
var watch = System.Diagnostics.Stopwatch.StartNew();

for (int i = 0; i < iterations; ++i)
{
    // List files directly (current implementation)
    watch.Restart();
    string[] files = System.IO.Directory.GetFiles(path);
    watch.Stop();
    timeListNames += watch.Elapsed;
    watch.Start();
    var accessTimes = (from fileName in files
                       let fi = new System.IO.FileInfo(fileName)
                       select fi.LastAccessTime).ToList();
    watch.Stop();
    timeLookupNames += watch.Elapsed;

    // List by DirectoryInfo
    watch.Restart();
    var accessTimes2 = (from fi in new System.IO.DirectoryInfo(path).GetFiles()
                        select fi.LastAccessTime).ToList();
    watch.Stop();
    timeInfo += watch.Elapsed;

}
Console.WriteLine($"List only names: {timeListNames}\nList + lookup by name: {timeLookupNames}\nList DirectoryInfo: {timeInfo}");

Output with 17000 files

List only names: 00:00:00.0264646
List + lookup by name: 00:00:58.3627105
List DirectoryInfo: 00:00:00.0315012

[duplicatibot]

This issue has been mentioned on Duplicati. There might be relevant details there:

https://forum.duplicati.com/t/why-restore-needs-so-much-time-of-verifying-remote-data/17013/44

[duplicatibot]

This issue has been mentioned on Duplicati. There might be relevant details there:

https://forum.duplicati.com/t/write-operations-on-source-disk/17375/10