ReFS parallel file cloning silent failure bug

[erikmav]

Windows-only.

Symptoms: When cloning a single source file in parallel (multiple threads or processes) to multiple destinations, sometimes success is returned but the file region assignment is never completed. This results in the destination file having all zeroes for its content.

Cause: When a file is not completely flushed to disk when the region clone operation starts, ReFS tries to flush the file to disk first. There is a race on multiple threads where failure to flush is ignored and the region clone proceeds anyway.

Workarounds:

• Serialize cloning system-wide per source path. This library takes this approach by default for single-process cloning by using an in-memory dictionary. You can opt into system-wide serialization using kernel mutexes by specifying useCrossProcessLocksWhereApplicable = true when calling CopyOnWriteFilesystemFactory.GetInstance().
• Ensure the source file is completely flushed to disk before cloning. This can be accomplished through one of the approaches below. Note that if you use these approaches, you can increase performance of cloning by using CloneFileFlags.NoSerializedCloning on your CloneFile calls.
  • Using the FlushFileBuffers API to force the file to be flushed from memory. This should be called at the end of writing the source file to disk while the file write handle is still open. Alternately it could be called on a new handle to the file opened with GENERIC_WRITE.
  • When writing the source file, open the file handle with FILE_FLAG_NO_BUFFERING. However, note this requires the code writing to the file to deal with writing chunks aligned with the sector size of the underlying volume, and using chunks that are a multiple of the sector size.
  • When writing the source file, open the file handle with FILE_FLAG_WRITE_THROUGH. This forces a flush on every write, which can decrease performance significantly.

Resolution: We currently have only workarounds (see above) and are awaiting resolution with the Windows team.

This issue tracks resolution in the Windows codebase and other approaches to work around the problem. Related to #1. PRs that have added workarounds:

• Serialize Windows clones by source path #6 - added single-process (in-memory dictionary) serialization of cloning of individual files.
• Cross-proc Windows safety and optimizations #9 - added optional cross-process serialization using kernel named mutexes, one per file path.
• Improve documentation for parallel clone problem on Windows #13 - documentation
• (See any linked future PRs)

[erikmav]

Not repro on Win11 2H22 with Jan 2023 patches. #20 to remove the serialization paths.