Fix F_PREALLOCATE delta calculation, use x/sys/unix, keep nullseed limit#1
Conversation
- Only preallocate the difference between the current file size and the target size: F_PREALLOCATE with F_PEOFPOSMODE allocates relative to the existing end of file, so passing the full size over-allocates when assembling in-place into an existing file (and can fail with ENOSPC). - Replace the raw syscall.Syscall/unsafe implementation with unix.FcntlFstore from golang.org/x/sys, which the module already depends on. This drops the hand-rolled fstore_t struct and constants. - Fall back to a plain truncate when the filesystem doesn't support F_PREALLOCATE (ENOTSUP, e.g. SMB or FUSE mounts); the sparse-hole issue is specific to APFS. - Create the file in the non-darwin preallocateFile too, so both implementations behave the same on a nonexistent target. - Revert the nullseed.go limit removal. The 100-chunk limit was added in #220 for worker load-balancing and removing it would serialize large zero-region writes on non-reflink filesystems when isBlank=false. The preallocation alone fixes the APFS corruption. - Add tests for preallocateFile covering new, grown, shrunk, same-size, and empty files. These run the real F_PREALLOCATE path on macOS CI.
On a nearly-full APFS volume (hdiutil image), re-preallocating an existing file to its current size must not request additional disk space. Passing the full size to F_PREALLOCATE with F_PEOFPOSMODE instead of only the missing difference over-allocates beyond the current end of file and fails with ENOSPC.
VerificationTo confirm the
The test fills most of a small APFS volume with a file already at its target size, then calls Also worth noting: the content-level tests alone could not catch this (the original Darwin code passed them — the over-allocation is transient since the trailing truncate releases the extra blocks), which is why the ENOSPC test exists. |
1 participant
Proposed fixes for folbricht#326, targeting its branch so they can be merged into the PR.
Changes
Only preallocate the size delta
F_PREALLOCATEwithF_PEOFPOSMODEallocates relative to the file's current end.AssembleFilealso runs against existing, non-empty files (in-place updates), where passing the full target size requests that much space beyond the current EOF — for a 350 GB in-place reassembly that's ~350 GB of extra allocation before the subsequent truncate trims it back, which can spuriously fail withENOSPC. Now the file is stat'd first and onlysize - currentis requested, skipping the fcntl entirely when no growth is needed (also avoidsF_PREALLOCATEwith a zero/negative length).Use
golang.org/x/sys/unixThe module already depends on
x/sys, which providesunix.FcntlFstore,unix.Fstore_t, and theF_PREALLOCATE/F_ALLOCATEALL/F_PEOFPOSMODEconstants. This drops the hand-rolled struct, magic constants,unsafe, and the deprecatedsyscall.Syscall.Fall back to truncate on
ENOTSUPNot all filesystems support
F_PREALLOCATE(SMB, some FUSE mounts). The sparse-hole issue is APFS-specific, so onENOTSUPfall back to the previous plain-truncate behavior instead of failing the whole assembly.Revert the nullseed.go change
The 100-chunk limit was added in folbricht#220 for worker load-balancing, not correctness: when
isBlank=falseand reflink is unavailable,nullChunkSection.WriteIntoreally does copy zeros, and without the limit a single worker serially zero-fills an unbounded region (plus the per-chunk read-back validation) while other workers idle. Chunks beyond the limit don't produce incorrect data — they're picked up by the nextLongestMatchWithcall or fetched from the store. The preallocation alone fixes the APFS corruption.Make the non-Darwin implementation create the file too
The Darwin version opens with
O_CREATEwhile the fallback used bareos.Truncate, which fails on a nonexistent file. Both now behave identically (caught by the new tests on Linux).Tests
Added
preallocate_test.gocovering new, grown (original content preserved, tail reads as zeros), shrunk, same-size, and empty files. These run the realF_PREALLOCATEpath on the macOS CI runners.