ComparisonService cannot produce a verdict for large files — checksum-only comparison regression

[ylangisc]

Ref: iterate-ch/mountainduck#2679 (endless re-download loop in Mountain Duck)

Summary

Since 3f75fcf (Compare using checksum, 2025-08-12, released with 9.5.x), B2Protocol returns a ComparisonService that compares files by checksum only:

if(type == ComparisonService.class) {
    return (T) new DefaultComparisonService(new ChecksumComparisonService(), ComparisonService.disabled);
}

B2 returns contentSha1: "none" for files uploaded through the large file API (b2_start_large_file, above b2.upload.largeobject.size = 100 MB), so neither the upload response nor b2_list_file_names ever carries a usable checksum for these files. ChecksumComparisonService therefore returns Comparison.unknown for every large file — even when size, modification date and versionId are byte-for-byte identical on both sides:

DEBUG DefaultComparisonService - Compare local attributes PathAttributes{size=270000962, modified=1784668715315, checksum='null', versionId='4_z..._f200de1010975a471_...'}
                                 with remote PathAttributes{size=270000962, modified=1784668715315, checksum='null', versionId='4_z..._f200de1010975a471_...'}
WARN  ChecksumComparisonService - No remote checksum available for comparison ...

This is a regression of the exact problem fixed in c7aab04 (Use file id for comparison because checksum may not be present for large file uploads, 2023-10-11), which had switched B2 to VersionIdComparisonService for this very reason.

Impact

In Mountain Duck (5.3.1 with core 9.5.3), the directory indexer treats Comparison.unknown as "differs" and schedules a download. Because the comparison can never succeed for B2 large files, every completed download triggers the next indexing pass and the next download — an endless, self-sustaining loop (iterate-ch/mountainduck#2679). In the reported case, four files (~6 GB combined) were each fully re-downloaded 65+ times in one evening (several hundred GB of Class B egress, confirmed via HTTP transcripts) until B2 rejected further downloads with 403 Cannot download file, download bandwidth or transaction (Class B) cap exceeded. Reproduced with both the Integrated (Cloud Files API) and Smart Synchronization (CBFS) connect modes; files below the large-file threshold receive a real SHA1 and never loop.

Any other consumer of the protocol's ComparisonService (sync/compare in Cyberduck itself) is equally unable to obtain a verdict for B2 large files.

Suggested fix

Chain the two services so the checksum decides when present, with the version id as fallback when no checksum is available on either side. This preserves the semantics of 3f75fcf (a server-side copy/move with unchanged content still compares equal via checksum) while restoring the large-file handling of c7aab04:

if(type == ComparisonService.class) {
    // No checksum set in metadata for large file uploads. Fall back to comparing the version id
    return (T) new DefaultComparisonService(new ChainedComparisonService(
            new ChecksumComparisonService(), new VersionIdComparisonService()), ComparisonService.disabled);
}

ChainedComparisonService skips only unknown by default, so:

• small file, SHA1 on both sides → checksum verdict (unchanged behaviour, B2MoveFeatureTest unaffected)
• large file, no checksum on either side, identical versionId → equal (breaks the loop)
• large file, differing versionId → notequal
• no checksum and no version id on either side → unknown (unchanged)

Verified against the attribute sets from the logs behind iterate-ch/mountainduck#2679 (unit test covering the three cases above).

Environment

• Cyberduck core 9.5.3 (as shipped in Mountain Duck 5.3.1.29526), regression present since 3f75fcf
• Backblaze B2, files > 100 MB (b2.upload.largeobject.size), with and without Cryptomator vault