Hash largest files first (LPT scheduling)

[martinus]

Closes #88 — the cheap alternative to the chunked-hashing approach in #90.

Problem

When a scan's files vary a lot in size, one very large file can end up as the last thing in the hash queue. A single csum thread then hashes it alone while every other thread has already gone idle — the makespan is dominated by that one straggler.

Approach — longest-processing-time-first, via size buckets

Hashing still starts immediately as files stream in from the walk. The only change is the order work is handed out: a free csum thread always takes work from the largest non-empty size class first.

Files are bucketed by size on a log scale — bucket 0 is everything <1 MiB, then one bucket per power of two (1, 2, 4, 8 MiB, …) — into per-bucket intrusive FIFOs, with a u64 occupancy bitmask. Push and pop are both O(1): pop finds the top non-empty bucket with a single clz on the mask (never a scan over buckets). A huge file sits alone in a high bucket and is dispatched first; the only slack vs exact ordering is the <2× spread within a bucket, which doesn't affect the idle tail. Walk order is preserved within a bucket, and a tree of only small files all lands in bucket 0, degrading to plain FIFO at zero cost.

Why buckets and not a sort function or a heap

• g_thread_pool_set_sort_function was the first cut and it's a trap: GLib's sorted push is O(queue depth), and the walk front-loads tens of thousands of files, so it turned a 13s ~/git scan into 35s.
• An exact O(log n) max-heap works and is makespan-equivalent, but costs ~0.45s more user CPU on the 174k-file ~/git tree (all files <1 MiB, ordering buys nothing). Buckets get that back and match plain FIFO's CPU.

Results

Adversarial tree — one 8 GiB file + 80×100 MiB, --io-threads=2, warm:

build	makespan (warm)
buckets	steady 2.16s — the max(largest_file, total/2) optimum
exact heap	steady 2.16s (identical)
FIFO (master)	2.16s best, up to 4.46s (~2×) when the big file lands late

Uniform ~/git (~174k files, all <1 MiB), mean user CPU / wall:

build	user CPU	wall
buckets	5.89s	13.1s
exact heap	6.34s	13.2s
FIFO (master)	5.85s	12.9s

Buckets match FIFO's CPU (the reorder is free when everything is bucket 0); wall is neutral across all three.

What this does not do

It can't parallelize a single file larger than the sum of everything else — only intra-file chunking (#90) can, at the cost of a digest redefinition + hashfile rebuild. This covers the common multi-file case for free: no schema bump, no new digest, no per-file concurrency path.

Verification

• New unit tests: scan_bucket boundaries, and the work queue itself (push/pop hand back largest-bucket-first, FIFO within a bucket).
• scripts/verify.sh green: build (warnings=errors), 90 integration + 9 unit tests, valgrind scan/dedupe/replay smoke clean on the new threading.

🤖 Generated with Claude Code

[martinus]

Switched the dispatch from the exact max-heap to your size-bucketing idea and A/B'd them head to head — you were right, it's the better call.

Design: bucket 0 = <1 MiB, then ×4 per bucket (bucket = 1 + (fls(size)-20)/2), each bucket a FIFO, plus a u32 occupancy bitmask so pop is 31 - clz(occupied). Push and pop are both O(1); walk order is kept within a bucket.

Adversarial tree (8 GiB file + 80×100 MiB, --io-threads=2, warm): buckets and the heap are indistinguishable — both a steady 2.16s (the optimum). The big file is alone in its high bucket, so it's always dispatched first, exactly like exact LPT.

~/git (~174k files, all <1 MiB), mean user CPU:

build	user CPU
buckets	5.89s
exact heap	6.34s
FIFO (master)	5.85s

This is the deciding data: with everything in bucket 0 the reorder is literally free, so buckets match plain FIFO's CPU while the heap pays ~0.45s for comparisons that buy nothing here. Same scheduling quality as the heap, lower overhead on the common case, and simpler O(1) code with no comparator/sift/realloc. Amended onto the branch.

[martinus]

Refined the buckets from ×4 to one bucket per power of two (2× granularity), so files within a size class differ by <2× instead of <4× — finer ordering among large files at no cost. This needs up to ~45 buckets, so the occupancy bitmask is now a u64 (clzll). Pop stays O(1) — it's 63 - clzll(occupied), a single instruction to find the top non-empty bucket, not an iteration over buckets. Perf is unchanged from the ×4 measurements by construction: small files still all fall in bucket 0 (identical CPU), and a lone huge file is still alone in its bucket (same optimum); the finer split only matters when several large files share what used to be one 4× bucket.