Vectorize dense bitmap decoding in ToArray with AVX-512 VBMI2 - #557
Merged
Conversation
bitmapContainer.fillLeastSignificant16bits walks set bits with a scalar
TZCNT/BLSR loop. On CPUs with AVX512_VBMI2 a single VPCOMPRESSB turns a
whole 64-bit word into 64 byte-sized bit positions; those are widened
with VPMOVZXBD and written with masked stores, so exactly popcount(word)
values are written and ToArray's exactly-sized output slice needs no
padding.
ToArray over the real-data corpus (Xeon Gold 6548N, Emerald Rapids):
census-income 8.48 ms -> 4.60 ms (1.84x)
weather_sept_85 16.19 ms -> 7.83 ms (2.07x)
The other ten datasets hold no bitmap containers and are unchanged.
Feature detection goes through golang.org/x/sys/cpu, so the OSXSAVE and
XCR0 opmask/ZMM checks are handled upstream and GODEBUG=cpu.avx512vbmi2=off
disables the path at run time. Containers below 4096 values, CPUs without
VBMI2, appengine builds and non-amd64 targets keep the scalar loop.
The compress-then-widen kernel follows simdjson's bit_indexer::write.
Based on the approach explored in #556 by Perfloop.
Claude-Session: https://claude.ai/code/session_0123ePBefqPjrCvhxdwWFakj
8 tasks
This was referenced Aug 27, 2026
lemire
added a commit
to RoaringBitmap/CRoaring
that referenced
this pull request
Aug 27, 2026
Both bitset_extract_setbits_avx512 and bitset_extract_setbits_avx512_uint16
kept the VPCOMPRESSB kernel but stored full 64-byte blocks unconditionally,
relying on the next word to overwrite the slack. That cost two things: the
caller had to leave 64 values of padding (so the last values always fell back
to the scalar tail), and every word paid for all four blocks even when it only
produced a handful of values.
Each block is now stored under a mask derived from BZHI(-1, popcount), so a
store writes exactly the values it produced, and only the ceil(popcount/16)
blocks that carry a value are computed at all. Empty words are skipped and the
base is carried in a ZMM incremented by a constant instead of being rebuilt
per word.
Measured on a Xeon Gold 6548N (Emerald Rapids), ns per value, kernel only:
cardinality scalar before after
256 2.91 6.93 2.95
1024 1.29 1.95 1.02
2048 0.76 1.07 0.65
4096 0.54 0.54 0.37
8192 0.42 0.27 0.18
The uint32 decoder now beats the scalar loop at every cardinality measured
from 16 to 65536, so bitset_container_to_uint32_array no longer needs its
cardinality heuristic.
array_container_from_bitset is the opposite case: every caller checks that the
result fits an array container first, so its input is always under 6.25%
density, and it had no heuristic at all. The old kernel was a net regression
there -- 2.4x slower than the scalar loop at 256 values. The new one is faster
from about 1024 values up, so that is where the gate goes.
End to end, with the repository's own benchmark:
microbench/ToArray/census-income 6671 ns -> 4705 ns (1.42x)
microbench/ToArray/weather_sept_85 25895 ns -> 11611 ns (2.23x)
bitset_container/to_array_convert 2.08 ns -> 1.47 ns (1.42x)
The other datasets hold no bitset containers and are unchanged. Benchmarks
outside these paths move by up to 1.5x in either direction, but rebuilding the
unmodified source reproduces the same swings, so they are code layout noise.
The masked-store idea is from RoaringBitmap/roaring#557, which applies it to
the Go port's ToArray.
A new test compares both decoders against the scalar ones across 64 densities,
at exactly the cardinality and at truncated capacities.
Claude-Session: https://claude.ai/code/session_015iZSVG9KJ595ktzGqaK7qR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Speeds up
Bitmap.ToArrayfor dense bitmap containers on amd64 CPUs withAVX512_VBMI2.This is an alternative to #556, which targets the same bottleneck. Same idea, different kernel and a lower density gate; measurements comparing the two are below.
Changes Made
bitmapContainer.fillLeastSignificant16bitswalks set bits with a scalar TZCNT/BLSR loop. With VBMI2, a singleVPCOMPRESSBturns a whole 64-bit word into 64 byte-sized bit positions in one shot; those are widened 16 at a time withVPMOVZXBD, added to a running base vector, and written with masked stores.The compress-then-widen kernel follows simdjson's
bit_indexer::write. It differs in the stores: simdjson writes whole 64-byte blocks and relies on its output buffer having up to 64uint32of slack.ToArrayallocates exactlyGetCardinality()values with no padding, so each block is written with a mask derived fromBZHI(-1, popcount). That keeps writes exactlypopcount(word)wide — and it is also faster on dense containers, because no store bandwidth goes to values nobody asked for (up to 35% faster than the unmasked form once the output leaves cache).Feature detection goes through
golang.org/x/sys/cpu, which already verifies OSXSAVE and the XCR0 opmask/ZMM state, and honorsGODEBUG=cpu.avx512vbmi2=offto disable the path at run time without rebuilding. This addsgolang.org/x/sysas a direct dependency; it was already in the module graph.Containers below 4096 values, CPUs without VBMI2, appengine builds and non-amd64 targets keep the scalar loop.
Performance Impact
All numbers on a Xeon Gold 6548N (Emerald Rapids), pinned with
taskset, Go 1.26.3.ToArrayover the RoaringBitmap real-data corpus, best of 5 x 20 iterations:census-incomeweather_sept_85The other ten datasets in the corpus contain no bitmap containers at all, so neither change affects them.
Kernel only, ns per set bit by container density:
On the density gate
The threshold is 4096 rather than 25% density. Measured break-even against the scalar loop is near 2000 values, and 4096 is the array-to-bitmap conversion point, so this covers essentially every bitmap container with margin.
That band is not empty in practice. Of
weather_sept_85's 561 bitmap containers, 349 (62%), holding 2.48M of 10.37M values, sit below 25% density; forcensus-incomeit is 42 containers and 285K values.Testing
go test ./...,roaring64, the smat corpus and smat-hit checks,go tool unconvert,gofmt,go vet, and cross-builds for 386/arm/arm64 and-tags appengine, all on VBMI2 hardware so the vector path actually executes.TestBitmapContainerFillLeastSignificant16bitsVectorcompares the vector decoder against the scalar one at cardinalities 4095/4096/4097 (the crossover), 20000, 40000, 60000, 65535 and 65536, withmask = 0xFFFF0000and sentinel padding on both sides. It was mutation-checked: perturbing a single branch bound in the kernel makes it fail.Breaking Changes
None. Output ordering, the return value and the allocation contract are unchanged; this only swaps how the values are produced.
Credits
Based on the approach explored in #556 by Perfloop. The kernel follows simdjson's
bit_indexer::write.https://claude.ai/code/session_0123ePBefqPjrCvhxdwWFakj