Skip to content

add FixedBitSet.nextUnsetBit and use it in MergingHnswGraphBuilder#16079

Closed
iprithv wants to merge 1 commit into
apache:mainfrom
iprithv:unset-bit-iteration
Closed

add FixedBitSet.nextUnsetBit and use it in MergingHnswGraphBuilder#16079
iprithv wants to merge 1 commit into
apache:mainfrom
iprithv:unset-bit-iteration

Conversation

@iprithv
Copy link
Copy Markdown
Contributor

@iprithv iprithv commented May 18, 2026

there was a TODO in MergingHnswGraphBuilder.build() where we were scanning all nodes even if most were already initialized, this change adds nextUnsetBit and uses that instead, so we only go over nodes that are not set, instead of checking every node, we just jump to the next unset one.

so earlier it goes through all nodes and check each one, now jump directly to next unset bit.
this is faster when most bits are already set, which is common during merges

benchmark (bitset iteration)

size set % scan loop (us) nextUnsetBit (us) speedup
1,000 0% 8.76 11.89 slower
1,000 75% 2.27 1.25 ~1.8x
1,000 90% 2.30 0.54 ~4.2x
1,000 99% 2.24 0.06 ~35x
100,000 75% 217 82.3 ~2.6x
100,000 90% 218 33.4 ~6.5x
100,000 99% 215 4.69 ~45x
1,000,000 99% 2157 43.3 ~49x

luceneutil (real run)

baseline candidate result
force merge 220.11s 218.68s same
index time 224.73s 229.25s same
recall 0.988 0.988 same

@iverase
Copy link
Copy Markdown
Contributor

iverase commented May 18, 2026

see #16072

@iprithv
Copy link
Copy Markdown
Contributor Author

iprithv commented May 18, 2026

thanks @iverase for pointing out, will close this. As that was merged just now i missed it :)

@iprithv iprithv closed this May 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants