perf: speed up SQLite numbits aggregation

[KRRT7]

Summary

Optimize SQLite numbits aggregation by accumulating packed bitsets as integers instead of repeatedly creating intermediate byte strings.

The aggregate now converts the final integer back to bytes only once during finalization.

Impact

This reduces Python-level work when combining many numbits values, especially during coverage data combination and reporting.

How this optimization was found

While reviewing the SQLite data-combination path, we found that NumbitsUnionAgg repeatedly called numbits_union() for every row.

Each call iterated over the byte strings and allocated a new byte string for the intermediate result. For aggregations over many rows, this caused repeated Python-level work and memory allocations.

The optimization keeps the accumulated bitset as a Python integer and combines each value with a bitwise OR. The result is converted back to the packed byte representation only once, during finalization. The original byte length is tracked so the serialized result preserves the expected representation, including empty aggregates.

this was found using a proprietary AI agent.

[devdanzin]

Nice! Do you have (micro?)benchmarks showing the effect of this change?

[KRRT7]

#2239 (comment)

Hi Daniel, big fan of your work :)

I'm not sure if I still have the benchmarks still, but I'll check.

[KRRT7]

Hi Daniel — I no longer have the original standalone microbenchmarks, but I reran the comparison using the ASV benchmark suite from PR #2238.

The relevant benchmark is TimeCombine, which exercises SQLite coverage-data combination:

main:
  time_combine_many_files_line_only       136 ± 0.7 ms
  time_combine_many_files_with_contexts   209 ± 10 ms

perf/sqlite-numbits:
  time_combine_many_files_line_only       120 ± 2 ms
  time_combine_many_files_with_contexts   193 ± 10 ms

That’s approximately 12% faster for line-only combination and 8% faster with contexts. The contexts result is less conclusive because the measurement variance overlaps.

The benchmark suite is in PR #2238. To reproduce the comparison:

git fetch origin pull/2238/head:asv-bench
git fetch origin pull/2239/head:perf/sqlite-numbits

git worktree add /tmp/coverage-main origin/main
git worktree add /tmp/coverage-sqlite perf/sqlite-numbits

git -C /tmp/coverage-main restore --source=asv-bench --worktree -- asv.conf.json benchmarks
git -C /tmp/coverage-sqlite restore --source=asv-bench --worktree -- asv.conf.json benchmarks

cd /tmp/coverage-main
uvx --from asv asv run --bench TimeCombine 'origin/main^!'

cd /tmp/coverage-sqlite
uvx --from asv asv run --bench TimeCombine 'perf/sqlite-numbits^!'

If asv is already installed, asv run can be used directly instead of uvx --from asv asv run.

[devdanzin]

Great result, thank you!

[nedbat]

Thanks!

[nedbat]

This is now released as part of coverage 7.15.3.