Implement batch_null_count to count nulls for multiple null masks by a single kernel call, and application in groupby aggregations

[ttnghia]

This implements batch_null_count API to count nulls by a batch processing kernel, allowing to process multiple null masks at the same time using a single kernel call to reduce overhead :

• API: Introduce cudf::batch_null_count to accept multiple bitmask pointers and a single bit range.
• Kernel: Add internal kernel batch_count_set_bit_kernel to accumulate unset bits (non-zero) counts per bitmask.
• Tests: Extend bitmask_tests.cpp with a simple test case covering nullable and non-nullable masks.
• Immediate application: Adopt the newly added API in groupby aggregation to compute nulls for the output columns, improving the total performance by up to nearly 50% for aggregations having multiple output columns.

Closes #19878.

---

Benchmark

# groupby_max

## [0] Quadro RTX 6000

|  T  |  cardinality  |  num_rows  |  null_probability  |  num_aggregations  |   Ref Time |   Ref Noise |   Cmp Time |   Cmp Noise |         Diff |   %Diff |  Status  |
|-----|---------------|------------|--------------------|--------------------|------------|-------------|------------|-------------|--------------|---------|----------|
| I32 |       0       |    2^12    |        0.5         |         1          | 115.411 us |      37.78% | 119.352 us |      39.66% |     3.940 us |   3.41% |   SAME   |
| I32 |       0       |    2^18    |        0.5         |         1          | 203.242 us |      31.11% | 194.991 us |      11.01% |    -8.252 us |  -4.06% |   SAME   |
| I32 |       0       |    2^24    |        0.5         |         1          |   7.732 ms |       4.63% |   7.651 ms |       4.46% |   -81.033 us |  -1.05% |   SAME   |
| I32 |       0       |    2^12    |        0.5         |         10         | 271.759 us |      30.79% | 198.096 us |      42.31% |   -73.663 us | -27.11% |   SAME   |
| I32 |       0       |    2^18    |        0.5         |         10         | 411.565 us |      10.61% | 350.587 us |      25.03% |   -60.979 us | -14.82% |   FAST   |
| I32 |       0       |    2^24    |        0.5         |         10         |  18.687 ms |       4.59% |  18.569 ms |       4.80% |  -118.242 us |  -0.63% |   SAME   |
| I32 |       0       |    2^12    |        0.5         |         50         | 948.672 us |      16.12% | 524.900 us |      21.92% |  -423.772 us | -44.67% |   FAST   |
| I32 |       0       |    2^18    |        0.5         |         50         |   1.482 ms |      10.56% |   1.087 ms |      17.12% |  -394.923 us | -26.65% |   FAST   |
| I32 |       0       |    2^24    |        0.5         |         50         |  65.810 ms |       4.81% |  65.044 ms |       4.90% |  -766.761 us |  -1.17% |   SAME   |
| I32 |       0       |    2^12    |        0.5         |        100         |   1.779 ms |       8.10% | 917.999 us |      13.12% |  -861.060 us | -48.40% |   FAST   |
| I32 |       0       |    2^18    |        0.5         |        100         |   2.816 ms |       6.85% |   1.910 ms |       8.81% |  -905.536 us | -32.16% |   FAST   |
| I32 |       0       |    2^24    |        0.5         |        100         | 124.833 ms |       4.79% | 123.247 ms |       4.87% | -1586.663 us |  -1.27% |   SAME   |
| I32 |       0       |    2^12    |        0.5         |        200         |   3.461 ms |       6.42% |   1.767 ms |       8.39% | -1693.559 us | -48.93% |   FAST   |
| I32 |       0       |    2^18    |        0.5         |        200         |   5.578 ms |       5.74% |   3.647 ms |       6.36% | -1930.731 us | -34.62% |   FAST   |
| I32 |       0       |    2^24    |        0.5         |        200         | 243.209 ms |       4.81% | 240.061 ms |       4.78% | -3148.085 us |  -1.29% |   SAME   |
| F64 |       0       |    2^12    |        0.5         |         1          | 109.562 us |      20.97% | 114.126 us |      20.59% |     4.564 us |   4.17% |   SAME   |
| F64 |       0       |    2^18    |        0.5         |         1          | 216.398 us |      18.75% | 229.467 us |      26.79% |    13.069 us |   6.04% |   SAME   |
| F64 |       0       |    2^24    |        0.5         |         1          |   8.208 ms |       4.33% |   8.167 ms |       4.37% |   -40.784 us |  -0.50% |   SAME   |
| F64 |       0       |    2^12    |        0.5         |         10         | 295.664 us |      27.29% | 219.304 us |      29.83% |   -76.360 us | -25.83% |   SAME   |
| F64 |       0       |    2^18    |        0.5         |         10         | 459.533 us |       7.91% | 388.749 us |      13.95% |   -70.785 us | -15.40% |   FAST   |
| F64 |       0       |    2^24    |        0.5         |         10         |  22.192 ms |       4.97% |  22.213 ms |       5.49% |    21.395 us |   0.10% |   SAME   |
| F64 |       0       |    2^12    |        0.5         |         50         |   1.016 ms |      16.50% | 568.097 us |      13.72% |  -447.757 us | -44.08% |   FAST   |
| F64 |       0       |    2^18    |        0.5         |         50         |   1.607 ms |       4.84% |   1.278 ms |      11.09% |  -329.574 us | -20.51% |   FAST   |
| F64 |       0       |    2^24    |        0.5         |         50         |  82.705 ms |       4.95% |  82.210 ms |       5.26% |  -495.187 us |  -0.60% |   SAME   |
| F64 |       0       |    2^12    |        0.5         |        100         |   1.889 ms |       8.57% |   1.043 ms |      13.98% |  -846.296 us | -44.80% |   FAST   |
| F64 |       0       |    2^18    |        0.5         |        100         |   3.279 ms |       8.58% |   2.334 ms |       8.23% |  -944.419 us | -28.81% |   FAST   |
| F64 |       0       |    2^24    |        0.5         |        100         | 158.409 ms |       4.90% | 157.065 ms |       4.92% | -1344.521 us |  -0.85% |   SAME   |
| F64 |       0       |    2^12    |        0.5         |        200         |   3.720 ms |       7.01% |   1.933 ms |       8.52% | -1787.515 us | -48.05% |   FAST   |
| F64 |       0       |    2^18    |        0.5         |        200         |   6.218 ms |       4.96% |   4.649 ms |       6.83% | -1569.373 us | -25.24% |   FAST   |

[Copilot]

Pull request overview

This PR introduces a batch_null_count API to efficiently count null elements across multiple validity bitmasks using a single kernel launch, reducing overhead from repeated kernel calls. The implementation is immediately applied to groupby aggregations, yielding performance improvements up to nearly 50% for operations with multiple output columns.

Key Changes

• Adds cudf::batch_null_count API and internal batch_count_set_bit_kernel to process multiple bitmasks in a single kernel call
• Refactors groupby aggregation output finalization to use batch processing instead of per-column null counting
• Includes test coverage for the new batch null counting functionality

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
cpp/include/cudf/null_mask.hpp	Adds public API declaration for batch_null_count with documentation
cpp/src/bitmask/null_mask.cu	Implements batch_count_set_bit_kernel and batch_null_count function
cpp/src/groupby/hash/output_utils.cu	Refactors to use batch_null_count instead of per-column null_count calls
cpp/tests/bitmask/bitmask_tests.cpp	Adds test case covering nullable and non-nullable columns

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[res-life]

Seems the improvement is introduced by reducing the scalar creations.
Originally, there are lots of scalar creations for each column to compute null count.

cudf::detail::device_scalar<size_type> non_zero_count(0, stream);

Refer to the null_mask.cu, function count_set_bits.

Your batch kernel allocates an array to hold all the result count values. It only allocates one time. But the original approach allocates multiple times for each count value(store in scalar).

If you have time, could you double check this?
E.g.:
Change

cudf::size_type count_set_bits(bitmask_type const* bitmask,
                               size_type start,
                               size_type stop,
                               rmm::cuda_stream_view stream)

to:

cudf::size_type count_set_bits(bitmask_type const* bitmask,
                               size_type start,
                               size_type stop,
                               size_type*  global_count, // memory on GPU to pass to kernel.
                               rmm::cuda_stream_view stream)

Create an array of count values on GPU, then pass a count pointer to count_set_bits for each column. Do not use batch kernel, and to see if we get the same performance?

[ttnghia]

Seems the improvement is introduced by reducing the scalar creations.

I ran a benchmark and it is likely that your guess is correct. When launching the kernel multiple times but creating the array of null count in device memory once, the performance is almost the same as using batch kernel. Probably here the kernel is small and doesn't have much overhead thus we don't observe its benefit. Nevertheless, I still want to have a unified (batch) kernel for doing bulk work instead.

---

Batch kernel vs individual kernel launches (but writing to a shared pre-allocated device array):

# groupby_m2_var_std

## [0] Quadro RTX 6000

|  T  |  U  |  value_key_ratio  |  num_rows  |  null_probability  |  num_aggs  |   Ref Time |   Ref Noise |   Cmp Time |   Cmp Noise |        Diff |   %Diff |  Status  |
|-----|-----|-------------------|------------|--------------------|------------|------------|-------------|------------|-------------|-------------|---------|----------|
| I32 | 11  |        100        |  1000000   |        0.5         |     1      | 622.287 us |      29.20% | 614.684 us |      22.06% |   -7.603 us |  -1.22% |   SAME   |
| I32 | 11  |        100        |  10000000  |        0.5         |     1      |   7.321 ms |       5.55% |   7.715 ms |       2.47% |  394.431 us |   5.39% |   SLOW   |
| I32 | 11  |        100        |  1000000   |        0.5         |     50     |   7.034 ms |       6.23% |   7.189 ms |       6.17% |  155.272 us |   2.21% |   SAME   |
| I32 | 11  |        100        |  10000000  |        0.5         |     50     |  56.101 ms |       6.02% |  56.710 ms |       6.71% |  608.469 us |   1.08% |   SAME   |
| I32 | 11  |        100        |  1000000   |        0.5         |    100     |  14.000 ms |       5.67% |  14.367 ms |       6.07% |  366.341 us |   2.62% |   SAME   |
| I32 | 11  |        100        |  10000000  |        0.5         |    100     | 109.524 ms |       6.40% | 109.059 ms |       5.81% | -465.527 us |  -0.43% |   SAME   |
| I32 | 11  |        100        |  1000000   |        0.5         |    200     |  28.200 ms |       6.06% |  28.921 ms |       5.57% |  721.143 us |   2.56% |   SAME   |
| I32 | 11  |        100        |  10000000  |        0.5         |    200     | 229.657 ms |       5.92% | 230.729 ms |       5.53% |    1.072 ms |   0.47% |   SAME   |

[res-life]

So, one option is:
First create the array of null count in device memory once.
Then for each column, calls null_count with passing in a null count pointer.

Need to add a variant of null_count to add a parameter like:

cudf::size_type null_count_with_pre_allocated_count_storage(bitmask_type const* bitmask,
                           size_type start,
                           size_type stop,
+                         cudf::detail::device_scalar<size_type> & gpu_count_storage,
                           rmm::cuda_stream_view stream = cudf::get_default_stream());

Compared to batch kernel, it reduces the code changes.

[ttnghia]

So, one option is: First create the array of null count in device memory once. Then for each column, calls null_count with passing in a null count pointer.

Let's prefer the batch kernel model as it consolidate the GPU work and minimize any overhead/delay/latency. We have more of such batch kernel coming.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[PointKernel]

One small suggestion but non-blocking.

Thanks for the iterations addressing all my comments @ttnghia

[ttnghia]

/merge