Cuda error in `test_rf_classification_seed` when `cudf.pandas` is enabled

[jcrist]

test_random_forest.py::test_rf_classification_seed[small_clf0-float64] has failed a few times lately when cudf.pandas is enabled. In this run the cause was a cuda error: https://github.com/rapidsai/cuml/actions/runs/13522196050/job/37786184646

[betatim]

Another case of this happening https://github.com/rapidsai/cuml/actions/runs/13552388869/job/37880823247?pr=6373 (#6373)

head of the failure:

=========================== short test summary info ============================
FAILED test_random_forest.py::test_rf_classification_seed[small_clf0-float64] - RuntimeError: CUDA error encountered at: file=/opt/conda/conda-bld/work/cpp/src/fil/infer.cu line=902: call='cudaPeekAtLastError()', Reason=cudaErrorDeviceUninitialized:invalid device context
Obtained 63 stack frames
#1 in /opt/conda/envs/test/lib/python3.10/site-packages/cuml/cluster/../../../../libcuml++.so: raft::cuda_error::cuda_error(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) +0xbd [0x7fbc4759375d]
#2 in /opt/conda/envs/test/lib/python3.10/site-packages/cuml/cluster/../../../../libcuml++.so: void ML::fil::infer_k_storage_template<ML::fil::storage<ML::fil::dense_node<double> > >::run<ML::fil::KernelTemplateParams<true, true, (ML::fil::leaf_algo_t)5, 1> >(ML::fil::predict_params) +0x383 [0x7fbc47b5cad3]
#3 in /opt/conda/envs/test/lib/python3.10/site-packages/cuml/cluster/../../../../libcuml++.so: void ML::fil::dispatch::dispatch_on_leaf_algo<ML::fil::KernelTemplateParams<true, true, (ML::fil::leaf_algo_t)5, 1>, ML::fil::infer_k_storage_template<ML::fil::storage<ML::fil::dense_node<double> > >, void>(ML::fil::infer_k_storage_template<ML::fil::storage<ML::fil::dense_node<double> > >, ML::fil::predict_params) +0x39e [0x7fbc47b8763e]
...

[jcrist]

Here's a run where this was a segfault, not a cuda error: https://github.com/rapidsai/cuml/actions/runs/13640042486/job/38129460183?pr=6388

Logs

Fatal Python error: Segmentation fault

Thread 0x00007f0b77bf9700 (most recent call first):
  File "/opt/conda/envs/test/lib/python3.10/threading.py", line 324 in wait
  File "/opt/conda/envs/test/lib/python3.10/threading.py", line 607 in wait
  File "/opt/conda/envs/test/lib/python3.10/site-packages/tqdm/_monitor.py", line 60 in run
  File "/opt/conda/envs/test/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/opt/conda/envs/test/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f11a352f700 (most recent call first):
  File "/opt/conda/envs/test/lib/python3.10/site-packages/execnet/gateway_base.py", line 534 in read
  File "/opt/conda/envs/test/lib/python3.10/site-packages/execnet/gateway_base.py", line 567 in from_io
  File "/opt/conda/envs/test/lib/python3.10/site-packages/execnet/gateway_base.py", line 1160 in _thread_receiver
  File "/opt/conda/envs/test/lib/python3.10/site-packages/execnet/gateway_base.py", line 341 in run
  File "/opt/conda/envs/test/lib/python3.10/site-packages/execnet/gateway_base.py", line 411 in _perform_spawn

Current thread 0x00007f11a4e3b400 (most recent call first):
  Garbage-collecting
  File "/opt/conda/envs/test/lib/python3.10/site-packages/rmm/allocators/cupy.py", line 36 in rmm_cupy_allocator
  File "/opt/conda/envs/test/lib/python3.10/site-packages/cupy/_creation/from_data.py", line 88 in asarray
  File "/opt/conda/envs/test/lib/python3.10/site-packages/cuml/internals/array.py", line 647 in to_output
  File "/opt/conda/envs/test/lib/python3.10/site-packages/nvtx/nvtx.py", line 122 in inner
  File "/opt/conda/envs/test/lib/python3.10/site-packages/cuml/internals/memory_utils.py", line 87 in cupy_rmm_wrapper
  File "/opt/conda/envs/test/lib/python3.10/site-packages/cuml/internals/array.py", line 1169 in from_input
  File "/opt/conda/envs/test/lib/python3.10/site-packages/nvtx/nvtx.py", line 122 in inner
  File "/opt/conda/envs/test/lib/python3.10/site-packages/cuml/internals/memory_utils.py", line 87 in cupy_rmm_wrapper
  File "/opt/conda/envs/test/lib/python3.10/site-packages/cuml/internals/input_utils.py", line 412 in input_to_cuml_array
  File "/opt/conda/envs/test/lib/python3.10/site-packages/nvtx/nvtx.py", line 122 in inner
  File "/opt/conda/envs/test/lib/python3.10/site-packages/cuml/internals/api_decorators.py", line 193 in wrapper
  File "/opt/conda/envs/test/lib/python3.10/site-packages/cuml/internals/api_decorators.py", line 193 in wrapper
  File "/opt/conda/envs/test/lib/python3.10/site-packages/cuml/internals/api_decorators.py", line 195 in wrapper
  File "/opt/conda/envs/test/lib/python3.10/site-packages/cuml/internals/api_decorators.py", line 416 in dispatch
  File "/opt/conda/envs/test/lib/python3.10/site-packages/cuml/internals/api_decorators.py", line 193 in wrapper
  File "/opt/conda/envs/test/lib/python3.10/site-packages/nvtx/nvtx.py", line 122 in inner
  File "/__w/cuml/cuml/python/cuml/cuml/tests/test_random_forest.py", line 493 in test_rf_classification_seed
  File "/opt/conda/envs/test/lib/python3.10/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/opt/conda/envs/test/lib/python3.10/site-packages/_pytest/python.py", line 1792 in runtest
  File "/opt/conda/envs/test/lib/python3.10/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/opt/conda/envs/test/lib/python3.10/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/opt/conda/envs/test/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call
  File "/opt/conda/envs/test/lib/python3.10/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/opt/conda/envs/test/lib/python3.10/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/opt/conda/envs/test/lib/python3.10/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/opt/conda/envs/test/lib/python3.10/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/opt/conda/envs/test/lib/python3.10/site-packages/xdist/remote.py", line 195 in run_one_test
  File "/opt/conda/envs/test/lib/python3.10/site-packages/xdist/remote.py", line 174 in pytest_runtestloop
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/opt/conda/envs/test/lib/python3.10/site-packages/_pytest/main.py", line 325 in _main
  File "/opt/conda/envs/test/lib/python3.10/site-packages/_pytest/main.py", line 271 in wrap_session
  File "/opt/conda/envs/test/lib/python3.10/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/opt/conda/envs/test/lib/python3.10/site-packages/xdist/remote.py", line 393 in <module>
  File "/opt/conda/envs/test/lib/python3.10/site-packages/execnet/gateway_base.py", line 1291 in executetask
  File "/opt/conda/envs/test/lib/python3.10/site-packages/execnet/gateway_base.py", line 341 in run
  File "/opt/conda/envs/test/lib/python3.10/site-packages/execnet/gateway_base.py", line 411 in _perform_spawn
  File "/opt/conda/envs/test/lib/python3.10/site-packages/execnet/gateway_base.py", line 389 in integrate_as_primary_thread
  File "/opt/conda/envs/test/lib/python3.10/site-packages/execnet/gateway_base.py", line 1273 in serve
  File "/opt/conda/envs/test/lib/python3.10/site-packages/execnet/gateway_base.py", line 1806 in serve
  File "<string>", line 8 in <module>
  File "<string>", line 1 in <module>

[csadorf]

Possibly related to or duplicate of #5991 ?

[csadorf]

Closing as assumed duplicate.