[BUG] Failing singlegpu Python tests for 11.8 nightlies on H100

[divyegala]

These tests seem to be segfaulting only in our 11.8 nightlies for this configuration:
conda-python-tests-singlegpu / 11.8.0, 3.10, amd64, ubuntu20.04, h100, latest-driver, latest-deps

Whereas the tests pass for 11.8 in PR for this config:
conda-python-tests-singlegpu / 11.8.0, 3.10, amd64, rockylinux8, l4, earliest-driver, oldest-deps

[betatim]

It might be worth rerunning the test. It looks like there was a segfault and/or the runner crashed (for other reasons?): worker 'gw0' crashed while running 'cuml/tests/test_device_selection.py::test_svc_methods[True-multi_class-ovr-gpu-cpu]'

If it is ok for you, can you restart the job so we can see what happens? Don't want to just restart it without you giving 👍

[viclafargue]

It looks like this is a flaky test failure. I could not reproduce it yet. The segfault seems to be happening in the decision_function function Cython code, even if it is still uncertain.

The same test fails with cublas_handle.hpp line=76: call='cublasSetStream(ret, get_cuda_stream(res))', Reason=1:CUBLAS_STATUS_NOT_INITIALIZED here and here.
Running it with compute-sanitizer did not reveal anything suspicious apart from CUDA_ERROR_INVALID_CONTEXT (error 201).