ROCm 3.5+ support roadmap

[emcastillo]

Tracking issue for ROCm tasks before v9
@leofang

• rocSOLVER/ hipMAGMA

[leofang]

Library support:

• sparse: ROCm: Support hipSPARSE/rocSPARSE #4101
• dnn: ROCm: Support hipDNN/MIOpen #4100

Improvement:

• move from hipBLAS to rocBLAS (ROCm: remove hipBLAS and use rocBLAS directly #3989)
• SVD (available since ROCm 3.8.0?)

Known bugs:

• ROCm 3.5.0 test: 32-bit cumsum, var, and nanvar mismatch with NumPy outcomes #4078
• ROCm 3.5.0 test: test_searchsorted #3841
• ROCm 3.5.0 test: test_cusparse.py #3834
• ROCm 3.5.0 test: TestCArray32BitBoundary #3833
• See also my working note sent to @kmaehashi

We also need a "known ROCm issues" doc page (discussed with @kmaehashi)

• hipRTC does not recognize the -I compiler option (hip runtime compilation bug in simple cupy code #4216)
• See also my working note sent to @kmaehashi

[leofang]

Looks like ROCm 3.9.0 is out...😞😞😞

[leofang]

Looks like ROCm 3.9.0 is out...😞😞😞

It seems CuPy master builds just fine on 3.9.0: https://ci.preferred.jp/cupy.py3.amd/64299/ 🦾

[leofang]

btw I noticed the cythonized .c & .cpp codes could be very different between CUDA and HIP builds, so if uploading sdist to PyPI is on the roadmap, this issue must be addressed (ex: can we upload two sets of sdist, one for CUDA and the other for HIP?)

[leofang]

ROCm 3.10:

https://rocmdocs.amd.com/en/latest/Current_Release_Notes/Current-Release-Notes.html#new-rocsolver-apis

[kmaehashi]

#4461

grids with total size of 2**32 do not launch threads at all.

this is not fixed as of ROCm 3.10/4.0.0.

[kmaehashi]

We have to take care of branching based on CUDA versions.
e.g.

cupy/cupy/core/core.pyx

Lines 1971 to 1977 in 65ca081

	cuda_path = cuda.get_cuda_path()
	if bundled_include is None and cuda_path is None:
	raise RuntimeError(
	'Failed to auto-detect CUDA root directory. '
	'Please specify `CUDA_PATH` environment variable if you '
	'are using CUDA versions not yet supported by CuPy.')

[leofang]

We have to take care of branching based on CUDA versions.

@kmaehashi This will be handled in #4102.

[kmaehashi]

We have to take care of branching based on CUDA versions.

@kmaehashi This will be handled in #4102.

Ah, nice! 👍🏼

[leofang]

I noticed that in many tests we call runtimeGetVersion() to determine whether to skip a test. The effect of this on HIP is to skip unconditionally, as in HIP runtimeGetVersion() gives an elusive version like 3212 (likely from the HIP runtime ROCclr, in which AMD_PLATFORM_BUILD_NUMBER is defined as such) that is much smaller that the currently supported CUDA versions. This is OK, but we should probably revisit it at some point.

[leofang]

For texture memory: ROCm has a renewed interest since 3.6.0 (the support is added in the ROCclr layer). I have a local branch that makes host-device texture copy work: https://github.com/leofang/cupy/tree/hip_texture, but device-device copy is not working. I consider this a low priority unless there are ROCm users who have a serious need, otherwise I think we should disable texture support and tests for ROCm/HIP until it catches up.

[kmaehashi]

I started to publish ROCm 4.0 test results for the latest matser branch commit, every 1 hour (note that the full unit test takes 4 hours):
https://github.com/kmaehashi/cupy-rocm-ci-report/commits/gh-pages
(It seems there’s a breaking change for ROCm between commit 1f6f9eb and f1bda44 ...)

I've asked @takagi to work on resolving test failures.

[leofang]

(It seems there’s a breaking change for ROCm between commit 1f6f9eb and f1bda44 ...)

tests/cupy_tests/cuda_tests/test_stream.py ..Fatal Python error: Segmentation fault

@kmaehashi @takagi It's likely we forgot to disable testing PTDS (#4322) on HIP. We hit the same segfault before. Stream 1 & 2 cannot be used on HIP.

[leofang]

I sent #4651 to fix PTDS.

[leofang]

Looks like we are back to the game after merging #4651 🙂
kmaehashi/cupy-rocm-ci-report@989f57f

[kmaehashi]

The CI script has died with srun: error: Node failure on xxx-xxx2-09... modified the script to respawn when node failed, and restarted.

[leofang]

Cool, thanks. Maybe it was due to allocation timeout?

[kmaehashi]

The CI script is on the login node, and I'm sruning for every test run, so I guess it's unlikely.
https://github.com/kmaehashi/cupy-rocm-ci-report/blob/master/tools/daemon.sh

[leofang]

I see, I thought it's run on the compute node interactively.

[leofang]

@kmaehashi @takagi Looks like something went wrong recently. The number of CI errors went from 9X to 18X:
https://github.com/kmaehashi/cupy-rocm-ci-report/compare/ce51038..d66d2c55
Looks like sorting is broken.

[leofang]

@kmaehashi @takagi Looks like something went wrong recently. The number of CI errors went from 9X to 18X:
https://github.com/kmaehashi/cupy-rocm-ci-report/compare/ce51038..d66d2c55
Looks like sorting is broken.

This is the offending commit: 74c54b0

[takagi]

Newly got strange failures...

FAILED tests/cupy_tests/lib_tests/test_polynomial.py::TestPoly1dInit_param_0_{variable=None}::test_poly1d_numpy_poly1d
FAILED tests/cupy_tests/lib_tests/test_polynomial.py::TestPoly1dInit_param_0_{variable=None}::test_poly1d_numpy_poly1d_variable
FAILED tests/cupy_tests/lib_tests/test_polynomial.py::TestPoly1dInit_param_1_{variable='y'}::test_poly1d_numpy_poly1d
FAILED tests/cupy_tests/lib_tests/test_polynomial.py::TestPoly1dInit_param_1_{variable='y'}::test_poly1d_numpy_poly1d_variable

E       AssertionError: 
E       Arrays are not equal
E       
E       Mismatched elements: 2 / 5 (40%)
E       Max absolute difference: 1.
E       Max relative difference: 1.
E        x: array([0.   , 1.875, 3.   , 4.   , 5.   ], dtype=float32)
E        y: array([1., 2., 3., 4., 5.], dtype=float32)

>>> import cupy, numpy
>>> cupy.asarray(numpy.arange(5, dtype='d')+1).get()
array([1., 2., 3., 4., 5.])
>>> cupy.asarray(numpy.arange(5, dtype='f')+1).get()
array([0.   , 1.875, 3.   , 4.   , 5.   ], dtype=float32)

1.875 is higher four bytes of double 1.0 interpreted as a (single) float so the first two elements seem to be occupied by 1.0 in double.

[kmaehashi]

It's ok to close this?

[leofang]

I dunno 😅 Do we want to resolve all xfail tests first (if at all possible)?

[takagi]

Let's close this as the situation is updated.