[Tracking][Vulkan] Extending topi/relay tests to run on Vulkan

[Lunderberg]

Summary

Currently, some unit tests fail when running on the Vulkan runtime. PRs #8903 and #8947 parametrized the tests that are currently failing, so that the vulkan target can be marked as xfail without impacting any other runtimes. The Vulkan runtime should be improved so that these unit tests can pass on vulkan as well.

Status

File	Test	Parameters	Failure Step	Observed on	Status	Owner	PR
test_topi_math.py	test_ewise	topi_name="tan"	Codegen	NVIDIA/AMD	TODO
test_topi_math.py	test_ewise	topi_name="erf"	Codegen	NVIDIA/AMD	TODO
test_topi_math.py	test_ewise	topi_name="isnan"	Codegen	NVIDIA/AMD	TODO
test_topi_math.py	test_ewise	topi_name="isfinite"	Codegen	NVIDIA/AMD	TODO
test_topi_math.py	test_ewise	topi_name="isinf"	Codegen	NVIDIA/AMD	TODO
test_topi_reduce.py	test_reduce_map	reduce_type="sum"	Codegen	NVIDIA/AMD	TODO
test_topi_reduce.py	test_reduce_map	reduce_type="any"	Codegen	NVIDIA/AMD	TODO
test_topi_reduce.py	test_reduce_map	reduce_type="all"	Codegen	NVIDIA/AMD	TODO
test_topi_vision.py	test_proposal		Codegen	NVIDIA/AMD	TODO
test_topi_conv1d_transpose	test_conv1d_transpose_ncw		Numeric Output	NVIDIA only	TODO
test_topi_softmax.py	test_softmax	dtype="float64"	Codegen	NVIDIA/AMD	TODO
tests/python/relay/test_vm.py	test_cond		Codegen	NVIDIA/AMD	TODO
tests/python/relay/test_vm.py	test_simple_if		Codegen	NVIDIA/AMD	TODO
tests/python/relay/test_op_level4.py	test_reduce_functions		Codegen	NVIDIA/AMD	TODO
tests/python/relay/test_op_level3.py	test_sparse_reshape		Codegen	NVIDIA/AMD	TODO
tests/python/relay/test_any.py	test_any_reduce		Codegen	NVIDIA/AMD	TODO
tests/python/relay/test_op_level5.py	TestResize1D		Numeric Output	NVIDIA/AMD	TODO
tests/python/relay/test_op_level5.py	TestResize2D		Numeric Output	NVIDIA/AMD	TODO
tests/python/relay/test_op_level5.py	TestCropAndResize		Numeric Output	NVIDIA only	TODO
tests/python/relay/test_op_level3.py	test_take		Numeric Output	NVIDIA only	TODO
tests/python/relay/test_op_level2.py	test_conv2d_run		Codegen	NVIDIA/AMD	Fixed		#9014
tests/python/relay/test_op_level3.py	test_segment_sum		Codegen	NVIDIA/AMD	TODO
tests/python/relay/test_op_level3.py	test_scatter_add		Codegen	NVIDIA/AMD	TODO
tests/python/relay/test_op_level1.py	test_unary_op	relay_op=erf	Codegen	NVIDIA/AMD	TODO
tests/python/relay/test_op_level1.py	test_unary_op	relay_op=tan	Codegen	NVIDIA/AMD	TODO
tests/python/relay/test_op_level1.py	test_unary_op	relay_op=atan	Codegen	NVIDIA/AMD	TODO
tests/python/relay/test_op_grad_level10.py	test_cross_entropy_grad		Codegen	NVIDIA/AMD	TODO
tests/python/relay/test_op_grad_level1.py	test_log_softmax_grad		Codegen	NVIDIA/AMD	TODO
tests/python/relay/test_op_grad_level1.py	test_softmax_grad		Codegen	NVIDIA/AMD	TODO
tests/python/relay/test_op_grad_level1.py	test_unary_op	Several	Codegen	NVIDIA/AMD	TODO
tests/python/relay/test_any.py	test_any_batch_matmul		Codegen	NVIDIA/AMD	TODO
tests/python/relay/test_any.py	test_any_conv2d_NCHWc		Codegen	NVIDIA/AMD	TODO
tests/python/relay/test_any.py	test_any_dense		Codegen	NVIDIA/AMD	TODO

[Lunderberg]

@mbrookhart Regarding your comments that several of the failing unit tests had run correctly on vulkan in the past, the main breaking point was in #8127, which reads the device parameters from the physical device when the target is "vulkan -from_device=0". Several of the unit tests had a hard-coded target of "vulkan", tried to run with the minimum vulkan capabilities, and failed at codegen because the capability requested (e.g. 64-bit float support) wasn't listed in the target. Those fixes came along for free by parametrizing the topi tests, since the default vulkan test target uses the device query.

That said, at some point I want to ensure all tests either run correctly or have an appropriate xfail for the minimum vulkan feature set, but that will be a different issue.

[masahi]

This result is on a NV driver, or do they also fail on AMD?

[Lunderberg]

Thank you for checking, and all except the test_conv1d_transpose_ncw occur on AMD as well. It's the only one that is a numerical failure, while the rest of errors that occur during codegen. I'll update the table with that information.

[Lunderberg]

Following #8947 , added the failing relay tests to the tracking issue.

[masahi]

@Lunderberg Are these two test cases any different? One has pytest.xfail("Known failing test for vulkan") but not for the other.

tvm/tests/python/relay/test_op_level2.py

Line 199 in 548675f

class TestConv2D:

tvm/tests/python/relay/test_op_level2.py

Line 360 in 548675f

def test_conv2d_run(target, dev):

[Lunderberg]

Thank you for that catch. When refactoring the tests in #8947, I added the updated version of test_conv2d_run, but didn't remove the original. I have #8993 open to remove the redundant test_conv2d_run, and have double-checked that there aren't any others that snuck in.

[masahi]

@Lunderberg The last three items in test_any.py are not specific to vulkan (fails on cuda as well), so I think we should drop them from the list.

They don't work on gpu targets since we don't support dynamic height or width in conv2d, for example.