Optest is unable to xfail on parametrized tests

[NicolasHug]

It is currently not possible to xfail on a parametrized test that is marked e.g. as

      "TestPSRoIAlign.test_faketensor__test_forward[x_dtype0-True-mps]": {
        "comment": "AssertionError: Dtypes torch.int64 and torch.int32 are not equal!",
        "status": "xfail"
      },

in the failures_dict. (I thought #111797 was a fix, but it's not enough)

From what I can tell, the problem is that here

pytorch/torch/testing/_internal/optests/generate_tests.py

Lines 748 to 753 in cf5479b

	def get_status(self, qualname: str, test_name: str) -> str:
	if qualname not in self.data:
	return "xsuccess"
	dct = self.data[qualname]
	if test_name not in dct:
	return "xsuccess"

test_name isn't found in dct (i.e. in the failures_dict) because test_name looks like TestPSRoIAlign.test_faketensor__test_forward, i.e. it's missing the [x_dtype0-True-mps] part. That's because this name is generated all the way up in

pytorch/torch/testing/_internal/optests/generate_tests.py

Lines 252 to 254 in cf5479b

	for attr in test_methods:
	for prefix, tester in test_utils.items():
	construct_method(attr, prefix, tester)

where attr is test_forward, but of course it's missing any parametrization info because this is only looking at the "static" test class object.

I'm not sure how to fix this without a significant rework of the whole opcheck logic, but perhaps @zou3519 has an idea?

Unfortunately, for now we cannot really use optests on the torchvision test suite. Note that we can't just put TestPSRoIAlign.test_faketensor__test_forward without the [...] part in the failure_dict, because that test actually passes on cpu and cuda, which would lead to an "unexpected success".

cc @zou3519

[albanD]

cc @jbschlosser