[BUG] Incorrect Tempest test identifiers in test lists

[michal-gubricky]

Summary

Some Tempest tests currently defined in our test lists are not being discovered correctly. This is due to missing the required [,negative] suffix in their identifiers. Probably there are some more incorrect identifiers.

Problem Description

When defining specific Tempest tests, simply using the class path is often insufficient for negative tests. If the [,negative] suffix (including the test ID) is missing, the tempest run command fails to match the test, resulting in it being ignored during execution.

Example

For instance, the test test_update_non_existent_security_group is currently not correctly identified unless the full string is provided.

Incorrect/Incomplete reference:

tempest.api.compute.security_groups.test_security_groups_negative.SecurityGroupsNegativeTestJSON.test_update_non_existent_security_group

Correct reference:

tempest.api.compute.security_groups.test_security_groups_negative.SecurityGroupsNegativeTestJSON.test_update_non_existent_security_group[id-27edee9c-873d-4da6-a68a-3c256efebe8f,negative]

The correct identifiers can be verified by running the following command within the tempest container/environment:

tempest run --list-tests | grep test_update_non_existent_security_group

[toothstone]

As far as I understand the issue currently there are two things to look at:

• missing qualifiers(?) such as negative or image
• outdated UUIDs (I simply used those from the OPC 2022.11.json, didn't expect them to change)

Both seem to be avoided by my earlier regex-based test selection approach(includes 16 admin level tests in addition to the current SCS-0128 definition), however that is very much not human-readable and as such a poor fit for standardization text.

[michal-gubricky]

As far as I understand the issue currently there are two things to look at:

• missing qualifiers(?) such as negative or image
• outdated UUIDs (I simply used those from the OPC 2022.11.json, didn't expect them to change)

Yes, you understand it correctly.

[mbuechse]

I would be surprised if the UUIDs changed! (With OpenStack, I probably shouldn't be.)
Can we get a diff?

[depressiveRobot]

@anjastrunk How did you solve the problem with the wrong UUIDs? Maybe you can help here.

[Schmexs]

@depressiveRobot
I fixed it by creating a new "full-tests" list with
stestr list > all_discovered_tests.txt

Then I created a Python Script to filter the new IDs
update_test_ids.py

I created a new file with: python3 update_test_ids.py tempest-tests-non-admin.lst all_discovered_tests.txt ready_to_run.lst

[depressiveRobot]

@michal-gubricky Does this fixed list created with the help of the tooling from @Schmexs work for you?

https://gist.githubusercontent.com/depressiveRobot/1d38cd338a52712d12a67954acd8c626/raw/169311ac1e10c0c694dd0811796b67170e915f13/tempest-tests-non-admin-fixed.lst

[michal-gubricky]

@michal-gubricky Does this fixed list created with the help of the tooling from @Schmexs work for you?

https://gist.githubusercontent.com/depressiveRobot/1d38cd338a52712d12a67954acd8c626/raw/169311ac1e10c0c694dd0811796b67170e915f13/tempest-tests-non-admin-fixed.lst

Yes, I can confirm that with this list, I was able to run all 260 test cases.