Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Profile selection overloading in SSGTS combined mode causes issues #4952

Closed
matusmarhefka opened this issue Oct 29, 2019 · 2 comments
Closed
Labels
Test Suite Update in Test Suite.

Comments

@matusmarhefka
Copy link
Member

Description of problem:

Profile selections in test case metadata are ignored when running SSGTS in combined mode for certain profile. This causes that the test scenarios which are not targeted for the tested profile are also run and will fail.

Example:
The test policy_default_set.pass.sh for the configure_crypto_policy rule fails when running SSGTS for the ospp profile because it is implemented only for the standard profile and it is N/A for the ospp. This is caused by the SSGTS (combined mode) overloading profile metadata in test scenarios.

@matusmarhefka matusmarhefka added Python Test Suite Update in Test Suite. labels Oct 29, 2019
@jan-cerny
Copy link
Collaborator

I think that this behavior has been created because the test scenarios aren't labeled properly - eg. some rule is part of multiple profiles on multiple products but the test scenario is labeled only that it belongs to 1 profile. Labeling them properly is very costly, though. I think you would have to add a lot of profiles to the headers and then test it. Also maintaining it is be difficult - each time anybody changes a profile he should change all the test scenarios that test rules added/removed from the profile. I think we can't ask the our contributors to do this. Instead, we should rethink the profile metadata.

@evgenyz evgenyz removed the Python label Jul 15, 2021
@matusmarhefka
Copy link
Member Author

Closing, this has been addressed by #6123 and #6146

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Test Suite Update in Test Suite.
Projects
None yet
Development

No branches or pull requests

3 participants