Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSGTS combined mode: use all profile where applicable #6146

Merged

Conversation

matusmarhefka
Copy link
Member

This commit introduces a small change into the SSGTS combined mode.
When testing a selected profile and test scenario does not contain
profiles metadata the test is rather performed using ALL virtual
profile instead of the selected (tested) profile.
This will prevent failures which might be caused by the tested profile
selecting different values in tested variables compared to defaults.
The ALL profile is always selecting default values.

What this change means?

  • If rule is general and behaves the same way no matter the profile
    (i.e. does not use variables) do not list profiles in the profiles
    metadata - rule will always be tested using the ALL profile.
  • If rule is applicable only to some profiles or you would like to test
    some specific regression when certain values are selected use the
    profiles metadata to limit test scenario to be ran only when testing
    profiles listed in the metadata.

This commit introduces a small change into the SSGTS combined mode.
When testing a selected profile and test scenario does not contain
`profiles` metadata the test is rather performed using ALL virtual
profile instead of the selected (tested) profile.
This will prevent failures which might be caused by the tested profile
selecting different values in tested variables compared to defaults.
The ALL profile is always selecting default values.

What this change means?
- If rule is general and behaves the same way no matter the profile
  (i.e. does not use variables) do not list profiles in the profiles
  metadata - rule will always be tested using the ALL profile.
- If rule is applicable only to some profiles or you would like to test
  some specific regression when certain values are selected use the
  profiles metadata to limit test scenario to be ran only when testing
  profiles listed in the metadata.
@matusmarhefka matusmarhefka added the Test Suite Update in Test Suite. label Oct 2, 2020
@matusmarhefka matusmarhefka added this to the 0.1.53 milestone Oct 2, 2020
@mildas
Copy link
Contributor

mildas commented Oct 2, 2020

@openscap-ci test this please

@mildas
Copy link
Contributor

mildas commented Oct 2, 2020

Before:

$ python3 test_suite.py combined --libvirt qemu:///system test-suite-rhel8 --datastream ../bu
ild/ssg-rhel8-ds.xml ospp
INFO - Performing combined test using profile: xccdf_org.ssgproject.content_profile_ospp
INFO - xccdf_org.ssgproject.content_rule_grub2_pti_argument
INFO - Script wrong_value.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script correct.pass.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script arg_not_there.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - xccdf_org.ssgproject.content_rule_grub2_uefi_password
INFO - Script password-set.pass.sh using profile xccdf_org.ssgproject.content_profile_ospp OK

After:

$ python3 test_suite.py combined --libvirt qemu:///system test-suite-rhel8 --datastream ../build/ssg-rhel8-ds.xml ospp
INFO - Performing combined test using profile: xccdf_org.ssgproject.content_profile_ospp
INFO - xccdf_org.ssgproject.content_rule_grub2_pti_argument
INFO - Script wrong_value.fail.sh using profile (all) OK
INFO - Script correct.pass.sh using profile (all) OK
INFO - Script arg_not_there.fail.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_grub2_uefi_password
INFO - Script password-set.pass.sh using profile (all) OK

I like this change because this often caused error results in test scenarios that were using default values different from profile-specific values.

In future, we would like to have test scenarios with variables that will have profile-specific values (or default).

@mildas
Copy link
Contributor

mildas commented Oct 2, 2020

@openscap-ci test this please

@mildas mildas merged commit 3143730 into ComplianceAsCode:master Oct 2, 2020
@matusmarhefka matusmarhefka deleted the ssgts_combined_all_profile branch October 5, 2020 10:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Test Suite Update in Test Suite.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants