Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Automatus in CI #11494

Merged
merged 3 commits into from
Feb 1, 2024
Merged

Conversation

jan-cerny
Copy link
Collaborator

Description:

Some rules depend on rule installed_OS_is_FIPS_certified which makes
it difficult to run test scenarios of these rules on different systems
that aren't certified by FIPS. For example, this happens when you run
test scenarios on Fedora container or CentOS container.

This situation is currently handled by a feature of Automatus using the
--add-product-to-fips-certified option which extends the OVAL in rule
installed_OS_is_FIPS_certified to make it pass on a selected platform.
Unfortunately, this depends on assumption that there exist an OVAL
definition installed_OS_is_${product}. After recent changes in build
system, it doesn't have to be true, because if this definition isn't
used by any rule, it gets removed by code that filters out unused
definitions.

This option depends on assumption that the given SCAP source data
stream contains definitions of all possible products which might
not be the case if the definiton isn't used by any rule in that
data stream. The feature is fragile

We will add a new Automatus option '--remove-fips-certified' which will
remove all <oval-def:extend_definition> elements that reference OVAL
definition for rule installed_OS_is_FIPS_certified from all other
rules. As a result, no rule will depend on
installed_OS_is_FIPS_certifed when this option will be used.

Review Hints:

python3 tests/automatus.py rule --libvirt qemu:///system ssgts_rhel8 --remove-fips-certified --scenario  correct_value.pass.sh sshd_use_approved_ciphers

Check the Automatus CI jobs.

@jan-cerny jan-cerny added the Test Suite Update in Test Suite. label Jan 29, 2024
@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Used by openshift-ci bot. label Jan 29, 2024
Copy link

openshift-ci bot commented Jan 29, 2024

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Copy link

github-actions bot commented Jan 29, 2024

Start a new ephemeral environment with changes proposed in this pull request:

Fedora Environment
Open in Gitpod

Oracle Linux 8 Environment
Open in Gitpod

@jan-cerny jan-cerny force-pushed the FIPS_certified_automatus branch 2 times, most recently from 2b1a253 to 13eb69d Compare January 29, 2024 15:55
Some rules depend on rule `installed_OS_is_FIPS_certified` which makes
it difficult to run test scenarios of these rules on different systems
that aren't certified by FIPS. For example, this happens when you run
test scenarios on Fedora container or CentOS container.

This situation is currently handled by a feature of Automatus using the
`--add-product-to-fips-certified` option which extends the OVAL in rule
`installed_OS_is_FIPS_certified` to make it pass on a selected platform.
Unfortunately, this depends on assumption that there exist an OVAL
definition `installed_OS_is_${product}`. After recent changes in build
system, it doesn't have to be true, because if this definition isn't
used by any rule, it gets removed by code that filters out unused
definitions.

We will add a new Automatus option '--remove-fips-certified' which will
remove all `<oval-def:extend_definition>` elements that reference OVAL
definition for rule `installed_OS_is_FIPS_certified` from all other
rules. As a result, no rule will depend on
`installed_OS_is_FIPS_ceritifed` when this option will be used.
Use `--remove-fips-certified` in CI instead of using
`--add-product-to-fips-certified` which is unreliable and broken
at this moment.
This option depends on assumption that the given SCAP source data
stream contains definitions of all possible products which might
not be the case if the definiton isn't used by any rule in that
data stream. The feature is fragile. We have replaced it by the
`--remove-fips-certified` option.
@jan-cerny
Copy link
Collaborator Author

This is a link to the output of the Automatus Fedora / Run Tests job: https://github.com/ComplianceAsCode/content/actions/runs/7707656693/job/21005286727?pr=11494. Now, I will remove the test commit so that I can make it ready for review.

@jan-cerny jan-cerny marked this pull request as ready for review January 30, 2024 07:58
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Used by openshift-ci bot. label Jan 30, 2024
@jan-cerny jan-cerny added this to the 0.1.73 milestone Jan 30, 2024
Copy link

codeclimate bot commented Jan 30, 2024

Code Climate has analyzed commit 3ca2eb0 and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 100.0% (50% is the threshold).

This pull request will bring the total coverage in the repository to 58.3% (0.0% change).

View more on Code Climate.

@jan-cerny
Copy link
Collaborator Author

/packit retest-failed

1 similar comment
@Mab879
Copy link
Member

Mab879 commented Jan 30, 2024

/packit retest-failed

@Mab879 Mab879 self-assigned this Jan 30, 2024
@Mab879
Copy link
Member

Mab879 commented Jan 30, 2024

@mildas are aware anything else that might need adjusting with this change?

@mildas
Copy link
Contributor

mildas commented Feb 1, 2024

I don't think. Other places where AutoMatus is used doesn't use this option.

@mildas mildas merged commit 0dabc1a into ComplianceAsCode:master Feb 1, 2024
44 of 45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Test Suite Update in Test Suite.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants