-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Integration tests #151
Integration tests #151
Conversation
ca42126
to
077961c
Compare
366e9a7
to
17514e6
Compare
plans/integration.fmf
Outdated
- name: Generate ARF files | ||
how: shell | ||
script: | ||
- ./generate_arf.sh ssg no fedora ${TMT_PLAN_DATA}/arf.xml |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we parametrize the product?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Or it should be marked as only executable in Fedora.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it can be parameterized, but how do we determine which product to use?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FMF must somehow provide the name of the platform is it trying to execute the test on. We might need to map FMF name to our product name, though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have an idea of how to map the distro value to product value.
17514e6
to
5294de1
Compare
5294de1
to
73d4371
Compare
73d4371
to
279de0f
Compare
Changes:
|
This PR creates a set of integration tests using the
tmt
tool, a script that generates ARF reports with different configurations, modifies the smoke test to use the given ARF file, and creates a GitHub action to run these integration tests weekly.The tests use ARF results generated with the latest content available as an SSG package and content from the ComplianceAsCode/content repository.
Some integration tests could have failed because the test used the latest released openscap-report package. The goal is to check the released package of openscap-repot is behave right with changes in content in ComplianceAsCode/content and with shipped content in the SSG package.
The integration test could be executed in Tesitng Farm with PackIt. It would need some adjustments to the content product. In Testing Farm could be provided RPM package from changes in PR. Execution of this test in Testing Farm takes about 30min.
That's why I decided to run the integration test scheduled with GitHub Actions. I think using these tests for changes in PR would be pointless.