Add unit tests of XCCDF 1.2 elements #9617
Conversation
|
Start a new ephemeral environment with changes proposed in this pull request: Fedora Environment Oracle Linux 8 Environment |
|
I believe that the 2 similar blocks of code are OK because they work with different items: rules and groups. |
|
I am not a big fan of tests that contain an overwhelming amount of low-level asserts. Those tests may be challenging to maintain, because when an assert starts to fail, what to do? Modify the test, modify test data, or look into the source code? In some cases when the situation is unclear and time pressure is present, a wrong action may be taken. |
It can cause tracebacks when we call a method on None object. This doesn't happen during normal usage, but it can happen during unit tests.
This adds simple unit tests for _to_xml_element() methods of Rule, Profile, Value, and Group class in ssg.build_yaml module.
jan-cerny
force-pushed
the
xccdf12_unittests
branch
from
14f634d
to
faf7299
Compare
|
I have resolved the conflicts and rebased the PR on the top of the latest upstream master branch. |
|
I have removed the hard coded definition location paths. |
We will use xmldiff in unit tests of to_xml_element methods because it allows us to store expected XML outputs as separate files and also it allows us to detect all differences from the expected output immediately.
jan-cerny
force-pushed
the
xccdf12_unittests
branch
from
614edea
to
50a029a
Compare
The conditional import will make skip tests that require xmldiff and lxml because xmldiff isn't present on all systems where we would like to run our tests because xmldiff can be at this moment installed only using pip, it isn't packaged in most of Linux distributions.
|
We have discussed this to @matejak and we have found out it's also possible to use xmldiff with this type of tests it allows us to store expected XML outputs as separate files and also it allows us to detect all differences from the expected output immediately. I have add conditional imports to make the tests skip that require xmldiff and lxml when they aren't available because xmldiff isn't present on all systems where we would like to run our tests because xmldiff can be at this moment installed only using pip, it isn't packaged in most of Linux distributions. |
There was a problem hiding this comment.
I like this approach better. I think we should document the need for the xmldiff python package in the building CaC doc.
Can you add a new line of the files like tests/unit/ssg-module/data/accounts_tmout.xml or will that cause issues with the XML diff?
I think we can waive the code climate duplication errors.
|
I have add documentation text.
Unfortunately, xmldiff is sensitive on whitespace before and after tags. I don't like this behavior of xmldiff, but it seems that we will either have to live with that or invent some sort of whitespace normalization that would be executed before the test data would be passed to xmldiff. @matejak has pointed to me https://xmldiff.readthedocs.io/en/stable/api.html#using-formatters but we have found that it's related to the diff output instead of input. Then I did an experiment where I add a newline after a tag to the test data and whatever argument I put into this |
But now, an unexpected twist: we have found that this isn't true for the So we can save the actual outputs produced by the SUT to temporary files and then compare the temporary files with the test data files using the |
Originally, we thought xmldiff is sensitive on whitespace before and after tags. But now we have found that this isn't true for the diff_files method. When using the diff_trees it is sensitive on whitespace and when using diff_files method it isn't sensitive on whitespace. So we can save the actual outputs produced by the SUT to temporary files and then compare the temporary files with the test data files using the diff_files.
|
I have se diff_files instead of diff_trees |
|
Code Climate has analyzed commit 4b98c1a and detected 2 issues on this pull request. Here's the issue category breakdown:
The test coverage on the diff in this pull request is 0.0% (50% is the threshold). This pull request will bring the total coverage in the repository to 41.9% (0.6% change). View more on Code Climate. |
Description:
This PR adds simple unit tests for
_to_xml_element()methods of Rule, Profile, Value, and Group class in ssg.build_yaml module.The unit test fixtures will use example data created from real-world resolved rules, profile, value, and group that were taken from build executed on current master.
Rationale:
Increases testing of build scripts.