Generate OCIL different way #7794
Conversation
This commit adds the option to build OCIL file to the build_shorthand.py script. Currently, the OCIL data are present in compiled rules. The build_shorthand.py script iterates over compiled rules when composing the shorthand and puts the data to the generated shorthand XML. Then, a series of XSLT transformations is employed during the build process to extract the OCIL from the shorthand XML and to replace OCIL data in the shorthand XML by a reference. Instead, we can create the OCIL file directly, during processing of rule data because we already have the data in the memory. That will enable us to simplify the build process in future. We won't need to process the shorthand XML to extract the OCIL, we will be able to use the OCIL generated by this script and therefore we won't need some XSLT transformation modifying shorthand XML.
We will generate OCIL during the generation of shorthand XML,
in the same build step as shorthand XML is generated. We will
use the new feature of build_shorthand.py from the previous commit.
The OCIL file generated by build_shorthand.py is the same as
ocil-unlinked.xml previously produced by XSLT.
The shorthand XML file generated by build_shorthand.py now
contains check-content-ref elements instead of the OCIL checks
from the moment of its creation. Therefore, there is no need to replace
OCIL checks by check-content-ref elements by using XSLT transformations.
This allows us to remove the following CMake targets:
* generate-internal-${PRODUCT}-ocil-unlinked.xml - ocil-unlinked.xml
will be generated in the generate-internal-${PRODUCT}-shorthand.xml
target
* generate-internal-${PRODUCT}-xccdf-unlinked-ocilrefs.xml - it isn't
needed to replace the OCIL check content by reference in XCCDF file
because the build_shorthand.py can put references to OCIL right to
the shorthand XML.
As a result, the CMake target
generate-internal-${PRODUCT}-linked-xccdf-oval-ocil.xml
can directly depend on OCIL produced in CMake target
generate-internal-${PRODUCT}-shorthand.xml.
We don't need to add XCCDF Value "conditional_clause" anymore because it
was removed by the transformation that we remove.
Finally, the change allows us to remove 2 XSLT files completely.
The target generate-internal-${PRODUCT}-shorthand.xml now started
to produce also OCIL file (ocil-unlinked.xml) so we rename the
target to better describe its situation.
Move the code specific for each rule from the LinearLoader class to the Rule class.
Prevent "None" appearing in the queestion_text element
|
Hello @jan-cerny! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:
|
yuumasato
left a comment
yuumasato
left a comment
There was a problem hiding this comment.
Nice work!
The build process got simpliifed while still outputting the same OCIL content.
The TODOs or enhancements can be done in subsequent PRs.
|
I have realized that build_shorthand.py probably isn't the best place where OCIL generation should happen. But I think that at this moment we don't have any "best place". OCIL generation can happen in any of our scripts that iterate over compiled rules. There are multiple scripts that iterate over compiled rules, for example the combine_remediations.py, build_shorthand.py, build_templated_content.py. Each of them loads compiled rules from the disk and processes them, which means that compiled rules are loaded and processed multiple times during a build of a single product. I think that one way that we could go in future is to merge these in a single script so that the compiled rules will be loaded and processed only once. |
3 participants
Description:
OCIL will be generated directly to a file while generating shorthand XML (in other words while processing compiled/resolved rules).
For more details, please read commit messages of every commit.
To facilitate review you can use the new feature of compare_ds.py to compare OCILs in the built data stream which is proposed in PR #7793.
Rationale:
Current way of generating OCIL is to put the OCIL data to shorthand XML, then extract it using XSLT, then replace it by reference using XSLT. This is complex and not necessary. The way proposed in this PR is simpler than the current way. We will generate OCIL while processing the data we have in memory. This reduces the amount of build steps to remove some build targets and removes some XSLT templates. In future, it can help when getting rid of shorthand because it removes some dependencies of shorthand.xml.
The disadvantage is a possible confusion caused by the fact that now the build_shorthand.py script generates 2 output files.