Description
We had/have lot of files in repository (remediations/oval). Most of such files was once generated from some templates & csv files and then were these files pushed into repository. Later, some of them were updated manually.
Every new product is created by copy&paste of old product. So we have LOT of duplicates in files. When somebody fix e.g. remediations in RHEL7, RHEL6 remediations still contain old code.
It is very hard to maintain, because in such amount of files, we cannot see e.g. we have remediation script for some rule, but we miss oval check for it.
When I needed to add ansible remediations to several products, it wasn't able to do that without copy/pasting of new ansible. So I started to do cleaning of generated files and started to generate these files during build. It is much easier to change it from one place than in maybe hundreds of generated files. It is partially done( for remediations).
Currently I need to finish this change with oval files.
Final set of scripts for product are build from:
- shared files generated from templates - e.g.
package_<package_name>_removed- shared for all products - shared static files - for very specific check, or special case of standard check
- product files generated from templates
- product static files
So e.g. you can have shared template for package*removed (dnf remove <package name>) and many product can use it without copy&paste. But If you create our own package*removed template in our product, you can use own implementation e.g. (apt-get remove <package name>).
How are files generated : CSV + template => files
e.g. packages_removed.csv + template_package_removed => package_abrt_removed.xml, package_acpid_removed.xml, ...
So my current plan(for every product):
- remove generated files
- update .csv files and start to use templates (be able to generate files from point 1)
- files what cannot be generated move to "static" folder
- delete input folder
This should reduce amount of files and then it will be easier to fix bugs or find duplicated files(it is easier to compare 1 vs 1 template, than compare 20 vs 20 files).
Once we reduce amount of files, it should be much easier to modify whole structure.
I am going to produce lot of PRs.
- they should mostly contains points 1 and 2 from plan
- they shouldn't change output datastreams (no bugfixing in general)
- I use https://github.com/OpenSCAP/scap-security-guide/blob/master/shared/utils/compare_generated.sh to check whether I changed something - I compare datastreams before and after changes
- I check it for oval_5.10 and oval_5.11 https://github.com/OpenSCAP/scap-security-guide/blob/master/shared/product-make.include#L47
- I will try to do independent PR, so reviewer should be able to merge several PRs into one and then check them
Do you have any better ideas?