Use distributed product properties #10554

matejak · 2023-05-11T16:13:27Z

Description:

~~The PR builds on top of #10529. After that one is merged, this should get rebased in order to become easily reviewable.~~

It introduces product properties that can be read from a directory that can contain jinja2-powered yaml files that are processed in lexicographic order.
During processing of those files, they have access to the existing product properties. Properties read from those yaml files can be overriden by later files, while the properties specified in product yaml always stay on top.

Rationale:

This change will allow to move aspects of products from constants and source code in general to human-accessible yaml files in a top-level directory.

Review Hints:

Diff compiled product yamls from before this PR with once produced after the PR is applied. There should be no difference.
There are three kinds of changes in this PR, each of them is relatively small, but they somehow added up:
- The new capability and its unit-test and documentation,
- Modifications to existing utility scripts and tests that don't have access to compiled products, so they have to read product properties themselves, and
- Extraction of two product properties from the product yaml and source code to the product properties yamls.

openshift-ci · 2023-05-11T16:13:31Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

github-actions · 2023-05-11T16:14:58Z

Start a new ephemeral environment with changes proposed in this pull request:

rhel8 (from CTF) Environment (using Fedora as testing environment)

Fedora Testing Environment

Oracle Linux 8 Environment

jan-cerny

great idea!

docs/manual/developer/06_contributing_with_content.md

products/fedora/product.yml

jan-cerny · 2023-05-17T11:44:44Z

docs/manual/developer/06_contributing_with_content.md

+Use this functionality to associate product properties with product versions,
+so you can use only product properties in the content.
+In other words, use this functionality to avoid referencing product versions in macros used in checks or remediations.
+Instead, use properties that directly relate to configurations being checked or set, and that help to reveal the intention of the check or remediation code.


This is now way better!

We probably should have a blog post in which we would show the "good" and "bad" examples.

product_properties/20-grub.yml

jan-cerny · 2023-05-17T14:39:23Z

ssg/products.py

+        data.update(self._acquired_data)
+        data.update(self._primary_data)


I think it would be great to print an error if there are multiple definitions of the same property.

utils/rule_dir_json.py

jan-cerny · 2023-05-17T14:46:44Z

product_properties/README.md

+Processing of those files can use `jinja2`, and macros or conditionals have access to product properties defined previously and in the `product.yml`.
+Properties read from these yaml files can be overriden by properties from files processed later, while the properties specified in the `product.yml` override all previous definitions.
+
+Conventionally, use the filename prefix `10-` to define defaults, and use subsequent prefixes to override those defaults based on products.


I don't like it, I would prefer to have a single source of truth, i.e. define a given option only in 1 file.

This is tricky: YAML requires that in mappings, keys have to be unique, so setting a value to the default and redefining it later isn't legal, although libraries may support it.
The idea to define everything in one file is reasonable to me, so here are the options:

What is illegal:

key: value key: another-value

Dedicated default section:

default: key: value overrides: key: another-value

while the build system has to write overrides over defaults.

List of mappings:

- key: value - key: another-value

while the build system has to assemble the set of key-value mappings gradually by iterating over the list.

I think that the first option can be unreliable and the third option can be unexpected by newcomers so I prefer the second option because it's explicit. But let's take a look how it fits into the whole context of the "product properties" feature.

What are the benefits of having the key overrides in centrally placed lexicographically ordered files?
Why not keep a centrally defined default, and each product overrides properties in its own product.yml, or maybe a product property file in the product directory, like products/rhel8/product_properties.yml?
This also facilitates the addition and removal of a product.
For example, one can copy the file with the default values, and tweak / remove keys.

What are the benefits of having the key overrides in centrally placed lexicographically ordered files? Why not keep a centrally defined default, and each product overrides properties in its own product.yml, or maybe a product property file in the product directory, like products/rhel8/product_properties.yml? This also facilitates the addition and removal of a product. For example, one can copy the file with the default values, and tweak / remove keys.

Sorry, race condition. The override possibility is no longer considered, and if we introduce core product attributes s.a. "family" (ubuntu, debian, ol, rhel) and major version (8, 2204), we may even make the core product properties available at other properties definition time, which would take the ordering away completely.

product_properties/20-grub.yml

yuumasato · 2023-05-24T12:17:08Z

product_properties/README.md

+Processing of those files can use `jinja2`, and macros or conditionals have access to product properties defined previously and in the `product.yml`.
+Properties read from these yaml files can be overriden by properties from files processed later, while the properties specified in the `product.yml` override all previous definitions.
+
+Conventionally, use the filename prefix `10-` to define defaults, and use subsequent prefixes to override those defaults based on products.


What are the benefits of having the key overrides in centrally placed lexicographically ordered files?
Why not keep a centrally defined default, and each product overrides properties in its own product.yml, or maybe a product property file in the product directory, like products/rhel8/product_properties.yml?
This also facilitates the addition and removal of a product.
For example, one can copy the file with the default values, and tweak / remove keys.

matejak · 2023-05-25T12:08:10Z

The code has been altered to use the default/overrides keys, and tests and docs have been updated accordingly, and I believe that questions were addressed.
I like my nested ifs, as they either raise an easy-to-understand exception, so I like to separate "is it a dict" from extracting data from that dict.

jan-cerny

I like this change very much. I believe it will help us keep the data better organized and it will make the content more approachable to the authors.

I have tried to change some keys or override some keys and the behavior I observed was expected.

Please resolve the CodeClimate problem and take a look at the directory structure docs.

docs/manual/developer/06_contributing_with_content.md

Product properties can be expanded by other means, so the only way how to make sure that they remain stable is to check references for stability.

- Assign the grub helper exec to the product - Remove most of the references to product names from grub macros - Substitute literal invocations of e.g. grubby with macros

Grubby is not used in the context of grub, but in another context of BLS utility, and the macro doesn't understand this.

jan-cerny · 2023-06-13T12:39:03Z

@ComplianceAsCode/ubuntu-maintainers @ComplianceAsCode/suse-maintainers @ComplianceAsCode/oracle-maintainers Can you take a look, please?

freddieRv

Changes affecting OL products LGTM.

I like this new way to set product properties. Thanks for the efforts @matejak!

dodys

overall Ubuntu changes lgtm, thanks for doing this!

jan-cerny · 2023-06-21T11:26:00Z

@teacup-on-rockingchair PTAL

teacup-on-rockingchair

Besides the grubby vs grub2-mkconfig, I think rest of the changes are OK on SLE platforms. Nice improvement 🙇

tests/data/product_stability/sle15.yml

jan-cerny · 2023-06-29T13:41:29Z

@matejak ping

codeclimate · 2023-06-29T16:10:47Z

Code Climate has analyzed commit 1ca395c and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 100.0% (50% is the threshold).

This pull request will bring the total coverage in the repository to 53.2% (0.4% change).

View more on Code Climate.

jan-cerny · 2023-07-03T07:50:46Z

@teacup-on-rockingchair Can you check if the latest changes feel accurate for you to approve this PR?

CODEOWNERS

jan-cerny · 2023-07-17T06:39:07Z

@matejak PTAL at @marcusburghardt comment
@teacup-on-rockingchair Can you check if the latest changes are OK for you?

marcusburghardt

LGTM

marcusburghardt · 2023-07-18T10:07:05Z

I reviewed the PR again now and confirmed that all pending comments were properly managed. I marked some simpler comments as resolved while kept others open because they have useful information.

This is new to the project and improvements are naturally expected along the time but the current state is pretty good to land. We can merge it and have its benefits already in 0.1.69 release.

teacup-on-rockingchair

Great stuff 🙇

openshift-ci bot added the do-not-merge/work-in-progress Used by openshift-ci bot. label May 11, 2023

matejak force-pushed the real_properties branch 6 times, most recently from f0849c0 to 7e53bcc Compare May 17, 2023 08:58

jan-cerny reviewed May 17, 2023

View reviewed changes

docs/manual/developer/06_contributing_with_content.md Outdated Show resolved Hide resolved

docs/manual/developer/06_contributing_with_content.md Outdated Show resolved Hide resolved

products/fedora/product.yml Show resolved Hide resolved

matejak force-pushed the real_properties branch 2 times, most recently from 2fe1174 to 3a40f80 Compare May 17, 2023 11:30

jan-cerny reviewed May 17, 2023

View reviewed changes

matejak force-pushed the real_properties branch from 3a40f80 to 4769dbd Compare May 17, 2023 12:03

matejak marked this pull request as ready for review May 17, 2023 13:13

matejak requested review from a team as code owners May 17, 2023 13:13

openshift-ci bot removed the do-not-merge/work-in-progress Used by openshift-ci bot. label May 17, 2023

jan-cerny self-assigned this May 17, 2023

jan-cerny reviewed May 17, 2023

View reviewed changes

matejak mentioned this pull request May 22, 2023

Add a product stability test #10606

Merged

matejak force-pushed the real_properties branch from 4769dbd to b3744b9 Compare May 24, 2023 08:24

yuumasato reviewed May 24, 2023

View reviewed changes

matejak force-pushed the real_properties branch from 016fd5f to 75ce8e8 Compare May 25, 2023 10:48

jan-cerny requested changes May 26, 2023

View reviewed changes

docs/manual/developer/06_contributing_with_content.md Outdated Show resolved Hide resolved

matejak force-pushed the real_properties branch from 75ce8e8 to 85a6075 Compare May 29, 2023 12:40

jan-cerny added this to the 0.1.69 milestone May 29, 2023

matejak added 6 commits June 9, 2023 18:22

Expand the product stability test to autofix reference data

5c02d15

Guard product test references, not products

1ba3b1a

Product properties can be expanded by other means, so the only way how to make sure that they remain stable is to check references for stability.

Consolidate grub rules

1f85974

- Assign the grub helper exec to the product - Remove most of the references to product names from grub macros - Substitute literal invocations of e.g. grubby with macros

Simplify aide rules by leveraging product properties

9fbe3c5

Add support for loading Jinja without the product context

2dabca3

Revert macro usage in zipl rules

79b1fd8

Grubby is not used in the context of grub, but in another context of BLS utility, and the macro doesn't understand this.

matejak force-pushed the real_properties branch from f2e6aee to 79b1fd8 Compare June 9, 2023 16:22

jan-cerny added help-wanted This PR/Issue needs help to go forward. Infrastructure Our content build system Highlight This PR/Issue should make it to the featured changelog. labels Jun 13, 2023

freddieRv approved these changes Jun 13, 2023

View reviewed changes

dodys approved these changes Jun 16, 2023

View reviewed changes

teacup-on-rockingchair reviewed Jun 26, 2023

View reviewed changes

tests/data/product_stability/sle15.yml Outdated Show resolved Hide resolved

Fix the grub update executable for SLE15

1ca395c

marcusburghardt reviewed Jul 3, 2023

View reviewed changes

CODEOWNERS Show resolved Hide resolved

jan-cerny approved these changes Jul 10, 2023

View reviewed changes

matejak mentioned this pull request Jul 17, 2023

Fix STIG references for SLE15 #10850

Merged

marcusburghardt requested a review from teacup-on-rockingchair July 18, 2023 09:44

marcusburghardt approved these changes Jul 18, 2023

View reviewed changes

teacup-on-rockingchair approved these changes Jul 18, 2023

View reviewed changes

jan-cerny merged commit 60dc337 into ComplianceAsCode:master Jul 18, 2023
33 of 34 checks passed

vojtapolasek mentioned this pull request Jul 18, 2023

update version to 0.1.70 #10865

Merged

Mab879 mentioned this pull request Jul 18, 2023

Automatus Sanity Tests Are Failing #10866

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use distributed product properties #10554

Use distributed product properties #10554

matejak commented May 11, 2023 •

edited

openshift-ci bot commented May 11, 2023

github-actions bot commented May 11, 2023 •

edited

jan-cerny left a comment

jan-cerny May 17, 2023

jan-cerny May 17, 2023

jan-cerny May 17, 2023

matejak May 18, 2023

jan-cerny May 24, 2023

yuumasato May 24, 2023 •

edited

matejak May 24, 2023

yuumasato May 24, 2023 •

edited

matejak commented May 25, 2023

jan-cerny left a comment

jan-cerny commented Jun 13, 2023

freddieRv left a comment

dodys left a comment

jan-cerny commented Jun 21, 2023

teacup-on-rockingchair left a comment

jan-cerny commented Jun 29, 2023

codeclimate bot commented Jun 29, 2023

jan-cerny commented Jul 3, 2023

jan-cerny commented Jul 17, 2023

marcusburghardt left a comment

marcusburghardt commented Jul 18, 2023

teacup-on-rockingchair left a comment

		data.update(self._acquired_data)
		data.update(self._primary_data)

Use distributed product properties #10554

Use distributed product properties #10554

Conversation

matejak commented May 11, 2023 • edited

Description:

Rationale:

Review Hints:

openshift-ci bot commented May 11, 2023

github-actions bot commented May 11, 2023 • edited

jan-cerny left a comment

Choose a reason for hiding this comment

jan-cerny May 17, 2023

Choose a reason for hiding this comment

jan-cerny May 17, 2023

Choose a reason for hiding this comment

jan-cerny May 17, 2023

Choose a reason for hiding this comment

matejak May 18, 2023

Choose a reason for hiding this comment

jan-cerny May 24, 2023

Choose a reason for hiding this comment

yuumasato May 24, 2023 • edited

Choose a reason for hiding this comment

matejak May 24, 2023

Choose a reason for hiding this comment

yuumasato May 24, 2023 • edited

Choose a reason for hiding this comment

matejak commented May 25, 2023

jan-cerny left a comment

Choose a reason for hiding this comment

jan-cerny commented Jun 13, 2023

freddieRv left a comment

Choose a reason for hiding this comment

dodys left a comment

Choose a reason for hiding this comment

jan-cerny commented Jun 21, 2023

teacup-on-rockingchair left a comment

Choose a reason for hiding this comment

jan-cerny commented Jun 29, 2023

codeclimate bot commented Jun 29, 2023

jan-cerny commented Jul 3, 2023

jan-cerny commented Jul 17, 2023

marcusburghardt left a comment

Choose a reason for hiding this comment

marcusburghardt commented Jul 18, 2023

teacup-on-rockingchair left a comment

Choose a reason for hiding this comment

matejak commented May 11, 2023 •

edited

github-actions bot commented May 11, 2023 •

edited

yuumasato May 24, 2023 •

edited

yuumasato May 24, 2023 •

edited