Implement a tool for parsing profiles and outputing rules #10455

rhmdnd · 2023-04-11T18:44:01Z

This adds the basic skeleton of a tool that parses CIS profiles using an
XLSX parser and converts controls to rules that we can use in OpenSCAP.

The primary purpose is to help maintain CIS profiles by generating most
of the boilerplate rule template.

github-actions · 2023-04-11T18:45:43Z

Start a new ephemeral environment with changes proposed in this pull request:

Fedora Environment

Oracle Linux 8 Environment

Mab879

Thanks for the PR!

I have few other items that are not in my code comments:

Please take look at the Code Climate findings. Especially the PEP8 findings.
Add some documentation about this script in docs/manual/developer/05_tools_and_utilities.md. Explaining what format of spreadsheets is needed would be helpful.

utils/generate_profile.py

Mab879 · 2023-04-13T12:54:19Z

utils/generate_profile.py

+
+yaml.add_representer(literal_unicode, literal_unicode_representer)
+
+parser = argparse.ArgumentParser(


please move this to a method

utils/generate_profile.py

rhmdnd · 2023-04-19T01:50:29Z

Thanks for the review @Mab879 - I still need to get the documentation ironed out.

rhmdnd · 2023-04-19T01:51:40Z

utils/generate_profile.py

+    def parse(self):
+        return NotImplemented
+
+class Section:


The Section, Control, and Benchmark classes really could be their own library. That'd actually be pretty useful in other scripts, too.

jhrozek · 2023-04-19T13:14:19Z

utils/generate_profile.py

+        self.description = None
+        self.level = None
+        self.remediation = None
+        self.rationale = None


to bring a slack conversation over here and tag @Mab879 - would it be a good idea to merge this class with the control class from srg/controls.py?

There is another version of this in #10469

@Mab879 what are your thoughts on a shared library that modes a benchmark?

I think that shared library would sense.

Ok - I'll help get the deps squared away in #10474 and #10473. Then I'll start pulling the common bits into a shared library that we can import as a python-specific requirement.

I threw together a quick library https://github.com/rhmdnd/pycompliance

This is a port from a patch proposed to another project: ComplianceAsCode/content#10455 Documentation and release will come after initial round of feedback.

rhmdnd · 2023-04-26T21:55:04Z

utils/generate_profile.py

+
+import json
+import yaml
+from pycompliance import pycompliance


This is based on the latest content in https://github.com/rhmdnd/pycompliance

If folks are ok with that approach, I'll package it and release to PyPI.

Is it correct to say that this library will be focused on making compliance content in various formats (xslx, pdf) available in CaC content formats (i.e. controls, profiles, rule in yaml)?

I wasn't sure if we'd want to put all the parsing in the pycompliance library. I kept it pretty limited so that it only focused on the common parts of my script and the script from #10469

rhmdnd · 2023-04-27T03:13:07Z

The tool currently outputs the following if you have the benchmark available locally:

$ python utils/generate_profile.py -i ~/Downloads/CIS_RedHat_OpenShift_Container_Platform_Benchmark_v1.3.0.xlsx -p ocp generate -c 2.3
documentation_complete: false
prodtype: ocp
title: |-
  Ensure that the --auto-tls argument is not set to true
description: |-
  Do not use self-signed certificates for TLS.
rationale: |-
  etcd is a highly-available key value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should not be available to unauthenticated clients. You should enable the client authentication via valid certificates to secure the access to the etcd service.
severity: PLACEHOLDER
references: PLACEHOLDER
ocil: PLACEHOLDER
ocil_clause: PLACEHOLDER
warnings: PLACEHOLDER
template: PLACEHOLDER

utils/generate_profile.py

yuumasato

I like and support the direction the library and script are going

yuumasato · 2023-05-02T16:13:05Z

utils/generate_profile.py

+        raise NotImplemented
+
+
+class XLSXParser(Parser):


Maybe the XLSXParser should actually be in pycompliance library?

It could be. I figured once we take that step, we may get asked to support parsing additional formats (PDFs).

yuumasato · 2023-05-02T16:24:23Z

utils/generate_profile.py

+        cols = [
+            'section #',
+            'recommendation #',
+            'profile',
+            'title',
+            'assessment status',
+            'description',
+            'remediation procedure',
+            'rationale statement',
+            'audit procedure']


And in the future these columns would be easily customizable for other spreadsheets, not only CIS.

yuumasato · 2023-05-02T16:29:53Z

utils/generate_profile.py

+
+import json
+import yaml
+from pycompliance import pycompliance


Is it correct to say that this library will be focused on making compliance content in various formats (xslx, pdf) available in CaC content formats (i.e. controls, profiles, rule in yaml)?

yuumasato

@Mab879 Do you have anything to add?

Mab879 · 2023-05-03T14:59:20Z

@Mab879 Do you have anything to add?

I would like some basic docs before this merged.

Mab879

Looks good other than that one code climate issue. Once that is fixed we should be good to merge.

This adds the basic skeleton of a tool that parses CIS profiles using an XLSX parser and converts controls to rules that we can use in OpenSCAP. The primary purpose is to help maintain CIS profiles by generating most of the boilerplate rule template.

rhmdnd · 2023-05-04T20:26:01Z

Looks good other than that one code climate issue. Once that is fixed we should be good to merge.

Fixed - thanks for the reviews!

Mab879

Thanks!

codeclimate · 2023-05-04T21:21:19Z

Code Climate has analyzed commit 1c5812d and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 100.0% (50% is the threshold).

This pull request will bring the total coverage in the repository to 52.4% (0.0% change).

View more on Code Climate.

rhmdnd · 2023-05-04T21:27:47Z

Thanks!

I'll get a pycompliance release on PyPI tomorrow and we can iterate there.

Mab879 · 2023-05-04T21:31:00Z

/packit build

Mab879 self-assigned this Apr 11, 2023

Mab879 added the Infrastructure Our content build system label Apr 11, 2023

Mab879 requested changes Apr 13, 2023

View reviewed changes

marcusburghardt mentioned this pull request Apr 18, 2023

Add script processing the PCI DSS standard #10469

Open

rhmdnd force-pushed the profile-parser-script branch from a3670cf to 2fe2bbb Compare April 19, 2023 01:49

rhmdnd commented Apr 19, 2023

View reviewed changes

rhmdnd force-pushed the profile-parser-script branch from 2fe2bbb to 60143a7 Compare April 19, 2023 13:00

jhrozek reviewed Apr 19, 2023

View reviewed changes

rhmdnd added a commit to rhmdnd/pycompliance that referenced this pull request Apr 26, 2023

Add basic functionality

9eb9ebe

This is a port from a patch proposed to another project: ComplianceAsCode/content#10455 Documentation and release will come after initial round of feedback.

rhmdnd force-pushed the profile-parser-script branch from 60143a7 to dba466f Compare April 26, 2023 21:49

rhmdnd commented Apr 26, 2023

View reviewed changes

rhmdnd force-pushed the profile-parser-script branch from dba466f to 85c2246 Compare April 27, 2023 03:11

rhmdnd commented Apr 27, 2023

View reviewed changes

utils/generate_profile.py Outdated Show resolved Hide resolved

rhmdnd force-pushed the profile-parser-script branch 3 times, most recently from 10df58c to 5d30159 Compare April 27, 2023 20:50

rhmdnd mentioned this pull request Apr 27, 2023

Add control file for OCP 1.3.0 #10505

Merged

rhmdnd force-pushed the profile-parser-script branch from 5d30159 to 76caca0 Compare April 28, 2023 12:52

yuumasato reviewed May 2, 2023

View reviewed changes

yuumasato approved these changes May 3, 2023

View reviewed changes

rhmdnd force-pushed the profile-parser-script branch from 76caca0 to 7222f78 Compare May 3, 2023 21:57

openshift-merge-robot added the needs-rebase Used by openshift-ci bot. label May 3, 2023

rhmdnd force-pushed the profile-parser-script branch from 7222f78 to 6615f4b Compare May 3, 2023 21:58

openshift-merge-robot removed the needs-rebase Used by openshift-ci bot. label May 3, 2023

Mab879 assigned yuumasato May 4, 2023

Mab879 added this to the 0.1.68 milestone May 4, 2023

Mab879 requested changes May 4, 2023

View reviewed changes

rhmdnd force-pushed the profile-parser-script branch from 6615f4b to 1c5812d Compare May 4, 2023 20:25

Mab879 approved these changes May 4, 2023

View reviewed changes

Mab879 merged commit b47f3c2 into ComplianceAsCode:master May 9, 2023
32 of 33 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement a tool for parsing profiles and outputing rules #10455

Implement a tool for parsing profiles and outputing rules #10455

rhmdnd commented Apr 11, 2023

github-actions bot commented Apr 11, 2023

Mab879 left a comment

Mab879 Apr 13, 2023

rhmdnd commented Apr 19, 2023 •

edited

rhmdnd Apr 19, 2023

jhrozek Apr 19, 2023

rhmdnd Apr 19, 2023 •

edited

Mab879 Apr 20, 2023

rhmdnd Apr 21, 2023

rhmdnd Apr 26, 2023

rhmdnd Apr 26, 2023

yuumasato May 2, 2023

rhmdnd May 2, 2023

rhmdnd commented Apr 27, 2023

yuumasato left a comment

yuumasato May 2, 2023

rhmdnd May 2, 2023

yuumasato May 2, 2023

yuumasato May 2, 2023

yuumasato left a comment

Mab879 commented May 3, 2023

Mab879 left a comment

rhmdnd commented May 4, 2023

Mab879 left a comment

codeclimate bot commented May 4, 2023

rhmdnd commented May 4, 2023

Mab879 commented May 4, 2023


		yaml.add_representer(literal_unicode, literal_unicode_representer)

		parser = argparse.ArgumentParser(

Implement a tool for parsing profiles and outputing rules #10455

Implement a tool for parsing profiles and outputing rules #10455

Conversation

rhmdnd commented Apr 11, 2023

github-actions bot commented Apr 11, 2023

Mab879 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhmdnd commented Apr 19, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhmdnd Apr 19, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhmdnd commented Apr 27, 2023

yuumasato left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuumasato left a comment

Choose a reason for hiding this comment

Mab879 commented May 3, 2023

Mab879 left a comment

Choose a reason for hiding this comment

rhmdnd commented May 4, 2023

Mab879 left a comment

Choose a reason for hiding this comment

codeclimate bot commented May 4, 2023

rhmdnd commented May 4, 2023

Mab879 commented May 4, 2023

rhmdnd commented Apr 19, 2023 •

edited

rhmdnd Apr 19, 2023 •

edited