OCPBUGS-11696: Update encryption type to support 4.13 deployments #10974

rhmdnd · 2023-08-09T19:35:28Z

The latest CIS guidance allows OpenShift users to configure the
encryption type for the API server to aescbc or aescgm. Previously,
the variable used to evaluate this check only supported aescbc. This
commit update the variable to accept either type, as per the CIS
guidance.

applications/openshift/api-server/var_apiserver_encryption_type.var

github-actions · 2023-08-09T19:37:01Z

Start a new ephemeral environment with changes proposed in this pull request:

ocp4 (from CTF) Environment (using Fedora as testing environment)

Fedora Testing Environment

Oracle Linux 8 Environment

rhmdnd · 2023-08-09T19:37:32Z

/test help

openshift-ci · 2023-08-09T19:37:35Z

@rhmdnd: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test e2e-aws-ocp4-cis
/test e2e-aws-ocp4-cis-node
/test e2e-aws-ocp4-e8
/test e2e-aws-ocp4-high
/test e2e-aws-ocp4-high-node
/test e2e-aws-ocp4-moderate
/test e2e-aws-ocp4-moderate-node
/test e2e-aws-ocp4-pci-dss
/test e2e-aws-ocp4-pci-dss-node
/test e2e-aws-ocp4-stig
/test e2e-aws-ocp4-stig-node
/test e2e-aws-rhcos4-e8
/test e2e-aws-rhcos4-high
/test e2e-aws-rhcos4-moderate
/test e2e-aws-rhcos4-stig
/test images

Use /test all to run the following jobs that were automatically triggered:

pull-ci-ComplianceAsCode-content-master-images

In response to this:

/test help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

rhmdnd · 2023-08-09T19:38:35Z

/test e2e-aws-ocp4-cis
/test e2e-aws-ocp4-cis-node

yuumasato · 2023-08-10T08:49:25Z

/retest

The infra was running into buildah signature issues, maybe they are solved.

xiaojiey · 2023-08-10T09:07:59Z

/hold for test

xiaojiey · 2023-08-10T10:01:33Z

@rhmdnd The rule will pass when the encryption is aescbc or aesgcm. The only problem is: for both releases(4.14 and 4.12), the encryption type after auto remediation is aescbc. Not sure if it is expected result? Thanks.
====for 4.12,
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.12.0-0.nightly-2023-08-09-195826 True False 164m Cluster version is 4.12.0-0.nightly-2023-08-09-195826
$ oc get ccr| grep encryption
upstream-ocp4-cis-api-server-encryption-provider-cipher PASS medium
$ oc get apiserver/cluster -o=jsonpath={.spec.encryption.type}
aescbc
====for 4.14
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.14.0-0.nightly-2023-08-08-222204 True False 8h Cluster version is 4.14.0-0.nightly-2023-08-08-222204
$ oc get apiserver/cluster -o=jsonpath={.spec.encryption.type}
aescbc
$ oc get ccr| grep encryption
upstream-ocp4-cis-api-server-encryption-provider-cipher PASS medium

yuumasato · 2023-08-10T13:05:02Z

Given that remediation for 4.12 and 4.14 need to be different, I think we will have to create another rule.

rhmdnd · 2023-08-10T13:43:52Z

Given that remediation for 4.12 and 4.14 need to be different, I think we will have to create another rule.

Would it be possible to select a different remediation using a CPE?

yuumasato · 2023-08-10T16:04:16Z

Given that remediation for 4.12 and 4.14 need to be different, I think we will have to create another rule.

Would it be possible to select a different remediation using a CPE?

The XCCDF standard allows for different remediations based on CPE. But support for it is not implemented in CaC/content. Plus, I'm not sure how OpenSCAP would process them.

yuumasato · 2023-08-11T12:09:20Z

I'm checking out what would be required to support rules with remediations of the same type for different platforms.

rhmdnd · 2023-08-11T13:55:36Z

If it's too heavy, or cumbersome, we can create a separate rule.

yuumasato · 2023-08-11T14:59:06Z

If it's too heavy, or cumbersome, we can create a separate rule.

Yeah, I think we will have to go with a separate rule for now.

github-actions · 2023-08-11T16:45:12Z

This datastream diff is auto generated by the check Compare DS/Generate Diff

Click here to see the full diff

kubernetes remediation for rule 'xccdf_org.ssgproject.content_rule_api_server_encryption_provider_cipher' differs.
--- xccdf_org.ssgproject.content_rule_api_server_encryption_provider_cipher
+++ xccdf_org.ssgproject.content_rule_api_server_encryption_provider_cipher
@@ -5,4 +5,4 @@
   name: cluster
 spec:
   encryption:
-    type: aescbc
+    type: "{{.var_apiserver_encryption_type}}"

Vincent056 · 2023-08-11T17:06:46Z

/test e2e-aws-ocp4-cis
/test e2e-aws-ocp4-cis-node

xiaojiey · 2023-08-14T03:09:48Z

The scan for ocp4-cis will run into ERROR with code in this PR:

$ oc compliance bind -N test -S default-auto-apply profile/upstream-ocp4-cis
Creating ScanSettingBinding test
$ oc get suite 
NAME   PHASE   RESULT
test   DONE    ERROR
$ oc logs pod/compliance-operator-5f87d47dbf-pcn8c --all-containers| grep err
{"level":"error","ts":"2023-08-14T02:52:15.157Z","logger":"suitectrl","msg":"Could not update scan status","Request.Namespace":"openshift-compliance","Request.Name":"test","error":"Operation cannot be fulfilled on compliancesuites.compliance.openshift.io \"test\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancesuite.(*ReconcileComplianceSuite).reconcileScanStatus\n\tgithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancesuite/compliancesuite_controller.go:293\ngithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancesuite.(*ReconcileComplianceSuite).reconcileScans\n\tgithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancesuite/compliancesuite_controller.go:270\ngithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancesuite.(*ReconcileComplianceSuite).Reconcile\n\tgithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancesuite/compliancesuite_controller.go:180\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235"}
{"level":"error","ts":"2023-08-14T02:52:15.157Z","logger":"suitectrl","msg":"Retriable error","Request.Namespace":"openshift-compliance","Request.Name":"test","error":"Operation cannot be fulfilled on compliancesuites.compliance.openshift.io \"test\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/ComplianceAsCode/compliance-operator/pkg/controller/common.ReturnWithRetriableError\n\tgithub.com/ComplianceAsCode/compliance-operator/pkg/controller/common/errors.go:117\ngithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancesuite.(*ReconcileComplianceSuite).Reconcile\n\tgithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancesuite/compliancesuite_controller.go:182\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235"}
{"level":"error","ts":"2023-08-14T02:52:15.157Z","msg":"Reconciler error","controller":"compliancesuite-controller","object":{"name":"test","namespace":"openshift-compliance"},"namespace":"openshift-compliance","name":"test","reconcileID":"7ece2fcc-8480-45a6-a947-91873a588287","error":"Operation cannot be fulfilled on compliancesuites.compliance.openshift.io \"test\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235"}
$ oc get suite test -o=jsonpath={.status.scanStatuses[0].errormsg}
OpenSCAP Error: File '/content/ssg-ocp4-ds.xml' line 26974: Element '{http://oval.mitre.org/XMLSchema/oval-definitions-5#independent}yamlfilecontent_test', attribute 'check_existence': [facet 'enumeration'] The value 'any_exists' is not an element of the set {'all_exist', 'any_exist', 'at_least_one_exists', 'none_exist', 'only_one_exists'}.
 [/builddir/build/BUILD/openscap-1.3.7/src/XCCDF/xccdf_session.c:723]
File '/content/ssg-ocp4-ds.xml' line 26974: Element '{http://oval.mitre.org/XMLSchema/oval-definitions-5#independent}yamlfilecontent_test', attribute 'check_existence': 'any_exists' is not a valid value of the atomic type '{http://oval.mitre.org/XMLSchema/oval-common-5}ExistenceEnumeration'.
 [/builddir/build/BUILD/openscap-1.3.7/src/XCCDF/xccdf_session.c:723]
File '/content/ssg-ocp4-ds.xml' line 32005: Element '{http://oval.mitre.org/XMLSchema/oval-definitions-5}field', attribute 'operation': [facet 'enumeration'] The value 'equal' is not an element of the set {'equals', 'not equal', 'case insensitive equals', 'case insensitive not equal', 'greater than', 'less than', 'greater than or equal', 'less than or equal', 'bitwise and', 'bitwise or', 'pattern match', 'subset of', 'superset of'}.
 [/builddir/build/BUILD/openscap-1.3.7/src/XCCDF/xccdf_session.c:723]
File '/content/ssg-ocp4-ds.xml' line 32005: Element '{http://oval.mitre.org/XMLSchema/oval-definitions-5}field', attribute 'operation': 'equal' is not a valid value of the atomic type '{http://oval.mitre.org/XMLSchema/oval-common-5}OperationEnumeration'.
 [/builddir/build/BUILD/openscap-1.3.7/src/XCCDF/xccdf_session.c:723]
File '/content/ssg-ocp4-ds.xml' line 32006: Element '{http://oval.mitre.org/XMLSchema/oval-definitions-5}field', attribute 'operation': [facet 'enumeration'] The value 'equal' is not an element of the set {'equals', 'not equal', 'case insensitive equals', 'case insensitive not equal', 'greater than', 'less than', 'greater than or equal', 'less than or equal', 'bitwise and', 'bitwise or', 'pattern match', 'subset of', 'superset of'}.
 [/builddir/build/BUILD/openscap-1.3.7/src/XCCDF/xccdf_session.c:723]
File '/content/ssg-ocp4-ds.xml' line 32006: Element '{http://oval.mitre.org/XMLSchema/oval-definitions-5}field', attribute 'operation': 'equal' is not a valid value of the atomic type '{http://oval.mitre.org/XMLSchema/oval-common-5}OperationEnumeration'.
 [/builddir/build/BUILD/openscap-1.3.7/src/XCCDF/xccdf_session.c:723]
File '/content/ssg-ocp4-ds.xml' line 32006: Element '{http://oval.mitre.org/XMLSchema/oval-definitions-5}field': Duplicate key-sequence ['#'] in unique identity-constraint '{http://oval.mitre.org/XMLSchema/oval-definitions-5#independent}UniqueYamlFileValueFieldName'.
 [/builddir/build/BUILD/openscap-1.3.7/src/XCCDF/xccdf_session.c:723]
Invalid SCAP Source Datastream (1.3) content in /content/ssg-ocp4-ds.xml. [/builddir/build/BUILD/openscap-1.3.7/src/source/oscap_source.c:353]
Invalid SCAP Source Datastream (1.3) content in /content/ssg-ocp4-ds.xml [/builddir/build/BUILD/openscap-1.3.7/src/XCCDF/xccdf_session.c:839]

xiaojiey · 2023-08-14T03:09:56Z

/hold

applications/openshift/api-server/api_server_encryption_provider_cipher/rule.yml

yuumasato · 2023-08-14T14:07:59Z

/retest

applications/openshift/api-server/api_server_encryption_provider_cipher/rule.yml

The latest CIS guidance allows OpenShift users to configure the encryption type for the API server to `aescbc` or `aescgm`. Previously, we only evaluated `aescbc`, but this commit updates the check to also check that `aesgcm` is acceptable. It also updates the remediation to use the variable for the remediation, and clarifies its usage.

yuumasato · 2023-08-14T16:17:25Z

/test e2e-aws-ocp4-cis
/test e2e-aws-ocp4-cis-node

yuumasato

LGTM, just pending review from @xiaojiey

This is the yamlfilecontent_state that gets generated:

<ind:yamlfilecontent_state id="oval:ssg-state_api_server_encryption_provider_cipher:ste:1" version="1">
  <ind:value datatype="record" entity_check="all">
    <oval-def:field name="#" datatype="string" operation="pattern match">aescbc|aesgcm</oval-def:field>
  </ind:value>
</ind:yamlfilecontent_state>

yuumasato · 2023-08-14T16:24:53Z

Weird, CI didn't react. Issuing command again:

/test e2e-aws-ocp4-cis
/test e2e-aws-ocp4-cis-node

codeclimate · 2023-08-14T16:42:02Z

Code Climate has analyzed commit 0e54001 and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 100.0% (50% is the threshold).

This pull request will bring the total coverage in the repository to 53.3% (0.0% change).

View more on Code Climate.

rhmdnd · 2023-08-14T18:51:12Z

/test e2e-aws-ocp4-cis
/test e2e-aws-ocp4-cis-node

xiaojiey · 2023-08-15T02:30:32Z

Verified with 4.14.0-0.nightly-2023-08-11-055332 + code in the PR.
After remediation, the encryption type is still aescbc. Is it expected result? Thanks.

###########pass with etcd encryption type is aesgcm
$ oc get apiserver/cluster -o=jsonpath={.spec.encryption.type}
aesgcm
$ oc compliance bind -N test -S default-auto-apply profile/upstream-ocp4-cis
Creating ScanSettingBinding test
$ oc get suite -w
NAME   PHASE     RESULT
test   RUNNING   NOT-AVAILABLE
test   AGGREGATING   NOT-AVAILABLE
test   DONE          NON-COMPLIANT
test   DONE          NON-COMPLIANT
^C
$ oc get ccr upstream-ocp4-cis-api-server-encryption-provider-cipher
NAME                                                      STATUS   SEVERITY
upstream-ocp4-cis-api-server-encryption-provider-cipher   PASS     medium
##############When  etcd encryption type is NOT aesgcm/aescbc, aescbc will be applied as the remediation
$ oc compliance bind -N test -S default-auto-apply profile/upstream-ocp4-cisis
Creating ScanSettingBinding test
$ oc get suite -w
NAME   PHASE     RESULT
test   RUNNING   NOT-AVAILABLE
test   AGGREGATING   NOT-AVAILABLE
test   DONE          NON-COMPLIANT
test   DONE          NON-COMPLIANT
^C$ oc get cr
NAME                                                      STATE
upstream-ocp4-cis-api-server-encryption-provider-cipher   Applied
upstream-ocp4-cis-audit-profile-set                       Applied
$ oc get cr upstream-ocp4-cis-api-server-encryption-provider-cipher -o yaml
apiVersion: compliance.openshift.io/v1alpha1
kind: ComplianceRemediation
metadata:
  annotations:
    compliance.openshift.io/xccdf-value-used: var-apiserver-encryption-type
  creationTimestamp: "2023-08-15T02:17:30Z"
  generation: 2
  labels:
    compliance.openshift.io/scan-name: upstream-ocp4-cis
    compliance.openshift.io/suite: test
  name: upstream-ocp4-cis-api-server-encryption-provider-cipher
  namespace: openshift-compliance
  ownerReferences:
  - apiVersion: compliance.openshift.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ComplianceCheckResult
    name: upstream-ocp4-cis-api-server-encryption-provider-cipher
    uid: a81da50f-1556-4531-b94a-1cb6f2131d4b
  resourceVersion: "59304"
  uid: a817f619-bbae-4d8d-a3a3-4742041e1ace
spec:
  apply: true
  current:
    object:
      apiVersion: config.openshift.io/v1
      kind: APIServer
      metadata:
        name: cluster
      spec:
        encryption:
          type: aescbc
  outdated: {}
  type: Configuration
status:
  applicationState: Applied
$ oc compliance rerun-now scansettingbinding test
Rerunning scans from 'test': upstream-ocp4-cis
Re-running scan 'openshift-compliance/upstream-ocp4-cis'
$ oc get suite -w
NAME   PHASE     RESULT
test   RUNNING   NOT-AVAILABLE
test   AGGREGATING   NOT-AVAILABLE
test   DONE          NON-COMPLIANT
test   DONE          NON-COMPLIANT
^C
$ oc get ccr | grep encryption
upstream-ocp4-cis-api-server-encryption-provider-cipher                    PASS     medium
$ oc get apiserver/cluster -o=jsonpath={.spec.encryption.type}
aescbc

yuumasato · 2023-08-15T08:09:27Z

Verified with 4.14.0-0.nightly-2023-08-11-055332 + code in the PR. After remediation, the encryption type is still aescbc. Is it expected result? Thanks.

Yes.
CIS 1.2.31 allows for both, and we default to aescbc

yuumasato · 2023-08-15T08:11:36Z

/retest

xiaojiey · 2023-08-15T09:06:30Z

/unhold

rhmdnd commented Aug 9, 2023

View reviewed changes

applications/openshift/api-server/var_apiserver_encryption_type.var Outdated Show resolved Hide resolved

rhmdnd requested review from yuumasato, Vincent056 and xiaojiey August 9, 2023 19:39

rhmdnd added OpenShift OpenShift product related. CIS CIS Benchmark related. labels Aug 9, 2023

openshift-ci bot added the do-not-merge/hold Used by openshift-ci-robot bot. label Aug 10, 2023

rhmdnd force-pushed the OCPBUGS-11696 branch from 2df93dc to 258c722 Compare August 11, 2023 16:42

yuumasato requested changes Aug 14, 2023

View reviewed changes

rhmdnd force-pushed the OCPBUGS-11696 branch from 258c722 to 2d45593 Compare August 14, 2023 13:49

rhmdnd requested a review from yuumasato August 14, 2023 13:49

yuumasato reviewed Aug 14, 2023

View reviewed changes

applications/openshift/api-server/api_server_encryption_provider_cipher/rule.yml Show resolved Hide resolved

rhmdnd force-pushed the OCPBUGS-11696 branch from 2d45593 to 0e54001 Compare August 14, 2023 16:12

rhmdnd requested a review from yuumasato August 14, 2023 16:13

yuumasato approved these changes Aug 14, 2023

View reviewed changes

yuumasato self-assigned this Aug 15, 2023

yuumasato added this to the 0.1.70 milestone Aug 15, 2023

openshift-ci bot removed the do-not-merge/hold Used by openshift-ci-robot bot. label Aug 15, 2023

yuumasato approved these changes Aug 15, 2023

View reviewed changes

yuumasato merged commit 62f19fa into ComplianceAsCode:master Aug 15, 2023
33 of 37 checks passed

Mab879 added the Kubernetes Kubernetes remediation update. label Oct 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OCPBUGS-11696: Update encryption type to support 4.13 deployments #10974

OCPBUGS-11696: Update encryption type to support 4.13 deployments #10974

rhmdnd commented Aug 9, 2023

github-actions bot commented Aug 9, 2023 •

edited

rhmdnd commented Aug 9, 2023

openshift-ci bot commented Aug 9, 2023

rhmdnd commented Aug 9, 2023

yuumasato commented Aug 10, 2023 •

edited

xiaojiey commented Aug 10, 2023

xiaojiey commented Aug 10, 2023

yuumasato commented Aug 10, 2023

rhmdnd commented Aug 10, 2023

yuumasato commented Aug 10, 2023

yuumasato commented Aug 11, 2023

rhmdnd commented Aug 11, 2023

yuumasato commented Aug 11, 2023

github-actions bot commented Aug 11, 2023

Vincent056 commented Aug 11, 2023

xiaojiey commented Aug 14, 2023 •

edited

xiaojiey commented Aug 14, 2023

yuumasato commented Aug 14, 2023

yuumasato commented Aug 14, 2023

yuumasato left a comment

yuumasato commented Aug 14, 2023

codeclimate bot commented Aug 14, 2023

rhmdnd commented Aug 14, 2023

xiaojiey commented Aug 15, 2023 •

edited

yuumasato commented Aug 15, 2023

yuumasato commented Aug 15, 2023

xiaojiey commented Aug 15, 2023

OCPBUGS-11696: Update encryption type to support 4.13 deployments #10974

OCPBUGS-11696: Update encryption type to support 4.13 deployments #10974

Conversation

rhmdnd commented Aug 9, 2023

github-actions bot commented Aug 9, 2023 • edited

rhmdnd commented Aug 9, 2023

openshift-ci bot commented Aug 9, 2023

rhmdnd commented Aug 9, 2023

yuumasato commented Aug 10, 2023 • edited

xiaojiey commented Aug 10, 2023

xiaojiey commented Aug 10, 2023

yuumasato commented Aug 10, 2023

rhmdnd commented Aug 10, 2023

yuumasato commented Aug 10, 2023

yuumasato commented Aug 11, 2023

rhmdnd commented Aug 11, 2023

yuumasato commented Aug 11, 2023

github-actions bot commented Aug 11, 2023

Vincent056 commented Aug 11, 2023

xiaojiey commented Aug 14, 2023 • edited

xiaojiey commented Aug 14, 2023

yuumasato commented Aug 14, 2023

yuumasato commented Aug 14, 2023

yuumasato left a comment

Choose a reason for hiding this comment

yuumasato commented Aug 14, 2023

codeclimate bot commented Aug 14, 2023

rhmdnd commented Aug 14, 2023

xiaojiey commented Aug 15, 2023 • edited

yuumasato commented Aug 15, 2023

yuumasato commented Aug 15, 2023

xiaojiey commented Aug 15, 2023

github-actions bot commented Aug 9, 2023 •

edited

yuumasato commented Aug 10, 2023 •

edited

xiaojiey commented Aug 14, 2023 •

edited

xiaojiey commented Aug 15, 2023 •

edited