daemon/update.go: fix FIPS check and remove day 2 support #1252

yuqi-zhang · 2019-11-07T19:18:23Z

We no longer carry coreos-fips script in RHCOS, so that check fill
fail regardless. As a temporary fix, check state of node via
cat /proc/sys/crypto/fips_enabled instead.

Fixes: #1250

pkg/daemon/update.go

nee1esh · 2019-11-07T21:12:49Z

/retest

ashcrow

👍

ashcrow · 2019-11-07T21:23:17Z

/override ci/prow/e2e-aws-scaleup-rhel7

openshift-ci-robot · 2019-11-07T21:23:39Z

@ashcrow: Overrode contexts on behalf of ashcrow: ci/prow/e2e-aws-scaleup-rhel7

In response to this:

/override ci/prow/e2e-aws-scaleup-rhel7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ashcrow · 2019-11-07T21:24:41Z

verify failure:

pkg/daemon/update.go:331: File is not `goimports`-ed (goimports)
		return errors.Wrapf(err, "Error parsing FIPS file at %s", fipsFile,)
pkg/daemon/update.go:339:21: error strings should not be capitalized or end with punctuation or a newline (golint)
		return errors.New("Refusing to update FIPS mode. This is not a supported operation.")
		                  ^

jlebon · 2019-11-07T21:23:28Z

pkg/daemon/update.go

-		return errors.Wrapf(err, "%s FIPS: %s", arg, string(out))
+	nodeFIPS, err := strconv.ParseBool(string(content))
+	if err != nil {
+		return errors.Wrapf(err, "Error parsing FIPS file at %s", fipsFile,)


Extra comma here?

Whoops, fixing

jlebon · 2019-11-07T21:25:00Z

pkg/daemon/update.go

+		return nil
+	}
+
+	if current.Spec.FIPS != desired.Spec.FIPS {


Hmm, should we move this check higher up?

Hm so we ran into this error initially because we detected that the current current.Spec.FIPS != desired.Spec.FIPS right? Because it was nil?

As in, maybe I should change this just to return an error if !desired.Spec.FIPS == nodeFIPS

And flat out ignore what current.Spec.FIPS is

but then the Spec would not be reflective of the state of the system, perhaps I should update that as well just above?

I changed it to update current.Spec.FIPS so we don't get stuck in a reboot loop. This should definitely be improved of course but I think this can get us past the errors (not tested yet)

ericavonb · 2019-11-07T21:28:44Z

pkg/daemon/update.go

-	// fipsCommand is the command to use when enabling or disabling FIPS
-	fipsCommand = "/usr/libexec/rhcos-tools/coreos-fips"
+	// fipsFile is the file to check if FIPS is enabled
+	fipsFile = "/proc/sys/crypto/fips_enabled"


@cgwalters didn't approve of this method (should use the coreos-fips --status or whatever command): https://github.com/openshift/machine-config-operator/pull/1013/files#r307428010

Yeah we removed that binary unfortunately, which is what lead to this

fipscheck

See #1250 (comment). That said, checking for /proc/sys/crypto/fips_enabled is a good start.

We don't want to use coreos-fips anymore. RHCOS has embedded support for turning on FIPS now.

I think using the proc path above is fine. fipscheck is an alternate possibility. coreos-fips has been removed.

yuqi-zhang · 2019-11-07T21:53:51Z

/retest

Post http://boskos.ci/acquire?dest=leased&owner=ci-op-3smxij8t-c4a31&request_id=4447156788061523237&state=free&type=aws-quota-slice: dial tcp 172.30.131.17:80: connect: no route to host

jlebon · 2019-11-07T21:54:39Z

/approve

This looks sane to me!

yuqi-zhang · 2019-11-07T22:05:01Z

/retest

yuqi-zhang · 2019-11-07T22:09:54Z

Minor update: Instead of just returning nil, if FIPS changes were detected, do:
return errors.New("Detected change to FIPS flag. Refusing to modify FIPS on a running cluster.")
instead

cgwalters · 2019-11-08T00:16:38Z

/approve

sinnykumari · 2019-11-08T09:20:59Z

/retest

ashcrow · 2019-11-08T13:56:42Z

/override ci/prow/e2e-aws-scaleup-rhel7

openshift-ci-robot · 2019-11-08T13:56:57Z

@ashcrow: Overrode contexts on behalf of ashcrow: ci/prow/e2e-aws-scaleup-rhel7

In response to this:

/override ci/prow/e2e-aws-scaleup-rhel7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ashcrow · 2019-11-08T13:57:10Z

/retest

yuqi-zhang · 2019-11-08T15:52:21Z

/hold

Found an error while testing, investigating

We no longer carry coreos-fips script in RHCOS, so we can no longer update FIPS mode via Machineconfig. For now, let us instead check whether FIPS is expected via /proc/sys/crypto/fips_enabled. Signed-off-by: Yu Qi Zhang <jerzhang@redhat.com>

yuqi-zhang · 2019-11-08T16:15:56Z

/hold cancel

Can confirm this should work in existing clusters, as well as properly catch and reject day 2 fips applications as we expect. Have not tested with day 1 pending RHCOS updates

ashcrow · 2019-11-08T17:11:09Z

Have not tested with day 1 pending RHCOS updates

Understood. I don't think an issue will occur but if one does the fix can be done in a different PR.

/lgtm

openshift-ci-robot · 2019-11-08T17:11:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ashcrow, cgwalters, jlebon, yuqi-zhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ashcrow,cgwalters]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2019-11-08T17:54:51Z

@yuqi-zhang: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws-scaleup-rhel7	`9075a83`	link	`/test e2e-aws-scaleup-rhel7`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

ashcrow · 2019-11-08T18:16:44Z

/override ci/prow/e2e-aws-scaleup-rhel7

openshift-ci-robot · 2019-11-08T18:17:03Z

@ashcrow: Overrode contexts on behalf of ashcrow: ci/prow/e2e-aws-scaleup-rhel7

In response to this:

/override ci/prow/e2e-aws-scaleup-rhel7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

cgwalters · 2019-11-08T18:18:12Z

@ashcrow FWIW that context is non-blocking, you don't need to override it.

This effectively reverts fb1c4b4 e2e-fips is currently failing with `/bin/bash: line 15: nodes[i]: unbound variable` Looking at this...we already have code to validate the state of FIPS in the MCO, see: https://github.com/openshift/machine-config-operator/blob/091afde36ac117ef8b782a85b38ae8783ddf4b70/pkg/daemon/update.go#L571 openshift/machine-config-operator#1252 openshift/machine-config-operator#1233 I think these types of checks should be the MCO's role, or if we choose not to do that, let's at least implement them in Go in the existing e2e suite and avoid nontrivial shell-in-YAML.

openshift-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 7, 2019

openshift-ci-robot requested review from cgwalters and ericavonb November 7, 2019 19:19

yuqi-zhang force-pushed the fips-fixes branch from a574351 to 3ff1125 Compare November 7, 2019 19:24

cgwalters approved these changes Nov 7, 2019

View reviewed changes

pkg/daemon/update.go Outdated Show resolved Hide resolved

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 7, 2019

yuqi-zhang force-pushed the fips-fixes branch from 3ff1125 to fa3c675 Compare November 7, 2019 21:19

yuqi-zhang changed the title ~~WIP: monkey patch MCD fips~~ daemon/update.go: fix FIPS check and remove day 2 support Nov 7, 2019

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 7, 2019

ashcrow approved these changes Nov 7, 2019

View reviewed changes

jlebon reviewed Nov 7, 2019

View reviewed changes

ericavonb reviewed Nov 7, 2019

View reviewed changes

yuqi-zhang force-pushed the fips-fixes branch from fa3c675 to 1b679bb Compare November 7, 2019 21:33

yuqi-zhang force-pushed the fips-fixes branch from 1b679bb to 0be6403 Compare November 7, 2019 22:08

yuqi-zhang force-pushed the fips-fixes branch from 0be6403 to 542f116 Compare November 7, 2019 22:24

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 8, 2019

yuqi-zhang force-pushed the fips-fixes branch from 542f116 to 9075a83 Compare November 8, 2019 16:15

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 8, 2019

openshift-ci-robot assigned ashcrow Nov 8, 2019

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 8, 2019

openshift-merge-robot merged commit 0cffa48 into openshift:master Nov 8, 2019

cgwalters mentioned this pull request Jul 28, 2020

Remove FIPS check shell code, already done in the MCO openshift/release#10488

Closed

daemon/update.go: fix FIPS check and remove day 2 support #1252

daemon/update.go: fix FIPS check and remove day 2 support #1252

Conversation

yuqi-zhang commented Nov 7, 2019

nee1esh commented Nov 7, 2019

ashcrow left a comment

Choose a reason for hiding this comment

ashcrow commented Nov 7, 2019

openshift-ci-robot commented Nov 7, 2019

ashcrow commented Nov 7, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ashcrow Nov 7, 2019 • edited Loading

Choose a reason for hiding this comment

yuqi-zhang commented Nov 7, 2019

jlebon commented Nov 7, 2019

yuqi-zhang commented Nov 7, 2019

yuqi-zhang commented Nov 7, 2019

cgwalters commented Nov 8, 2019

sinnykumari commented Nov 8, 2019

ashcrow commented Nov 8, 2019

openshift-ci-robot commented Nov 8, 2019

ashcrow commented Nov 8, 2019

yuqi-zhang commented Nov 8, 2019

yuqi-zhang commented Nov 8, 2019

ashcrow commented Nov 8, 2019

openshift-ci-robot commented Nov 8, 2019

openshift-ci-robot commented Nov 8, 2019 • edited Loading

ashcrow commented Nov 8, 2019

openshift-ci-robot commented Nov 8, 2019

cgwalters commented Nov 8, 2019

ashcrow Nov 7, 2019 •

edited

Loading

openshift-ci-robot commented Nov 8, 2019 •

edited

Loading