Bug 1963730: writes the dynamic certs atomically #1103

p0lyn0mial · 2021-06-09T10:47:50Z

As of today, the dynamic certificates i.e. kube-apiserver-certs are accessed by at least two processes, namely the installer pod and the cert-syncer.
Up until now, there was no coordination between these processes that might have lead to many unexpected errors, like https://bugzilla.redhat.com/show_bug.cgi?id=1963730

This PR writes all certificates in an atomic way by first creating a temporary file, writing the content to it, and then renaming it to the original file.

os.Rename calls syscall.Rename which in turn uses the rename syscall (Linux) which provides atomicity (https://man7.org/linux/man-pages/man2/rename.2.html)

The previous attempt didn't work as it would break the DR scripts - #1098

openshift-ci · 2021-06-09T10:47:55Z

@p0lyn0mial: This pull request references Bugzilla bug 1963730, which is invalid:

expected the bug to target the "4.8.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1963730: writes the dynamic certs atomically

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

p0lyn0mial · 2021-06-09T10:49:52Z

/assign @sttts

openshift-ci · 2021-06-09T10:51:54Z

@p0lyn0mial: This pull request references Bugzilla bug 1963730, which is invalid:

expected the bug to target the "4.8.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1963730: writes the dynamic certs atomically

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sttts · 2021-06-09T12:14:38Z

pkg/operator/staticpod/file_utils.go

+}
+
+func writeTemporaryFile(content []byte, filePerms os.FileMode, contentDir, filename string) (string, error) {
+	klog.Infof("Creating a temporary file for %q ...", path.Join(contentDir, filename))


do we really need all this logging? Would streamline the func WriteFileAtomic to not output anything.

sttts · 2021-06-09T12:43:57Z

pkg/operator/staticpod/installerpod/cmd.go

@@ -408,3 +398,29 @@ func (o *InstallOptions) Run(ctx context.Context) error {
 	recorder.Eventf("StaticPodInstallerCompleted", "Successfully installed revision %s", o.Revision)
 	return nil
 }
+
+func writeConfig(content []byte, atomic bool, contentDir, filename string) error {


why do we need atomic bool?

because not all files need to be written in that way. The other files are written to a per revision directory, like

library-go/pkg/operator/staticpod/installerpod/cmd.go

Line 274 in 3af072d

if err := o.copySecretsAndConfigMaps(ctx, resourceDir, secretPrefixes, optionalSecretPrefixes, configPrefixes, optionalConfigPrefixes, true, false); err != nil {

we could use prefixed arg but that would be a little less explicit.

p0lyn0mial · 2021-06-09T14:32:59Z

/hold

until proof PRs are green

sttts · 2021-06-10T14:18:35Z

/lgtm
/approve

openshift-ci · 2021-06-10T14:21:47Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: p0lyn0mial, sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/operator/OWNERS~~ [sttts]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

p0lyn0mial · 2021-06-11T07:35:38Z

/hold cancel

p0lyn0mial · 2021-06-11T07:37:12Z

/bugzilla refresh

openshift-ci · 2021-06-11T07:37:19Z

@p0lyn0mial: This pull request references Bugzilla bug 1963730, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.8.0) matches configured target release for branch (4.8.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @wangke19

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2021-06-11T07:37:45Z

@p0lyn0mial: Some pull requests linked via external trackers have merged:

The following pull requests linked via external trackers have not merged:

openshift/cluster-etcd-operator#609 is open

These pull request must merge or be unlinked from the Bugzilla bug in order for it to move to the next state. Once unlinked, request a bug refresh with /bugzilla refresh.

Bugzilla bug 1963730 has not been moved to the MODIFIED state.

In response to this:

Bug 1963730: writes the dynamic certs atomically

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot added the bugzilla/severity-low Referenced Bugzilla bug's severity is low for the branch this PR is targeting. label Jun 9, 2021

openshift-ci bot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Jun 9, 2021

openshift-ci bot requested review from deads2k and sttts June 9, 2021 10:48

p0lyn0mial force-pushed the wire-dynamic-certs-atomicaly branch from 8e3e42c to 6fe78a9 Compare June 9, 2021 10:48

openshift-ci bot assigned sttts Jun 9, 2021

p0lyn0mial force-pushed the wire-dynamic-certs-atomicaly branch from 6fe78a9 to b84bc2e Compare June 9, 2021 11:48

sttts reviewed Jun 9, 2021

View reviewed changes

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 9, 2021

p0lyn0mial added 5 commits June 9, 2021 16:40

installerpod: write dynamic certificates in a atmomic way

ed37037

certsyncpod: write dynamic certificates in a atmomic way

fa48166

prune dangling temp files

cc4f180

log less

2abd13a

simplify

43d70e5

p0lyn0mial force-pushed the wire-dynamic-certs-atomicaly branch from 3af072d to 43d70e5 Compare June 10, 2021 14:06

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 10, 2021

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 10, 2021

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 11, 2021

openshift-ci bot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Jun 11, 2021

openshift-ci bot removed the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Jun 11, 2021

openshift-ci bot requested a review from wangke19 June 11, 2021 07:37

openshift-merge-robot merged commit ad411b3 into openshift:master Jun 11, 2021

openshift-ci bot mentioned this pull request Jul 23, 2021

Bug 1963730: kube-apiserver failed to load SNI cert and key openshift/cluster-kube-apiserver-operator#1153

Merged

Bug 1963730: writes the dynamic certs atomically #1103

Bug 1963730: writes the dynamic certs atomically #1103

p0lyn0mial commented Jun 9, 2021 •

edited

openshift-ci bot commented Jun 9, 2021

p0lyn0mial commented Jun 9, 2021

openshift-ci bot commented Jun 9, 2021

sttts Jun 9, 2021

sttts Jun 9, 2021

p0lyn0mial Jun 9, 2021

p0lyn0mial commented Jun 9, 2021

sttts commented Jun 10, 2021

openshift-ci bot commented Jun 10, 2021

p0lyn0mial commented Jun 11, 2021

p0lyn0mial commented Jun 11, 2021

openshift-ci bot commented Jun 11, 2021

openshift-ci bot commented Jun 11, 2021

Bug 1963730: writes the dynamic certs atomically #1103

Bug 1963730: writes the dynamic certs atomically #1103

Conversation

p0lyn0mial commented Jun 9, 2021 • edited

openshift-ci bot commented Jun 9, 2021

p0lyn0mial commented Jun 9, 2021

openshift-ci bot commented Jun 9, 2021

sttts Jun 9, 2021

Choose a reason for hiding this comment

sttts Jun 9, 2021

Choose a reason for hiding this comment

p0lyn0mial Jun 9, 2021

Choose a reason for hiding this comment

p0lyn0mial commented Jun 9, 2021

sttts commented Jun 10, 2021

openshift-ci bot commented Jun 10, 2021

p0lyn0mial commented Jun 11, 2021

p0lyn0mial commented Jun 11, 2021

openshift-ci bot commented Jun 11, 2021

openshift-ci bot commented Jun 11, 2021

p0lyn0mial commented Jun 9, 2021 •

edited