New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Pass ssh keys from clusterconfig to machineconfig #164

Open
wants to merge 15 commits into
base: master
from

Conversation

@kikisdeliveryservice
Member

kikisdeliveryservice commented Nov 10, 2018

Rough draft that closes openshift/installer#578

Summary of issue: the ClusterConfig has the SSH key and while that key is being passed into the MCO, the MCO isn't properly adding it to the MachineConfig. This prevents us from being able add functionality to MCD updating existing SSH keys in a MachineConfig's Spec.Config.Passwd.Users.

Still working on this, but wanted to make a PR with what I have so far. Comments & feedback welcome!
cc: @abhinavdahiya @wking

@openshift-ci-robot

This comment has been minimized.

openshift-ci-robot commented Nov 10, 2018

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kikisdeliveryservice
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: abhinavdahiya

If they are not already assigned, you can assign the PR to them by writing /assign @abhinavdahiya in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kikisdeliveryservice

This comment has been minimized.

Member

kikisdeliveryservice commented Nov 10, 2018

/assign @abhinavdahiya

@kikisdeliveryservice kikisdeliveryservice force-pushed the kikisdeliveryservice:pass-mco-keys branch from 265be24 to a59a92b Nov 15, 2018

@kikisdeliveryservice

This comment has been minimized.

Member

kikisdeliveryservice commented Nov 16, 2018

The best that I can tell is that the sshkeys need to move this way to make it to the MachineConfig: installerConfig(Admin.SSHKey)-> MCOConfig --> RenderConfig/ControllerConfig --> MachineConfig (Passwd.Users)

Is this flow correct? I've traced through quite a few times, but in need of expertise on this. Can someone PTAL and let me know if my logic is correct?
cc: @ashcrow @abhinavdahiya @wking

@abhinavdahiya

This comment has been minimized.

Member

abhinavdahiya commented Nov 16, 2018

The best that I can tell is that the sshkeys need to move this way to make it to the MachineConfig: installerConfig(Admin.SSHKey)-> MCOConfig --> RenderConfig/ControllerConfig --> MachineConfig (Passwd.Users)

Yes, that looks like state of the art right now.

@jlebon

This comment has been minimized.

Member

jlebon commented Nov 20, 2018

Looks like this needs a rebase.

@kikisdeliveryservice kikisdeliveryservice force-pushed the kikisdeliveryservice:pass-mco-keys branch 2 times, most recently from c92b354 to 2a729e4 Nov 20, 2018

@kikisdeliveryservice

This comment has been minimized.

Member

kikisdeliveryservice commented Nov 21, 2018

Rebased! @jlebon :)

@jlebon

Looks sane at a high level (lots of debug print statements left over I see :)). Just one comment.

@@ -129,6 +131,9 @@ type ControllerConfigSpec struct {
// PullSecret is the default pull secret that needs to be installed
// on all machines.
PullSecret *corev1.ObjectReference `json:"pullSecret,omitempty"`
// Public SSH
SSHKey string `json:"sshKey"`

This comment has been minimized.

@jlebon

jlebon Nov 21, 2018

Member

Only a single key? Would it make sense to support multiple keys here even if the installer right now only lets one specify one only?

This comment has been minimized.

@ashcrow

ashcrow Nov 26, 2018

Member

If an array of strings doesn't increase the complexity I say go ahead. Otherwise, it's fine to hold off until the feature set is expected/available at the higher level.

@kikisdeliveryservice kikisdeliveryservice force-pushed the kikisdeliveryservice:pass-mco-keys branch from 2a729e4 to 6acf9c0 Nov 29, 2018

@openshift-ci-robot openshift-ci-robot added size/L and removed size/M labels Nov 29, 2018

@openshift-bot

This comment has been minimized.

openshift-bot commented Nov 29, 2018

@kikisdeliveryservice: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@kikisdeliveryservice kikisdeliveryservice force-pushed the kikisdeliveryservice:pass-mco-keys branch from 6acf9c0 to a82e94c Dec 3, 2018

@kikisdeliveryservice kikisdeliveryservice force-pushed the kikisdeliveryservice:pass-mco-keys branch from a82e94c to af8a56b Dec 4, 2018

@kikisdeliveryservice

This comment has been minimized.

Member

kikisdeliveryservice commented Dec 4, 2018

/retest

@kikisdeliveryservice

This comment has been minimized.

Member

kikisdeliveryservice commented Dec 4, 2018

Update:
Confirming that the SSH Key made it all the way from installerConfig(Admin.SSHKey)-> MCOConfig --> RenderConfig/ControllerConfig --> MachineConfig (Passwd.Users).

Now checking the machineconfigs for both master and worker shows (example)

$ oc get machineconfig 00-worker  -o yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  creationTimestamp: 2018-12-04T01:07:56Z
  generation: 1
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 00-worker
  ownerReferences:
  - apiVersion: machineconfiguration.openshift.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: ControllerConfig
    name: machine-config-controller
    uid: fae0efad-f760-11e8-b3cc-42f92e967bf5
  resourceVersion: "23232"
  selfLink: /apis/machineconfiguration.openshift.io/v1/machineconfigs/00-worker
  uid: 088750c9-f761-11e8-b3cc-42f92e967bf5
spec:
  config:
    ignition:
      config: {}
      security:
        tls: {}
      timeouts: {}
      version: 2.2.0
    networkd: {}
    passwd:
      users:
      - name: core
        sshAuthorizedKeys:
        - |
          ssh-rsa ABC123....
@kikisdeliveryservice

This comment has been minimized.

Member

kikisdeliveryservice commented Dec 4, 2018

However, I am seeing the daemon degrade and I'm not sure if this is this a product of when and how I'm applying my binaries and their timing or if I also need to allow reconcilable to account for this initialization of the configs... I'm going to try to test this with better timing tomorrow (assuming I can get a cluster up again) as I think the mcd got a headstart on the mco/mcc when I applied my binaries. I also put a sketch of what I might add to reconcilable if necessary, but it's untested.

This is the last thing needed to wrap up the PR.

$ oc logs -f -n openshift-machine-config-operator machine-config-daemon-289tf
I1204 01:30:42.968291    5400 start.go:51] Version: 3.11.0-301-g13e15273
I1204 01:30:42.968962    5400 start.go:88] starting node writer
I1204 01:30:42.974021    5400 run.go:22] Running captured: chroot /rootfs rpm-ostree status --json
I1204 01:30:43.054368    5400 daemon.go:120] Booted osImageURL: registry.svc.ci.openshift.org/rhcos/maipo@sha256:e4b05527d762ba159c821d53bc8b2478c38b717b0fcfb4b76c57787c2f46ee2c (47.177)
I1204 01:30:43.072829    5400 start.go:139] Calling chroot("/rootfs")
I1204 01:30:43.072944    5400 daemon.go:294] CheckStateOnBoot
I1204 01:30:43.085725    5400 update.go:87] Checking if configs are reconcilable!
I1204 01:30:43.085837    5400 update.go:100] Old Config: {[] []}
I1204 01:30:43.085911    5400 update.go:101] New Config: {[] []}
I1204 01:30:43.085947    5400 daemon.go:565] No target osImageURL provided
I1204 01:30:43.095044    5400 start.go:158] Starting MachineConfigDaemon
I1204 01:30:43.095059    5400 daemon.go:195] Enabling Kubelet Healthz Monitor
I1204 01:30:52.359987    5400 daemon.go:371] handleNodeUpdate
I1204 01:30:52.629908    5400 daemon.go:371] handleNodeUpdate
I1204 01:33:09.537006    5400 update.go:33] Updating node with new config
I1204 01:33:09.538536    5400 update.go:87] Checking if configs are reconcilable!
I1204 01:33:09.538767    5400 update.go:100] Old Config: {[] []}
I1204 01:33:09.538859    5400 update.go:101] New Config: {[] [{<nil>  []  core false false false <nil>  [ssh-rsa ABC123]  false <nil>}]}
W1204 01:33:09.538950    5400 update.go:133] daemon can't reconcile state!
W1204 01:33:09.538990    5400 update.go:134] Ignition passwd section contains changes
I1204 01:33:09.539039    5400 daemon.go:385] Unable to apply update: daemon can't reconcile this config
I1204 01:33:09.539076    5400 daemon.go:371] handleNodeUpdate
E1204 01:33:09.539286    5400 writer.go:85] Marking degraded due to: daemon can't reconcile this config
F1204 01:33:09.548695    5400 start.go:163] failed to run: daemon can't reconcile this config

@kikisdeliveryservice kikisdeliveryservice force-pushed the kikisdeliveryservice:pass-mco-keys branch from af8a56b to 88395d6 Dec 4, 2018

@kikisdeliveryservice

This comment has been minimized.

Member

kikisdeliveryservice commented Dec 4, 2018

cc @ashcrow @abhinavdahiya (some updates and questions above)

@kikisdeliveryservice kikisdeliveryservice force-pushed the kikisdeliveryservice:pass-mco-keys branch 3 times, most recently from dc85455 to 4a18975 Dec 4, 2018

@ashcrow

This comment has been minimized.

Member

ashcrow commented Dec 6, 2018

/test e2e-aws

2 similar comments
@ashcrow

This comment has been minimized.

Member

ashcrow commented Dec 6, 2018

/test e2e-aws

@ashcrow

This comment has been minimized.

Member

ashcrow commented Dec 6, 2018

/test e2e-aws

@kikisdeliveryservice

This comment has been minimized.

Member

kikisdeliveryservice commented Dec 6, 2018

Getting blocked on this by: openshift/origin#21612
Can't get a running cluster up.

@kikisdeliveryservice

This comment has been minimized.

Member

kikisdeliveryservice commented Dec 7, 2018

Got a cluster up and hit a change to the install config from the installer: openshift/installer#771

Update: opened #219

@kikisdeliveryservice kikisdeliveryservice force-pushed the kikisdeliveryservice:pass-mco-keys branch from 4c892a3 to 994c867 Dec 10, 2018

@ashcrow

This comment has been minimized.

Member

ashcrow commented Dec 11, 2018

Rebase required.

@kikisdeliveryservice kikisdeliveryservice force-pushed the kikisdeliveryservice:pass-mco-keys branch from 994c867 to 09e9b85 Dec 15, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment