util: allow configuring VAULT_AUTH_MOUNT_PATH for Vault Tenant SA KMS #2322

nixpanic · 2021-07-26T09:50:35Z

The VAULT_AUTH_MOUNT_PATH is a Vault configuration parameter that allows
a user to set a non default path for the Kubernetes ServiceAccount
integration. This can already be configured for the Vault KMS, and is
now added to the Vault Tenant SA KMS as well.

Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

/retest ci/centos/<job-name>: retest the <job-name> after unrelated
failure (please report the failure too!)
/retest all: run this in case the CentOS CI failed to start/report any test
progress or results

internal/util/vault_sa.go

Madhu-1 · 2021-07-26T11:19:46Z

code = InvalidArgument desc = invalid encryption kms configuration: failed creating new Vault Secrets: Error making API request.
URL: PUT http://vault.cephcsi-e2e-2fdab8bd.svc.cluster.local:8200/v1/auth/v1/auth/kubernetes/login/login
Code: 400. Errors:

missing client token

@nixpanic CI is failing PTAL.

Madhu-1 · 2021-07-26T11:21:33Z

http://vault.cephcsi-e2e-2fdab8bd.svc.cluster.local:8200/v1/auth/v1/auth/kubernetes/login/login

URL seems problamatic.

Pull request has been modified.

nixpanic · 2021-07-26T15:31:13Z

/retest ci/centos/mini-e2e-helm/k8s-1.19

nixpanic · 2021-07-26T15:31:24Z

/retest ci/centos/mini-e2e/k8s-1.20

nixpanic · 2021-07-26T15:32:52Z

/retest ci/centos/mini-e2e-helm/k8s-1.19

New etcdserver timeout related error (logs):

Jul 26 14:17:53.181: INFO: waiting for PVC rbd-80810 (112 seconds elapsed)
Jul 26 14:18:00.773: INFO: Error getting pvc "rbd-80810" in namespace "rbd-8081": etcdserver: request timed out
Jul 26 14:18:01.182: INFO: waiting for PVC rbd-80810 (120 seconds elapsed)
Jul 26 14:18:01.711: INFO: Error getting pvc "rbd-80810" in namespace "rbd-8081": rpc error: code = Unavailable desc = transport is closing
Jul 26 14:18:01.711: INFO: failed to create PVC (rbd-80810): failed to get pvc: rpc error: code = Unavailable desc = transport is closing
Jul 26 14:18:01.711: FAIL: creating PVCs failed, 1 errors were logged

nixpanic · 2021-07-26T15:34:09Z

/retest ci/centos/mini-e2e/k8s-1.20

Failed to deploy CephFS RBAC (logs):

Jul 26 13:54:42.766: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.187:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-ea434921 create --namespace=cephcsi-e2e-ea434921 -f -'
Jul 26 13:55:04.448: INFO: rc: 1
Jul 26 13:55:04.448: FAIL: failed to create CephFS provisioner rbac with error error running /usr/bin/kubectl --server=https://192.168.39.187:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-ea434921 create --namespace=cephcsi-e2e-ea434921 -f -:
Command stdout:
role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created
rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created

stderr:
Error from server: error when creating "STDIN": etcdserver: request timed out
Error from server: error when creating "STDIN": etcdserver: request timed out
Error from server: error when creating "STDIN": etcdserver: request timed out

error:
exit status 1

nixpanic · 2021-08-03T09:49:00Z

/retest ci/centos/mini-e2e

nixpanic · 2021-08-03T09:49:20Z

/retest ci/centos/mini-e2e-helm

nixpanic · 2021-08-03T13:08:53Z

@Mergifyio rebase

Required for change in CentOS image used by e2e #2349

mergify · 2021-08-03T13:10:04Z

Command rebase: success

Branch has been successfully rebased

nixpanic · 2021-08-03T14:49:26Z

/retest ci/centos/mini-e2e-helm/k8s-1.21

nixpanic · 2021-08-03T14:50:46Z

/retest ci/centos/mini-e2e-helm/k8s-1.21

Failed due to etcdserver timeout (logs):

Aug  3 15:47:04.927: INFO: waiting for PVC raw-block-pvc (0 seconds elapsed)
Aug  3 15:47:11.936: INFO: Error getting pvc in namespace: '': etcdserver: request timed out
Aug  3 15:47:13.937: INFO: waiting for PVC  (9 seconds elapsed)
Aug  3 15:47:13.937: INFO: Error getting pvc in namespace: '': an empty namespace may not be set when a resource name is provided
Aug  3 15:47:13.937: FAIL: validating thick-provisioning failed: failed to expand PVC: failed to get pvc: an empty namespace may not be set when a resource name is provided

nixpanic · 2021-08-03T18:59:04Z

/retest ci/centos/mini-e2e-helm/k8s-1.19

nixpanic · 2021-08-04T06:06:43Z

@Mergifyio rebase

mergify · 2021-08-04T06:07:31Z

Command rebase: success

Branch has been successfully rebased

nixpanic · 2021-08-04T09:17:21Z

This should not have been closed 😞

nixpanic · 2021-08-04T10:17:34Z

/retest ci/centos/mini-e2e/k8s-1.21

nixpanic · 2021-08-04T10:19:41Z

/retest ci/centos/mini-e2e/k8s-1.21

e2e failed to deploy rbd component (logs):

Aug  4 10:46:40.250: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.12:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-212c5ecc create -f -'
Aug  4 10:46:47.691: INFO: rc: 1
Aug  4 10:46:49.691: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.12:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-212c5ecc create -f -'
Aug  4 10:46:50.116: INFO: rc: 1
Aug  4 10:46:50.117: INFO: will run kubectl (create) again (9 seconds elapsed)
Aug  4 10:46:50.117: FAIL: failed to create vault statefulset failed to run kubectl: error running /usr/bin/kubectl --server=https://192.168.39.12:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-212c5ecc create -f -:
Command stdout:

stderr:
Error from server (AlreadyExists): error when creating "STDIN": serviceaccounts "rbd-csi-vault-token-review" already exists
Error from server (AlreadyExists): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "rbd-csi-vault-token-review" already exists
Error from server (AlreadyExists): error when creating "STDIN": clusterrolebindings.rbac.authorization.k8s.io "rbd-csi-vault-token-review" already exists

error:
exit status 1

Should get fixed with #2329

nixpanic · 2021-08-04T11:10:57Z

/retest ci/centos/mini-e2e-helm/k8s-1.22

nixpanic · 2021-08-04T11:15:09Z

/retest ci/centos/mini-e2e-helm/k8s-1.22

resizing cephfs volume failed (logs):

Aug  4 11:56:29.209: INFO: current size in status {{1073741824 0} {<nil>} 1Gi BinarySI},expected size {{10737418240 0} {<nil>} 10Gi BinarySI}
Aug  4 11:56:29.209: INFO: waiting for PVC csi-cephfs-pvc (600 seconds elapsed)
Aug  4 11:56:29.212: INFO: current size in status {{1073741824 0} {<nil>} 1Gi BinarySI},expected size {{10737418240 0} {<nil>} 10Gi BinarySI}
Aug  4 11:56:29.212: FAIL: failed to resize PVC with error timed out waiting for the condition

nixpanic · 2021-08-04T12:56:37Z

/retest ci/centos/mini-e2e/k8s-1.21

nixpanic · 2021-08-04T15:47:14Z

/retest ci/centos/mini-e2e/k8s-1.21

nixpanic · 2021-08-04T18:50:48Z

@Mergifyio rebase

More PRs got merged, lets restart CI jobs

The VAULT_AUTH_MOUNT_PATH is a Vault configuration parameter that allows a user to set a non default path for the Kubernetes ServiceAccount integration. This can already be configured for the Vault KMS, and is now added to the Vault Tenant SA KMS as well. Signed-off-by: Niels de Vos <ndevos@redhat.com>

mergify · 2021-08-04T18:51:46Z

Command rebase: success

Branch has been successfully rebased

Madhu-1 · 2021-08-05T04:28:31Z

/test ci/centos/mini-e2e/k8s-1.21

Madhu-1 · 2021-08-05T04:29:15Z

/test ci/centos/mini-e2e/k8s-1.21

`Aug 4 20:23:22.683: INFO: >>> kubeConfig: /root/.kube/config
Aug 4 20:23:23.708: INFO: stdErr occurred: /bin/sh: 1: cannot create /var/lib/www/html/test: Read-only file system

Aug 4 20:23:23.708: FAIL: failed to write IO, err: command terminated with exit code 2, stdErr: /bin/sh: 1: cannot create /var/lib/www/html/test: Read-only file system
`
https://jenkins-ceph-csi.apps.ocp.ci.centos.org/blue/rest/organizations/jenkins/pipelines/mini-e2e_k8s-1.21/runs/624/nodes/87/steps/90/log/?start=0

pkalever · 2021-08-05T05:51:33Z

/test ci/centos/mini-e2e/k8s-1.21

`Aug 4 20:23:22.683: INFO: >>> kubeConfig: /root/.kube/config
Aug 4 20:23:23.708: INFO: stdErr occurred: /bin/sh: 1: cannot create /var/lib/www/html/test: Read-only file system

Aug 4 20:23:23.708: FAIL: failed to write IO, err: command terminated with exit code 2, stdErr: /bin/sh: 1: cannot create /var/lib/www/html/test: Read-only file system
`
https://jenkins-ceph-csi.apps.ocp.ci.centos.org/blue/rest/organizations/jenkins/pipelines/mini-e2e_k8s-1.21/runs/624/nodes/87/steps/90/log/?start=0

Cool!! good catch ceph-csi CI :-)

Looking into this now. And thanks for reporting this @Madhu-1 👍

Madhu-1 reviewed Jul 26, 2021

View reviewed changes

internal/util/vault_sa.go Show resolved Hide resolved

Madhu-1 previously approved these changes Jul 26, 2021

View reviewed changes

nixpanic force-pushed the rbd/encryption/VAULT_AUTH_MOUNT_PATH branch from 1d75a24 to 20fbe95 Compare July 26, 2021 12:28

nixpanic requested a review from Madhu-1 July 26, 2021 15:34

nixpanic mentioned this pull request Jul 26, 2021

e2e: retry deploying CephFS components on failure #2325

Merged

nixpanic requested a review from humblec July 27, 2021 07:11

nixpanic added backport-to-release-v3.4 component/rbd Issues related to RBD component/util Utility functions shared between CephFS and RBD labels Aug 2, 2021

Madhu-1 approved these changes Aug 3, 2021

View reviewed changes

humblec approved these changes Aug 3, 2021

View reviewed changes

nixpanic force-pushed the rbd/encryption/VAULT_AUTH_MOUNT_PATH branch from 20fbe95 to 2225381 Compare August 3, 2021 08:13

nixpanic force-pushed the rbd/encryption/VAULT_AUTH_MOUNT_PATH branch from 2225381 to e8d7742 Compare August 3, 2021 13:09

nixpanic mentioned this pull request Aug 3, 2021

e2e: use original namespace for retrying resize check #2354

Merged

nixpanic force-pushed the rbd/encryption/VAULT_AUTH_MOUNT_PATH branch from e8d7742 to 2c4dced Compare August 3, 2021 16:28

nixpanic force-pushed the rbd/encryption/VAULT_AUTH_MOUNT_PATH branch from 2c4dced to c22d034 Compare August 4, 2021 06:07

mergify bot closed this in #2354 Aug 4, 2021

nixpanic reopened this Aug 4, 2021

nixpanic mentioned this pull request Aug 4, 2021

util: add vaultAuthNamespace option for Vault KMS #2339

Merged

nixpanic force-pushed the rbd/encryption/VAULT_AUTH_MOUNT_PATH branch from c22d034 to 2a6b2c2 Compare August 4, 2021 11:56

nixpanic mentioned this pull request Aug 4, 2021

CI jobs often fail with minikube error X Exiting due to GUEST_PROVISION #2264

Closed

nixpanic mentioned this pull request Aug 4, 2021

Deploying Rook sometimes fails in the CI #1636

Closed

nixpanic force-pushed the rbd/encryption/VAULT_AUTH_MOUNT_PATH branch from 2a6b2c2 to ddac0f3 Compare August 4, 2021 17:24

nixpanic force-pushed the rbd/encryption/VAULT_AUTH_MOUNT_PATH branch from ddac0f3 to fa37ae8 Compare August 4, 2021 18:51

Madhu-1 mentioned this pull request Aug 5, 2021

rbd-nbd: occasionally file system entering into read-only on nodeplugin restart #2204

Open

1 task

mergify bot merged commit 4859f2d into ceph:devel Aug 5, 2021

mergify bot mentioned this pull request Aug 5, 2021

util: allow configuring VAULT_AUTH_MOUNT_PATH for Vault Tenant SA KMS (backport #2322) #2363

Merged

nixpanic mentioned this pull request Aug 5, 2021

e2e: CephFS resize failed #2364

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

util: allow configuring VAULT_AUTH_MOUNT_PATH for Vault Tenant SA KMS #2322

util: allow configuring VAULT_AUTH_MOUNT_PATH for Vault Tenant SA KMS #2322

nixpanic commented Jul 26, 2021

Madhu-1 commented Jul 26, 2021 •

edited

Loading

Madhu-1 commented Jul 26, 2021

nixpanic commented Jul 26, 2021

nixpanic commented Jul 26, 2021

nixpanic commented Jul 26, 2021

nixpanic commented Jul 26, 2021

nixpanic commented Aug 3, 2021

nixpanic commented Aug 3, 2021

nixpanic commented Aug 3, 2021

mergify bot commented Aug 3, 2021

nixpanic commented Aug 3, 2021

nixpanic commented Aug 3, 2021

nixpanic commented Aug 3, 2021

nixpanic commented Aug 4, 2021

mergify bot commented Aug 4, 2021

nixpanic commented Aug 4, 2021

nixpanic commented Aug 4, 2021

nixpanic commented Aug 4, 2021

nixpanic commented Aug 4, 2021

nixpanic commented Aug 4, 2021

nixpanic commented Aug 4, 2021

nixpanic commented Aug 4, 2021

nixpanic commented Aug 4, 2021

mergify bot commented Aug 4, 2021

Madhu-1 commented Aug 5, 2021

Madhu-1 commented Aug 5, 2021

pkalever commented Aug 5, 2021

util: allow configuring VAULT_AUTH_MOUNT_PATH for Vault Tenant SA KMS #2322

util: allow configuring VAULT_AUTH_MOUNT_PATH for Vault Tenant SA KMS #2322

Conversation

nixpanic commented Jul 26, 2021

Madhu-1 commented Jul 26, 2021 • edited Loading

Madhu-1 commented Jul 26, 2021

nixpanic commented Jul 26, 2021

nixpanic commented Jul 26, 2021

nixpanic commented Jul 26, 2021

nixpanic commented Jul 26, 2021

nixpanic commented Aug 3, 2021

nixpanic commented Aug 3, 2021

nixpanic commented Aug 3, 2021

mergify bot commented Aug 3, 2021

nixpanic commented Aug 3, 2021

nixpanic commented Aug 3, 2021

nixpanic commented Aug 3, 2021

nixpanic commented Aug 4, 2021

mergify bot commented Aug 4, 2021

nixpanic commented Aug 4, 2021

nixpanic commented Aug 4, 2021

nixpanic commented Aug 4, 2021

nixpanic commented Aug 4, 2021

nixpanic commented Aug 4, 2021

nixpanic commented Aug 4, 2021

nixpanic commented Aug 4, 2021

nixpanic commented Aug 4, 2021

mergify bot commented Aug 4, 2021

Madhu-1 commented Aug 5, 2021

Madhu-1 commented Aug 5, 2021

pkalever commented Aug 5, 2021

Madhu-1 commented Jul 26, 2021 •

edited

Loading