Add kubectl 'cert-manager check api' command #4205

inteon · 2021-07-12T13:42:44Z

What this PR does / why we need it:
Adds the kubectl 'cert-manager check api' command.
This check attempts to perform a dry-run create of a cert-manager v1alpha2
Certificate resource in order to verify that CRDs are installed and all the
required webhooks are reachable by the K8S API server.
We use v1alpha2 API to ensure that the API server has also connected to the
cert-manager conversion webhook.

Release note:

Added the kubectl 'cert-manager check api' command

/kind feature

jetstack-bot · 2021-07-12T13:42:59Z

Hi @inteon. Thanks for your PR.

I'm waiting for a jetstack or cert-manager member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

inteon · 2021-07-12T13:44:27Z

I reused some of your PR's (#4171) code.
/assign @wallrj

SgtCoDFish · 2021-07-12T13:52:59Z

/ok-to-test

pkg/util/cmapichecker/cmapichecker.go

wallrj

This works pretty well Tim, thanks.

But please:

Use it after the helm upgrade --install in the E2E tests.
Add unit-tests
Link to some prototype documentation for it.
Make it react quickly to ctrl-c
Make it print meaningful messages to the console, currently I see

bazel run //cmd/ctl --  check api
...
while attempting dry-run creation of Certificate: error finding the scope of the object: failed to get restmapping: no matches for kind "Certificate" in group "cert-manager.io"
while attempting dry-run creation of Certificate: error finding the scope of the object: failed to get restmapping: no matches for kind "Certificate" in group "cert-manager.io"
while attempting dry-run creation of Certificate: error finding the scope of the object: failed to get restmapping: no matches for kind "Certificate" in group "cert-manager.io"
^Cwhile attempting dry-run creation of Certificate: error finding the scope of the object: failed to get restmapping: no matches for kind "Certificate" in group "cert-manager.io"
while attempting dry-run creation of Certificate: error finding the scope of the object: failed to get restmapping: no matches for kind "Certificate" in group "cert-manager.io"
while attempting dry-run creation of Certificate: error finding the scope of the object: failed to get restmapping: no matches for kind "Certificate" in group "cert-manager.io"
while attempting dry-run creation of Certificate: error finding the scope of the object: failed to get restmapping: no matches for kind "Certificate" in group "cert-manager.io"

In the absence of cert-manager.

And if I leave it running while I install cert-manager I get:

while attempting dry-run creation of Certificate: error finding the scope of the object: failed to get restmapping: no matches for kind "Certificate" in group "cert-manager.io"
while attempting dry-run creation of Certificate: error finding the scope of the object: failed to get restmapping: no matches for kind "Certificate" in group "cert-manager.io"
while attempting dry-run creation of Certificate: error finding the scope of the object: failed to get restmapping: no matches for kind "Certificate" in group "cert-manager.io"

while attempting dry-run creation of Certificate: Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": x509: certificate signed by unknown authority
while attempting dry-run creation of Certificate: Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": x509: certificate signed by unknown authority
while attempting dry-run creation of Certificate: Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": x509: certificate signed by unknown authority
while attempting dry-run creation of Certificate: Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": x509: certificate signed by unknown authority
while attempting dry-run creation of Certificate: Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": x509: certificate signed by unknown authority
while attempting dry-run creation of Certificate: Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": x509: certificate signed by unknown authority
The Kubernetes Api is ready to created cert-manager resources against

Try and make it a bit more user friendly.

cmd/ctl/pkg/check/api/api.go

pkg/util/cmapichecker/cmapichecker.go

Signed-off-by: Inteon <42113979+inteon@users.noreply.github.com>

… fixed os.Exit(1) Signed-off-by: Inteon <42113979+inteon@users.noreply.github.com>

wallrj

@inteon This is looking good.

A few more comments and suggestions.

cmd/ctl/pkg/check/api/api.go

devel/addon/certmanager/install.sh

cmd/ctl/pkg/check/api/api.go

pkg/util/cmapichecker/cmapichecker.go

pkg/util/cmapichecker/cmapichecker_test.go

pkg/util/cmapichecker/cmapichecker.go

Signed-off-by: Inteon <42113979+inteon@users.noreply.github.com>

wallrj

This is great @inteon

I tested the various output flags and I really like the logged messages:

Without an API server

bazel-out/k8-fastbuild-ST-4c64f0b3d5c7/bin/cmd/ctl/kubectl-cert_manager check api
Error: while creating client: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused

Timeout while cert-manager is installing

bazel-out/k8-fastbuild-ST-4c64f0b3d5c7/bin/cmd/ctl/kubectl-cert_manager check api --wait=1m -v
2021/07/16 12:29:12 Not ready: the cert-manager CRDs are not yet installed on the Kubernetes API server (error finding the scope of the object: failed to get restmapping: no matches for kind "Certificate" in group "cert-manager.io")
2021/07/16 12:29:17 Not ready: the cert-manager CRDs are not yet installed on the Kubernetes API server (error finding the scope of the object: failed to get restmapping: no matches for kind "Certificate" in group "cert-manager.io")
2021/07/16 12:29:25 Not ready: the cert-manager CRDs are not yet installed on the Kubernetes API server (error finding the scope of the object: failed to get restmapping: no matches for kind "Certificate" in group "cert-manager.io")
2021/07/16 12:29:33 Not ready: the cert-manager CRDs are not yet installed on the Kubernetes API server (error finding the scope of the object: failed to get restmapping: no matches for kind "Certificate" in group "cert-manager.io")
2021/07/16 12:29:43 Not ready: the cert-manager webhook deployment is not ready yet (Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.96.12.222:443: connect: connection refused)
2021/07/16 12:29:47 Not ready: the cert-manager webhook deployment is not ready yet (Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.96.12.222:443: connect: connection refused)
2021/07/16 12:29:52 Not ready: the cert-manager webhook deployment is not ready yet (Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.96.12.222:443: connect: connection refused)
2021/07/16 12:29:57 Not ready: the cert-manager webhook deployment is not ready yet (Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.96.12.222:443: connect: connection refused)
2021/07/16 12:30:02 Not ready: the cert-manager webhook deployment is not ready yet (Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.96.12.222:443: connect: connection refused)
2021/07/16 12:30:08 Not ready: the cert-manager webhook CA bundle is not injected yet (Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": x509: certificate signed by unknown authority)
2021/07/16 12:30:12 Timed out after 1m0s

Eventual Success

bazel-out/k8-fastbuild-ST-4c64f0b3d5c7/bin/cmd/ctl/kubectl-cert_manager check api --wait=1m -v
2021/07/16 12:30:16 Not ready: the cert-manager webhook CA bundle is not injected yet (Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": x509: certificate signed by unknown authority)
2021/07/16 12:30:21 Not ready: the cert-manager webhook CA bundle is not injected yet (Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": x509: certificate signed by unknown authority)

2021/07/16 12:30:26 Not ready: the cert-manager webhook CA bundle is not injected yet (Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": x509: certificate signed by unknown authority)
2021/07/16 12:30:31 The cert-manager API is ready

Immediate success

bazel-out/k8-fastbuild-ST-4c64f0b3d5c7/bin/cmd/ctl/kubectl-cert_manager check api --wait=1m -v
2021/07/16 12:30:34 The cert-manager API is ready

Non-waiting user-friendly success output

bazel-out/k8-fastbuild-ST-4c64f0b3d5c7/bin/cmd/ctl/kubectl-cert_manager check api
The cert-manager API is ready

Unexpected error with clean ctrl-c exit


bazel-bin/hack/bin/kubectl-cert_manager check api -v --as anon --wait=1m
2021/07/16 12:37:59 Not ready: certificates.cert-manager.io is forbidden: User "anon" cannot create resource "certificates" in API group "cert-manager.io" in the namespace "default"
2021/07/16 12:38:05 Not ready: certificates.cert-manager.io is forbidden: User "anon" cannot create resource "certificates" in API group "cert-manager.io" in the namespace "default"
2021/07/16 12:38:10 Not ready: certificates.cert-manager.io is forbidden: User "anon" cannot create resource "certificates" in API group "cert-manager.io" in the namespace "default"
2021/07/16 12:38:15 Not ready: certificates.cert-manager.io is forbidden: User "anon" cannot create resource "certificates" in API group "cert-manager.io" in the namespace "default"
2021/07/16 12:38:20 Not ready: certificates.cert-manager.io is forbidden: User "anon" cannot create resource "certificates" in API group "cert-manager.io" in the namespace "default"
2021/07/16 12:38:25 Not ready: certificates.cert-manager.io is forbidden: User "anon" cannot create resource "certificates" in API group "cert-manager.io" in the namespace "default"
^C

And the tests now cover all the translateError cases

And the e2e test now precompiles the CLI for faster API check

For information on how to configure cert-manager to automatically provision
Certificates for Ingress resources, take a look at the `ingress-shim`
documentation:

https://cert-manager.io/docs/usage/ingress/
$TEST_TMPDIR defined: output root default is '/bazel-scratch/.cache/bazel' and max_idle_secs default is '15'.
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 30d7517b-66f5-411d-a266-3786d113e452
Loading: 
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
    currently loading: hack/bin
Analyzing: target //hack/bin:kubectl-cert_manager (1 packages loaded, 0 targets configured)
Analyzing: target //hack/bin:kubectl-cert_manager (17 packages loaded, 18 targets configured)
Analyzing: target //hack/bin:kubectl-cert_manager (93 packages loaded, 7373 targets configured)
Analyzing: target //hack/bin:kubectl-cert_manager (144 packages loaded, 7487 targets configured)
Analyzing: target //hack/bin:kubectl-cert_manager (225 packages loaded, 7872 targets configured)
Analyzing: target //hack/bin:kubectl-cert_manager (433 packages loaded, 9093 targets configured)
Analyzing: target //hack/bin:kubectl-cert_manager (639 packages loaded, 11271 targets configured)
Analyzing: target //hack/bin:kubectl-cert_manager (784 packages loaded, 12180 targets configured)
INFO: Analyzed target //hack/bin:kubectl-cert_manager (863 packages loaded, 13064 targets configured).
INFO: Found 1 target...
[2 / 545] [Prepa] BazelWorkspaceStatusAction stable-status.txt
[466 / 1,247] checking cached actions
[1,246 / 1,247] checking cached actions
Target //hack/bin:kubectl-cert_manager up-to-date:
  bazel-bin/hack/bin/kubectl-cert_manager_script
  bazel-bin/hack/bin/kubectl-cert_manager
INFO: Elapsed time: 19.311s, Critical Path: 2.89s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
$TEST_TMPDIR defined: output root default is '/bazel-scratch/.cache/bazel' and max_idle_secs default is '15'.
INFO: Invocation ID: 7c24871c-0cd8-48a4-9cb9-e7e74f8ed03f
2021/07/16 11:20:01 The cert-manager API is ready

-- https://storage.googleapis.com/jetstack-logs/pr-logs/pull/jetstack_cert-manager/4205/pull-cert-manager-e2e-v1-21/1415992708257812480/build-log.txt

I think the use of error.Unwrap is a bit unconventional. I'd rather have used errors.As, but it works fine so let's merge and get this into the next cert-manager alpha release.

Please add a PR to the website (release-next branch) with some documentation explaining when and how to use this command a release note paragraph about it, and recommend it in the https://cert-manager.io/docs/installation/kubernetes/#verifying-the-installation (if you haven't already done that).

I'm just going to run the e2e test against K8S 1.16 to check that our dry-run mechanism works there too:
/test pull-cert-manager-e2e-v1-16

/lgtm
/hold

devel/addon/certmanager/install.sh

jetstack-bot · 2021-07-16T11:56:36Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: inteon, wallrj

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [wallrj]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

wallrj · 2021-07-16T13:42:20Z

/unhold

jetstack-bot assigned wallrj Jul 12, 2021

jetstack-bot added ok-to-test and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 12, 2021

inteon force-pushed the kubectl_check_api branch from c7dbd73 to 323110e Compare July 12, 2021 14:54

inteon commented Jul 13, 2021

View reviewed changes

pkg/util/cmapichecker/cmapichecker.go Outdated Show resolved Hide resolved

wallrj requested changes Jul 14, 2021

View reviewed changes

jetstack-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 15, 2021

inteon force-pushed the kubectl_check_api branch from c941dcc to 9aef0e3 Compare July 15, 2021 12:33

inteon added 2 commits July 15, 2021 16:50

Add kubectl 'cert-manager check api' command

5458173

Signed-off-by: Inteon <42113979+inteon@users.noreply.github.com>

made errors human readable, added unit tests, added check api to e2e,…

ac7775b

… fixed os.Exit(1) Signed-off-by: Inteon <42113979+inteon@users.noreply.github.com>

wallrj requested changes Jul 15, 2021

View reviewed changes

improved ux

21bc989

Signed-off-by: Inteon <42113979+inteon@users.noreply.github.com>

inteon force-pushed the kubectl_check_api branch from 9aef0e3 to 21bc989 Compare July 16, 2021 11:11

wallrj approved these changes Jul 16, 2021

View reviewed changes

devel/addon/certmanager/install.sh Show resolved Hide resolved

jetstack-bot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm Indicates that a PR is ready to be merged. labels Jul 16, 2021

jetstack-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 16, 2021

jetstack-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 16, 2021

jetstack-bot merged commit 88e85d0 into cert-manager:master Jul 16, 2021

jetstack-bot added this to the v1.5 milestone Jul 16, 2021

inteon mentioned this pull request Jul 16, 2021

Use 'kubectl cert-manager check api' in verify-upgrade.sh #4218

Merged

wallrj mentioned this pull request Jul 19, 2021

Consider whether the exit codes and handling of SIGINT / ctrl-c in kubectl cert-manager could be improved #4222

Closed

This was referenced Aug 10, 2021

[WIP] Add install to cert-manager cli #4125

Closed

A webhook startup probe to check that the cert-manager API is usable #4171

Closed

This was referenced Mar 6, 2022

Retry on conflict for the end-to-end test "added an additional dnsName" #4924

Merged

Retry on conflict for the end-to-end test "CA Injector for api services should update data when the certificate changes" #4925

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add kubectl 'cert-manager check api' command #4205

Add kubectl 'cert-manager check api' command #4205

inteon commented Jul 12, 2021

jetstack-bot commented Jul 12, 2021

inteon commented Jul 12, 2021

SgtCoDFish commented Jul 12, 2021

wallrj left a comment

wallrj left a comment

wallrj left a comment

jetstack-bot commented Jul 16, 2021

wallrj commented Jul 16, 2021

Add kubectl 'cert-manager check api' command #4205

Add kubectl 'cert-manager check api' command #4205

Conversation

inteon commented Jul 12, 2021

jetstack-bot commented Jul 12, 2021

inteon commented Jul 12, 2021

SgtCoDFish commented Jul 12, 2021

wallrj left a comment

Choose a reason for hiding this comment

wallrj left a comment

Choose a reason for hiding this comment

wallrj left a comment

Choose a reason for hiding this comment

jetstack-bot commented Jul 16, 2021

wallrj commented Jul 16, 2021