Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to install cert-manager v0.11 via Rancher 2.3.2 #23850

Closed
dnauck opened this issue Nov 2, 2019 · 21 comments
Closed

Unable to install cert-manager v0.11 via Rancher 2.3.2 #23850

dnauck opened this issue Nov 2, 2019 · 21 comments

Comments

@dnauck
Copy link

dnauck commented Nov 2, 2019

What kind of request is this (question/bug/enhancement/feature request):
Bug?

Steps to reproduce (least amount of steps as possible):

  1. Add jetstack ( https://charts.jetstack.io ) repository as app catalog via rancher ui.
  2. Install cert-manager CRDs: kubectl apply --validate=false -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.11/deploy/manifests/00-crds.yaml
  3. Create "cert-manager" namespace in System project.
  4. Launch "cert-manager" app from catalog with name "cert-manager" and in existing namespace "cert-manager"

image

Result:

Error: resource's namespace kube-system doesn't match the current namespace cert-manager

[main] 2019/11/02 11:47:05 Starting Tiller v2.14+unreleased (tls=false)
[main] 2019/11/02 11:47:05 GRPC listening on :47978
[main] 2019/11/02 11:47:05 Probes listening on :36421
[main] 2019/11/02 11:47:05 Storage driver is ConfigMap
[main] 2019/11/02 11:47:05 Max history per release is 10
[tiller] 2019/11/02 11:47:06 getting history for release cert-manager
[storage] 2019/11/02 11:47:06 getting release history for "cert-manager"
Release "cert-manager" does not exist. Installing it now.
[tiller] 2019/11/02 11:47:06 preparing install for cert-manager
[storage] 2019/11/02 11:47:06 getting release history for "cert-manager"
[tiller] 2019/11/02 11:47:06 rendering cert-manager chart using values
2019/11/02 11:47:06 info: manifest "cert-manager/charts/cainjector/templates/psp-clusterrolebinding.yaml" is empty. Skipping.
2019/11/02 11:47:06 info: manifest "cert-manager/templates/servicemonitor.yaml" is empty. Skipping.
2019/11/02 11:47:06 info: manifest "cert-manager/templates/psp.yaml" is empty. Skipping.
2019/11/02 11:47:06 info: manifest "cert-manager/templates/psp-clusterrolebinding.yaml" is empty. Skipping.
2019/11/02 11:47:06 info: manifest "cert-manager/charts/cainjector/templates/psp.yaml" is empty. Skipping.
2019/11/02 11:47:06 info: manifest "cert-manager/templates/psp-clusterrole.yaml" is empty. Skipping.
2019/11/02 11:47:06 info: manifest "cert-manager/charts/cainjector/templates/psp-clusterrole.yaml" is empty. Skipping.
[tiller] 2019/11/02 11:47:06 performing install for cert-manager
[tiller] 2019/11/02 11:47:06 executing 0 crd-install hooks for cert-manager
[tiller] 2019/11/02 11:47:06 hooks complete for crd-install cert-manager
[tiller] 2019/11/02 11:47:06 executing 0 pre-install hooks for cert-manager
[tiller] 2019/11/02 11:47:06 hooks complete for pre-install cert-manager
[storage] 2019/11/02 11:47:06 getting release history for "cert-manager"
[storage] 2019/11/02 11:47:06 creating release "cert-manager.v1"
[storage] 2019/11/02 11:47:06 getting release history for "cert-manager"
[kube] 2019/11/02 11:47:06 building resources from manifest
[tiller] 2019/11/02 11:47:06 warning: Release "cert-manager" failed: resource's namespace kube-system doesn't match the current namespace cert-manager
[storage] 2019/11/02 11:47:06 updating release "cert-manager.v1"
[tiller] 2019/11/02 11:47:06 failed install perform step: release cert-manager failed: resource's namespace kube-system doesn't match the current namespace cert-manager
2019/11/02 11:47:07 [ERROR] AppController p-sv954/cert-manager [helm-controller] failed with : failed to install app cert-manager. Error: release cert-manager failed: resource's namespace kube-system doesn't match the current namespace cert-manager

Other details that may be helpful:

Environment information
Rancher 2.3.2 single instance, aks 1.14.7 cluster

@dnauck
Copy link
Author

dnauck commented Nov 5, 2019

It was also reported on an older version and closed as rancher bug by jetstack:

cert-manager/cert-manager#1380

@insekticid
Copy link

insekticid commented Nov 5, 2019

you are using two different namespaces cert-manager and kube-system, define namespace cert-manager when installing from helm chart

@dnauck
Copy link
Author

dnauck commented Nov 5, 2019

@insekticid I can only select one namespace via rancher ui, and that is "cert-manager"

@insekticid
Copy link

insekticid commented Nov 5, 2019

Look here:
https://github.com/jetstack/cert-manager/blob/dddc6abd2e0075648df2eddcff765b4986e9497d/deploy/charts/cert-manager/templates/webhook-rbac.yaml#L36

and here
https://github.com/jetstack/cert-manager/blob/57f6dad0c273f6c091329fda0e8935d4f535cc20/design/release-notes/release-0.11/draft-release-notes.md

Change the default leader election namespace to 'kube-system' instead of the same namespace as the cert-manager pod, to avoid multiple copies of cert-manager accidentally being run at once (#2155, @munnerz)

Maybe there is bug when installing via helm chart, report it here https://github.com/jetstack/cert-manager/issues

@r734
Copy link

r734 commented Nov 9, 2019

I installed cert-manager v0.11.0 with Helmv3 into Rancher 2.3.2 with the following commands, as described on the official helm chart page:

kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.11/deploy/manifests/00-crds.yaml
helm repo add jetstack https://charts.jetstack.io
kubectl create namespace cert-manager
helm install --generate-name --namespace cert-manager jetstack/cert-manager -f k8s/files/cert-manager-cfg.yaml

Contents of k8s/files/cert-manager-cfg.yaml:

podDnsConfig: None
podDnsConfig:
  nameservers:
    - 1.1.1.1

(Necessary in my setup to avoid issues with the internal network's DNS preventing cert-manager from seeing that TXT records are propagated)

I don't really trust installing this sort of thing from a GUI...Something always seems to go wrong.

Hope this helps!

@dnauck
Copy link
Author

dnauck commented Nov 11, 2019

So it is a limitation or bug in Rancher's helm handling ?! :(

@wc-matteo
Copy link

wc-matteo commented Nov 14, 2019

same issue. It seems rancher doesn't handle installing on multiple ns.

@blackholegalaxy
Copy link

blackholegalaxy commented Nov 28, 2019

If you want to use the catalog, you can add the following option:
global.leaderElection.namespace = cert-manager

cert-manager allow us to override the second destination namespace.

@wc-matteo
Copy link

wc-matteo commented Nov 28, 2019

thanks @blackholegalaxy. Does it have any consequence installing everything in the same ns?

@munnerz
Copy link

munnerz commented Nov 28, 2019

It's not uncommon for a single Helm chart to need to deploy resources into more than once namespace, although it isn't necessarily 'expected'. In this instance, it is required as we need to ensure that leader election is performed between all installations of cert-manager, to prevent two instances running in a single cluster due to being installed multiple times via app catalogs etc.

It seems like Rancher has a Helm operator of some description that is responsible for applying/installing things from the catalog - IMO, this should be extended to not fail/error in cases where resources need to go in other namespaces, as it is an arbitrary restriction and cert-manager will not be the only tool that requires this.

@dnauck
Copy link
Author

dnauck commented Nov 28, 2019

If you want to use the catalog, you can add the following option:
global.leaderElection.namespace = cert-manager

cert-manager allow us to override the second destination namespace.

Sadly that does not work (tested with v.0.12) .. i still get the same error .. strange

Any idea @munnerz? Looks like that parameter is not used by the helm chart.

@blackholegalaxy
Copy link

blackholegalaxy commented Nov 28, 2019

@dnauck I used this technique this afternoon. Added the catalog, then create the App from this catalog with the given option and everything is deployed properly.

image

image

image

@dnauck
Copy link
Author

dnauck commented Nov 28, 2019

It only works with webhook.enabled = false

@blackholegalaxy thanks!!

@himpierre
Copy link

himpierre commented Jan 5, 2020

well, with webook disabled it's pretty much useless, no?

@avluis
Copy link

avluis commented Jan 6, 2020

@himpierre if you need the webhook (in my case for testing), I just edited the annotation webhook.enabled after install and set it to true:

Capture

Hard part is remembering to set the template version to 0.12.0 when upgrading but downgrading works there as well:

Capture
Capture

@himpierre
Copy link

himpierre commented Jan 12, 2020

@avluis Thanks. Seems to work.

@soriyath
Copy link

soriyath commented Jan 18, 2020

for some reasons, it did not work for me.
I had to install it in the kube-system namespace, with this config:

---
  image: 
    tag: "v0.12.0"
  ingressShim: 
    defaultIssuerKind: "ClusterIssuer"
    defaultIssuerName: "letsencrypt-staging"
  webhook: 
    enabled: "true"

@beshkenadze
Copy link

beshkenadze commented Feb 4, 2020

My work solution is:

  1. Install the custom resource definition before the helm app: kubectl apply --validate=false -f https://raw.githubusercontent.com/jetstack/cert-manager/v0.13.0/deploy/manifests/00-crds.yaml

  2. Install the helm app with values:

global.leaderElection.namespace = cert-manager
webhook.enabled = false
---
  global: 
    leaderElection: 
      namespace: "cert-manager"
  webhook: 
    enabled: "false"
  1. Upgrade the helm app with values:
global.leaderElection.namespace = cert-manager
webhook.enabled = true
---
  global: 
    leaderElection: 
      namespace: "cert-manager"
  webhook: 
    enabled: "true"
  1. Profit!

Screenshot 2020-02-04 at 18 24 57

@caiconkhicon
Copy link

caiconkhicon commented Mar 27, 2020

Thank @beshkenadze , your solution works for me. However, it's really bad for our automation (using Terraform). Do you have any idea when will Rancher/Cert-manager fix this issue?

@chrisfosterelli
Copy link

chrisfosterelli commented May 19, 2020

It seems kind of surprising how challenging it is to get letsencrypt running on Rancher. We've got great out of the box support for nginx ingress controllers on RKE clusters... I feel like letsencrypt is another part of that puzzle that could be a first class integration. To do that this needs to be wayyyyy more smooth and without issues like this one.

I'm only offering this as a friendly beginners perspective feedback 😁 I imagine this is seems a bit more of a trivial issue from an experts perspective but the overall "OK I've got an ingress, how do I get certificates?" has been a multi-hour task for someone with less kubernetes background.

@stale
Copy link

stale bot commented Jul 10, 2021

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

@stale stale bot added the status/stale label Jul 10, 2021
@stale stale bot closed this as completed Jul 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests