The Ziti Controller Helm Chart Hangs During Install #133
Comments
|
Is it possible it is successful but slow? That configmap is one of the last things to be automatically created. It's used by a Trust Manager Bundle resource to store Ziti's trust anchors. It's created by trust-manager when all the necessary certificates are finally created by cert-manager. If trust-manager is waiting for cert-manager then there should be a complaint in the logs from one or both of them. |
|
@qrkourier Thank you for the insight. Also, this is the end of the |
|
@TheDarkula Does this seem to be reproducible with the latest release of the ziti-controller chart, or reproducible with an experimental ziti-controller chart, or more or less sporadic? Since the conditions seem to be related to the DNS SANs on the leaf certificate that's bound to the edge API web listeners, does it recur when you change the advertised address of one or both of those APIs? |
|
@qrkourier I am using the latest helm chart version. I have this section in the I am just thinking things through, does that DNS record need to exist publicly? |
Yes, in most cases, global, public DNS records make the most sense for a Ziti network that provides an overlay spanning multiple points on the internet. The client API's advertised address, value The Ziti controller has exactly one client API advertised address, and all SDKs and routers must use this same domain name to connect to the client API. This allows the SDKs and routers to verify the DNS SAN presented by the controller during TLS negotiation. |
|
@qrkourier Thank you for the information :) I created a public DNS record for the ziti-controller. That moved things along a bit. Now, the Running Running Running Looking at the |
|
Thanks for supplying the logs! Working backward from the controller pod waiting for its volume I suspect the trust-manager "trust namespace" (link to doc) is not aligned perfectly. This must be set to the K8s namespace where cert-manager is composing the secrets and configmaps that are sourced by trust-manager. The trust-manager trust namespace is configured with Helm value This raises the question, "How can we detect this misalignment or, ideally, ensure the alignment automatically?" |
|
@qrkourier Sure thing! I just tried re-installing the chart with the namespace set to Here is the output of The |
|
I'm unsure why the resources that clearly do exist are not found. Barring any cluster-level security measures, I suspect an inconsistent state could have been caused by changing the namespace at the time of a Helm release upgrade. Do you get a different result if you first delete the Helm release then install fresh with an aligned namespace? You're installing trust-manager and cert-manager as sub-charts of ziti-controller, or installing one or both of those separately in advance of ziti-controller? Are you installing the trust-manager or cert-manager CRDs separately in advance, or as part of their respective chart releases? If you're unsure, and using ziti-controller chart defaults, then I'll be able to reason it out. |
|
@qrkourier I just ran I then re-ran I installed cert-manager and trust-manager ahead of time, not as a sub-chart of the controller chart. As far as the errors shown when describing the controller pod, the secrets definitely do exist, but it also complains about one configmap: Looking at the output from I had the thought that the Something tells me that the |
|
Here's the controller's Dockerfile. |
|
The controller's init container is waiting to start because that CM isn't available. |
|
I understand. With that, it seems that the Thank you for the link to the Where is that file? I have a feeling that there is some sort of hard pause or a failure to wait/retry going on there. |
|
Gack. I overlooked what you needed there. Yes, the controller's container image is relatively minimal. That initialization script is provided by the Helm chart here: https://github.com/openziti/helm-charts/blob/main/charts/ziti-controller/templates/configmap.yaml#L9 The logic simply checks whether the controller's BBolt DB file exists and runs the initialization procedure to create it if not. The way the controller's pod is specified, the init container must succeed before the app container(s) start. Are you saying you deleted the controller pod that was seemingly stuck at the init state and then both ran successfully? |
|
No worries :) Yes, I deleted the controller pod when it was complaining about the secrets. |
|
That means the secrets-related warning messages like this occurred during the first run of the controller pod, and were resolved before the second run that happened when you deleted the first pod. This lends credibility to the hypothesis that the problem is still trust-manager can't "see" the secrets that it needs to compose the Bundle resource that creates the Let's look closer at that. Here's a warning from trust-manager that represents the problem: Does it exist in the "ziti" namespace? Is "ziti" trust-manager's trust namespace? |
|
The secret does exist. However, the second output is blank, because I have trust-manager deployed to the From my understanding of the link you provided, if I want to use trust-manager with multiple deployments, leaving the trust namespace set to The only other way I could see it working, is if I create multiple trust-manager namespaces, like For example:
This seems a bit redundant, but I am not sure if that is the recommended way to configure everything. Would it be more logical to have the ziti-controller chart use the |
|
Does it show us the trust namespace arg if you give it the correct namespace? |
|
Yes, it does, |
|
The issue is that trust-manager is only able to source secrets from trusted namespace "cert-manager," but the necessary secrets are in namespace "ziti." Thanks for raising the question "How should cert/trust-manager be aligned for concurrent OpenZiti namespaces?" I haven't personally explored that deeply yet. The main constraint I've encountered is that trust manager can only have one "trust namespace," and I've engaged the devs to express the need for configurable multiple trust namespaces. As far as I know, that's on their backlog. Meanwhile, we could have a separate discussion to explore multiple instances of trust-manager, one per OpenZiti namespace, assuming the Helm release names are uniquely named and that trust-manager's Cluster-level resources, adopt the Helm release name to ensure uniqueness. |
|
Indeed! Sure thing :) What seems to be most logical for the ziti-controller chart, is to have two different configurations:
Something like this: |
|
Helm is limited in its ability to check that an arbitrary set of input values expresses a sane configuration. It's still worthwhile to ensure that the default values are sane together, but things get complicated from there. For this reason, it's necessary to have oversight or orchestration or expertise at some level that's higher than Helm. I've experimented with using Terraform to both document and automate feeding a sane set of inputs to the Helm chart. That worked pretty well, but I'm still hoping to arrive at something that feels more satisfying and acceptable to a wider audience. Hypothetically, maintaining separate charts for each sane set of default input values would work, but it would also be hard work to rectify drift between them, so it would eventually demand some kind of nanny/parent system that renders each variant. That doesn't feel like the right direction. A more appealing alternative is Kustomize, which can patch the rendered templates. I've fiddled with Kustomize, which is built-in to
|
|
The top-level Kustomize would work well in this case when trust-manager is installed as a sub-chart. Helm could render the chart and sub-chart templates to manifests which can then be patched by Kustomize to ensure alignment. We're currently pinning the upstream version of trust-manager, but another option would be to maintain a fork that requires a custom input value like the ziti-controller namespace, and ensures that it matches the trust namespace. |
|
Thank you for all of the information! With that, I do have one more query. How can I specify |
|
Glad to hear that! Also: I'm happy to respond here and want to make sure you're aware of a rich resource: the Discourse forum: https://openziti.discourse.group/. We'd be happy to have you asking/answering over there too. A lot of those topics start out as symptoms or ideas and mature into GitHub Issues when they become sufficiently well-defined. I'll start working on the |
|
I will create an account momentarily :) |
I started playing with the helm charts, and I tried to install the ziti-controller.
Unfortunately, the
initContainernever succeeds, but hangs on a configmap:I installed the chart with this command:
The text was updated successfully, but these errors were encountered: