Helm support: long term plan #4401
Comments
k8s-ci-robot
added
needs-kind
|
/triage accepted |
k8s-ci-robot
added
triage/accepted
|
/label area/helm |
|
@natasha41575: The label(s) In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
FWIW we don't consider krm functions usable in most of the environments we use kustomize. On the other hand, I think the deprecation would help if the plugin is not going to get further support in kustomize itself. It might be worth directing people to https://github.com/mgoltzsche/khelm |
|
Would you mind providing some more feedback for why krm functions are not usable for you? We would like to address friction points as we move toward more krm-function-oriented solutions and extensions. |
|
|
It's also worth mentioning that flux2's kustomize-controller doesn't natively support KRM config functions by way of plugin enablement, nor does it include the helm binary in the container image, and thus the current options are to fork or pre-hydrate (which still sounds like it would be a challenge for @james-callahan until other runtimes are available for KRM config functions) |
|
In #3536 I did get a special built version of Our need matches what was previously stated since we use argocd; we did not want to install docker onto their repo-server. It would be a lot of work if the helm plugin was moved to a krm function and we'd likely stay on an older version or fork for quite some time. Side note, in some prototyping we got around the whole |
|
We are also looking to use this in argocd (as soon as kostumize gains OCI support - which was just released as stable with Helm 3.8.0) - so it would be bad, to limit it like this :( |
|
We are looking at ways to support alternatives to docker, including containerized alternatives as well as simple and safe ways to run exec KRM functions. The latter would run very similarly to the existing builtin plugin. |
|
Since I have been receiving several messages about this, I would like to reiterate again that we will be supporting the new KRM Helm function as executable, meaning that you would not need to run docker in order to use it. In terms of execution, this will be almost exactly the same as running the current builtin Helm plugin, except that the code lives elsewhere and is compiled separately. |
|
I noticed that the helm chart inflation has been moved to the |
@QuinnBast Could you elaborate by what you mean by kubernetes-sigs repository? kubernetes-sigs is the org, and has many repositories. Edited to add: If you are referring to the code in the krm functions registry, it is not ready yet. We are still working on setting up the repo infrastructure. |
|
I have noticed that the lack of the possibility to pass extra-args to helm makes it impossible to properly use helm charts that check capabilities, e.g. cert-manager's in-built Prometheus ServiceMonitor creation, see https://github.com/cert-manager/cert-manager/blob/9887baac3357beb1ad949f9b62f25dc3eea27fb7/deploy/charts/cert-manager/templates/servicemonitor.yaml#L1 My thoughts on that:
|
See #3816 |
|
@james-callahan You sure are persistent in your efforts. +1 Some thoughts
|
|
Please note that in light of slow progress on the KRM Function version, we've updated our mid-term plan around the current built-in. The new policy is outlined in our docs and copied below for ease of reference. Re: extra-args in particular: We used to have such a field, but we removed it in #3784 because it represented a security risk. As such we will not be considering adding it again. Long term supportThe helm chart inflation generator in kustomize is intended to be a limited subset of helm features to help with The current builtinFor enhancements to the helm chart inflation generator feature, we will only support the following changes:
We will not add support for:
Future supportThe next iteration of the helm inflation generator will take the form of a KRM function, which will have |
|
Its kind of a poor Developer Experience that the KRM repo is referenced for helm inflation, but no contributions have put on it for about 6 months - this is typically an indicator of a dying project in the open source world. Im specifically thinking of #4335 where contributions were made by the community, but disregarded for the below option (clearly not an active project) |
|
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
k8s-ci-robot
added
the
lifecycle/stale
|
/lifecycle frozen |
k8s-ci-robot
added
lifecycle/frozen
As a downstream user, I'd like to be able to make use of Helm's |
|
Is this still an active plan? I am seeing no activity in the mentioned repository, nor am I seeing any significant contributions in this repository to refactor the HELM setup. Furthermore, I would like to extend upon the ideas of @mwmahlberg here.
But I see the main reason for me to upgrade has been merged recently(I was looking for --api-versions support); #4926 |
The |
Yep, I was an ignorant first-time Kustomize user when I posted that... and realize that using Kustomize as a way to apply Helm charts kind of removes a lot of the nice features of Helm as a K8s-equivalent of a OS package-manager. |
|
There is some movement on this PR: #5167 |
|
Is the plan to keep requiring the |
|
@KnVerey, @natasha41575 I've reached this page from the docs. It doesn't seem that this Long-term plan for Helm is actually happening, as it's been a year since anyone has contributed to the KRM repo. It does seem like problems with the Is it time to revisit the ideas outlined in this issue? Namely, there are PRs already done with support for private repos, OCI registry, and a few bugs. It would be great to reopen this. Use-case: Cluster setup with ArgoCDHelm in an Argo app can be a bit cumbersome to do with GitOps. There is a Chicken & Egg problem, where you need an App of Apps to initialize an Argo repo from Git, but Helm charts don't fit well with that. Argo UI also puts some limits in how it displays Helm charts too, with some nice advantages such as viewing values and overrides. But for the cluster admin use-case, it can get a bit cumbersome. There is a case to be made that Examples:
While this can be achieved in other ways, having |
|
For our open source project ( https://github.com/Obmondo/k8id ) - we only use helm charts and the 2.6+ feature with having values files in diff. repo - and in the very few instances where we need more, we add templates to the "wrapper helm chart" - and we also have a "static yaml" application which files are in the same "sepeate config repo" - or we extend helm chart and do upstream PR.. But helm + kustomize would give an extra option, which is definely not bad, but AFAIK kostumize's biggest strength is more the ability to handle patching of resources.. and a helm chart would "bark" at that afterwards? (unless argocd does helm template and runs kostumize locally - so it knows the end state - and there will then not be a diff.. I assume thats hot it would work). I still prefer upstream patch - but its definetely a good flexibility to have, to patch "resut of helm charts" - instead of the helm chart itself if necessary. So if you use Helm umbrella model (you have a parent helm chart of your own - that simply depends on upstream chart) - then you have an easy way to extend helm charts (not quite as flexible as kustomize) that works too. You can see in k8id project - under argocd-helm-charts folder for examples :) |
I actually do this, but instead of in the GitOps application, it's done at the pre-commit step. By computing the combined hash of all files in each directory to find which releases have changed, I can automatically generate the relevant Kustomize output, with inflated charts in a It's not a very comprehensive solution, since it's in a single pre-commit python script, but I think that such a model has the potential to be compatible with both Argo and Flux, and also enable static-analysis based CI tooling, like for example to upgrade helm charts like Dependabot. Here is how I manage my lab cluster (I'm planning to refactor the folder structure to support |
|
Hi all. I'm currently the subproject lead of kustomize. In the first step, I plan to import helm lib in kustomize because it currently functions by directly running the Current status: I'm waiting for this PR to be merged. |
<!--Thanks for your contribution. See [CONTRIBUTING](CONTRIBUTING.md)
for Pulumi's contribution guidelines.
Help us merge your changes more quickly by adding more details such
as labels, milestones, and reviewers.-->
## Proposed changes
This PR implements the Kustomize `Directory` resource (v2) as a
multi-language component resource.
The `Directory` resource creates child resources based on a
kustomization directory. The input is a path to a directory containing
'kustomization.yaml', or a git repository URL with a path suffix and
other qualifiers.
### Related issues (optional)
<!--Refer to related PRs or issues: #1234, or 'Fixes #1234' or 'Closes
#1234'.
Or link to full URLs to issues or pull requests in other GitHub
repositories. -->
Closes #2786
### API
|property|description|
|-|-|
| `directory` | The directory or git URL containing the kustomization to
apply. |
| `namespace` | Overrides the default namespace. |
| `skipAwait` | Applies the skipAwait annotation. |
| `resourcePrefix` | Prefix for child resources, defaults to the
component name. |
### Feature: Remote Targets
This implementation supports remote kustomization targets as described
[here](https://github.com/kubernetes-sigs/kustomize/blob/master/examples/remoteBuild.md).
Note: the v1 implementation also supports git references, but resolves
them using the Pulumi SDK
([`RetrieveGitFolder`](https://github.com/pulumi/pulumi/blob/v3.117.0/sdk/go/common/workspace/templates.go#L392))
rather than using Kustomize's own implementation.
### Feature: Enable Alpha Plugins
This implementation always enables Kustomize's plugin support, akin to
`kustomize build --enable-alpha-plugins`.
Plugins are drawn from `KUSTOMIZE_PLUGIN_HOME` (default:
`~/.config/kustomize/plugin/`). Note: the kustomize library doesn't
allow for easy customization of the plugin home.
### Feature: Unrestricted Loading
Kustomize has a strict and a relaxed mode with respect to path
references outside the kustomization base directory
(`--load-restrictor`, default is `strict`). The feature seems intended
to encourage portability, similarly to paths in Dockerfiles. This
implementation simply enables the relaxed mode.
### Feature: Enable Helm Charts
This implementation enables [Helm chart support in
Kustomize](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/helmcharts/),
which is an experimental feature of Directory v1 (see:
#2470). Note that
chart support in Kustomize is limited and evolving; see
kubernetes-sigs/kustomize#4401 for the
long-term support plan.
The `helm` binary is assumed to be on the path.
### Feature: Namespace Override
New to v2 is support for overriding the default namespace (default is
from provider configuration), as a convenience. Kustomize itself has a
similar facility (see:
[namespace](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/namespace/))
but it makes sense to support it natively.
### Feature: Resource Ordering
The Directory resource uses Pulumi's engine to install the resources
produced by kustomize. It automatically detects dependencies between
some resources. For example, it knows to install namespaces and Custom
Resource Definitions (CRDs) first.
Use the `config.kubernetes.io/depends-on` annotation to declare an
explicit resource dependency. See [blog
post](https://www.pulumi.com/blog/kubernetes-yaml-v2/#resource-ordering)
for more info.
### Limitation: Kubernetes-Style Transformations
The older v1 implementation provides the ability to transform the
Kubernetes objects produced by kustomize before being sent to the
server. That feature isn't available in Directory v2 at this time. You
may still use Pulumi's [transform
option](https://www.pulumi.com/docs/concepts/options/transformations/)
to modify the child resources.
### Tests
The PR includes unit tests covering the provider implementation code.
### Example
Here's an example of deploying a couple of kustomizations. The `local`
kustomization (see:
[helloWorld](https://github.com/kubernetes-sigs/kustomize/tree/master/examples/helloWorld)
for details) shows how to use a variable to select a kustomization
overlay.
```yaml
name: issue-2786-yaml
runtime: yaml
description: |
Demonstrates the Directory resource.
Reference: https://github.com/kubernetes-sigs/kustomize/tree/master/examples/helloWorld
config:
variant:
type: string
default: staging
outputs:
name: ${local.resources}
resources:
ns:
type: kubernetes:core/v1:Namespace
local:
type: kubernetes:kustomize/v2:Directory
properties:
namespace: ${ns.metadata.name}
directory: ./config/${variant}
skipAwait: true
remote:
type: kubernetes:kustomize/v2:Directory
properties:
namespace: ${ns.metadata.name}
directory: https://github.com/kubernetes-sigs/kustomize//examples/helloWorld/?ref=v3.3.1
skipAwait: true
```
There are many open helm-related issues in kustomize: https://github.com/kubernetes-sigs/kustomize/issues?q=is%3Aissue+is%3Aopen++label%3Aarea%2Fhelm+
Many of these issues are requesting new options in the helm chart generator configuration that will become flags to
helm template. Other issues are related to problems with the--enable-helmflag and wanting more flexibility in how to enable helm.For some context, there are limited options in the plugin configuration and the
--enable-helmflag exists because of a past security issue that has since been addressed. However, it is clear that the helm chart generator built into kustomize has limitations and we need a plan for long term support.What we would like to do is introduce a new function in the new krm-functions-registry that can inflate a helm chart. This new function will be treated as a third party extension by kustomize so it will not need any special treatment. Additionally, because it will be a containerized function, there will be fewer security problems to worry about.
Once this function is ready, we can start recommending it to users and deprecate the existing helm-related fields in kustomize.
The text was updated successfully, but these errors were encountered: