Publish Constructs as Helm Chart

[tabern]

Auto-generation of helm charts

[cweidinger]

One specific way that we're using helm that cannot be replicated by cdk8s (until this feature is implemented) is optionally creating a resource based on helm values provided deploy time. We'd like one deployable artifact (a helm chart) that can be deployed to any region or level and use helm (or native integration into ArgoCD) values to optionally create resources.

[zbrookle]

@cweidinger If you’re currently using helm I may have a solution that better suits your needs. I’m currently developing a python package called avionix, which is very similar to this package, but it writes out to helm charts and templates rather than just one kube yaml document. The development is almost finished and it should be released in the coming weeks

[excavador]

Any updates? There is very important issue. Without ability to generate helm charts, cdk8s useless for us :( We need release artifacts

[excavador]

Any updates?

[edobry]

Not a contributor, but I do think its worth mentioning how massively non-trivial this feature is; I see two implementation paths and both require large amounts of work:

1. cross-compilation to gotemplate format, possibly as a jsii target
2. implementing arbitrary rendering engine support in helm, to shell out to cdk8s for chart rendering

both have huge associated challenges, but the latter option is a bit more realistic. as such, IMO this feature request would be better suited for the Helm repo/team. Helm used to have a pluggable rendering engine in v2, but they cut support for that in v3 due to lack of interest, so they'd have to be convinced that it would actually be used this time.

Personally, I'm taking the approach of building a wrapper tool for Helm, similar to Helmfile but more generic, which will be able to support both "chart types". Its already being used internally, and I hope to open source it eventually, but that won't be for a while, if ever.

There just isn't really a good solution here otherwise, which is why I'm moving away from Helm.

[excavador]

@edobry thank you for sharing

I understand how non-trivial these changes could be, but AWS CDK already solved this problem (i.e. conditions, parameters, ssm parameters), and this mechanism could be adopter for helm chart as well

[edobry]

@excavador could you share a link to the equivalent AWS CDK solution? I find it hard to imagine how there could be an analog there and would love to learn

[excavador]

@edobry

• https://docs.aws.amazon.com/cdk/api/latest/docs/core-readme.html#intrinsic-functions-and-condition-expressions
• https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_core.Fn.html

[edobry]

@excavador the difference is that the AWS CDK is being "compiled" to the CloudFormation format, which is highly flexible, expressive, and machine-readable, being JSON. Helm templates are not really any of those things, making these situations incomparable

[excavador]

@edobry I do not think there is some real big difference.
Yes, CloudFormation supports conditions directly, but from generation point of view in helm there is just

{{ if <your expression> }}
{{ end }}

around underlying construct tree.

core.Fn is not really different from helm built-in "sprig" functions: you just need to transcode invocation of these functions properly.

If you have construct tree, like in CDK, conditions and intristics functions could be easily presented/rendered.

[excavador]

@edobry I am talking not only just theoretically. We have own internal cdk-like helm chart generation and we already solved these problems.

We have another problems, like with adopting k8s API to typescript-classes or dirty code (too dirty to publish it as opensource).

But conditions and intristic functions are NOT a problem

[eladb]

@edobry is right. This is not a trivial (and perhaps not even feasible) feature to support. The only way I can think of supporting this is by being able to execute arbitrary code when helm expands its templates and at that point, execute cdk8s synth and inject the output. Any ideas?

[excavador]

@eladb you do NOT need to run arbitrary code.
You need to provide way to inject {{ if }} {{ end }} for condition around generated tree (in yaml) and insert invocation of template functions provided by helm (sprig) exactly like AWS CDK insert invocation of AWS CloudFormation intristic functions (see AWS CDK core.Fn)

[eladb]

@excavador I feel like I am missing something. How would you publish the following construct as a Helm chart without statically analyzing the code?

interface MyConstructProps {
  readonly count: number;
}

class MyConstruct extends Construct {
  constructor(scope: Construct, id: string, props: MyConstructProps) {
    super(scope, id, props);

    for (const i = 0; i < count; ++i) {
      new Pod(this, `pod-${i}`, { ... });
    }
  }
}

[excavador]

@eladb this example will not work AND SHOULD NOT WORK.
Try to look to terraform, or AWS CDK, in IaC example like this DOES NOT WORK.

There is really tricky question, why you should NEVER write CDK applications like this, but I will try to explain.

Let's consider AWS CDK.
AWS CDK - typescript application, which produce (using cdk synth) CloudFormation Template.

Let's look closer to CloudFormation Template. You have no loop there at all. You can NOT provide input parameter "N=5" and receive five S3 buckets.
The only possible way to do it - provide CloudFormation CustomResource, but there is hack and is not intended.

Why AWS CloudFormation designed in this way?

There a some different reasons, but let me highlight key points:

• complex template - complex logic - hard to review and understand how it works. Template for IaC (infrastructure as code) never should be complex, it should be reflection of your deployment, and this deployment should be static.
• you never really need loops. You example - you are creating some pods, number of pods depends from input variable. Why you do this? You have k8s Deployments and StatefulSets with "replicaCount" argument and there is direct way how you should create pods in kubernetes (not manually, but using k8s controller).

So, technically you CAN write some AWS CDK applications with loops, BUT on synth phase it would be rendered to fixed number of resources.
Technically you CAN insert loop to helm chart template, BUT is does not make any sense and should be avoided at all.

You if avoid look based on input arguments (you can, of course, produce loops based on static arrays, it will be resolved to separate static templates), then you have very simple approach:

• you have FIXED number of children resource in the chart (without any loops at all)
• you need to support condition for arbitrary children construct tree (insert in helm template in proper location {{ if <condition> }} ... {{ end }}
• you need to support sprig functions like supported in core.Fn in AWS CDK.

I have never seen any good helm chart with loop. In general you never need loops

[eladb]

@excavador but this is what this feature is about, and generally what the CDK is about - it's about being able to leverage the full power of programming languages in order to define cloud abstractions. Loops, conditions, polymorphism, etc. It's totally legitimate if users prefer to use simple templating to generate their helm charts but this feature is about the idea of being able to create helm charts from constructs.

As I said earlier, there might be a feasibility issue here (exactly because helm is [intentionally] not expressive enough)...

[excavador]

@eladb yes, I totally agree with your point about the full power of programming language.

The only difference is understanding what exactly RESULT of this program is.

From your point of view it should be some objects in kubernetes.
From my point of view it should be deployable artifacts (helm chart).

Some regulations requires to provide source code and binary artifacts, used to deploy to production and keep it for 5 years.
From this point of view helm chart is very good defined format to describe installable package for kubernetes.

You believe that generated helm chart should reflect ALL possible things from AWS CDK8S application in helm chart - like for loop in your example.
I believe (based on my 4 years experience of devops, developing helm charts and applications for kubernetes) that a useful and GOOD helm chart NEVER will have any loops and your example and desire makes the task very complex without any real needs for it.

Let's consider examples:

• your for loop for creating pod. Example is broken by design, you never should create pod manually. You should create Job, CronJob, Deployments, StatefulSets and let these controllers to create necessary children pods (and, by the way, restart/recreate them if something going wild)
• multiple host for ingress (some bad charts contains loops) - should be avoided at all, you always can use "toYaml" function from helm pre-defined helpers and reflect what-ever-you-want structure from values to ingress definition (this is the default way to solve a problem like this).

If you take to account, that good-defined charts do not contain any loop, then you will follow my lead and point, how this GitHub issue could be done in exactly the same way like AWS CDK did for CloudFormation and this way is the only way to generated artifacts with delayed installation.

By the way, sometimes I USE LOOPS in my AWS CDK applications for ad-hoc installation instead cdk synth and generated CloudFormationTemplate.
For this purpose, AWS CDK8S is already good enough.

Let's move back to REASON why this GitHub issue was born - it's not about ad-hoc installation, this issue is about deployable well-defined artifacts, installation package, which could be used by some tools like helm or helmfile.

I hope I explained my point.

[excavador]

@eladb also please take a look to "AWS CDK Tokens" - https://docs.aws.amazon.com/cdk/latest/guide/tokens.html

There is how sprig functions could be injected to helm chart.

[edobry]

@excavador if I'm understanding correctly, the central issue here seems to be that your expectations for how cdk8s, and more generally Helm/K8s generation tools SHOULD be used are different than many other peoples'. Lets assume for the sake of argument that your opinion on loops in templates being bad design is correct; how would you propose to enforce such a rule, when cdk8s is explicitly meant to be used within a full-fledged programming language, as @eladb mentioned? Would the cdk8s-cli need to statically-analyze your program and exit if it detected any for or .map(...) tokens?

While I'm sure you have great reasons, based on experience, for your design preferences, they're far from universal, and you may not get very far expecting tools to strictly adhere to them, when they're explicitly meant for other usecases.

[excavador]

@edobry Thank you for the great question!

So, direct answer: just as I described. If AWS CDK8S application has some loop, it means, loop should be in the application.
It could be loop through name of the microservices: like generate the same pattern for microservice A, microservice B, microservice C.
It's really does not matter at all.

As the result of execution of CDK application (at least, I guess for CDK8S the same story) you have the Construct Tree

The task "generate helm chart" should be solved as "transform every leaf from Construct Tree to single gotmpl-based yaml file for helm chart" and, of course, collect parameters (see AWS CDK Token link above) to helm chart default values.

Any loop which exists in AWS CDK8S application will be executed during chart generation, but in the result helm chart you will have no loop at all.

There is the key.

Conditions - different story. Imagine with one parameter value triggered "true" branch, while on another - "false" branch.
There is example where you need to include to helm chart BOTH branches.
How to do it? Replace "if" in your code to "condition" like I showed for AWS CDK
https://docs.aws.amazon.com/cdk/api/latest/docs/core-readme.html#intrinsic-functions-and-condition-expressions

[dududko]

@edobry I think that @excavador means the following:

If there is the following CDK stack

interface MyConstructProps {
  readonly count: number;
}

class MyConstruct extends Construct {
  constructor(scope: Construct, id: string, props: MyConstructProps) {
    super(scope, id, props);
    const N = props.count;

    for (const i = 0; i < N; ++i) {
      new Pod(this, `pod-${i}`, { ... });
    }
  }
}

The code eventually should be transformed into a single gotmpl file with no loops

cat templates/pod.yaml
--- pod 1
apiVersion: v1
kind: Pod
metadata:
  name: {{ $.Release.Name }}-{{ $.Chart.Name}}-pod-1

--- pod 2
apiVersion: v1
kind: Pod
metadata:
  name: {{ $.Release.Name }}-{{ $.Chart.Name}}-pod-2

...

--- pod N
apiVersion: v1
kind: Pod
metadata:
  name: {{ $.Release.Name }}-{{ $.Chart.Name}}-pod-N

[excavador]

@dududko thank you for the example.
There is exactly what I meant.

[excavador]

Any updates?

[edobry]

@excavador don't want to speak for @eladb, but we've already outlined why this is not really feasible. I know you responded but there are complex reasons why your proposal would not work, primarily having to do with differences in expressiveness between TS/gotmpl, and the difficulties of cross-compiling an arbitrary TS program. Contrary to what you said, its not actually possible to limit this cross-compilation to simple loops and branches, as a CDK8s chart can utilize any TS language construct/pattern, including potentially-unbounded recursion, complex NPM packages, and so on.

What COULD be done is just to output generated YAML packaged in the Helm format, to be used as a simple static deployment package, but this does not seem to be what you're asking for.

[github-actions]

This issue is now marked as stale because it hasn't seen activity for a while. Add a comment or it will be closed soon.

[excavador]

So, could you please describe the official cdk8s position: have we ever receive ability to generate helm chart (in helm chart format, tar gzip archive with specific gotmpl-based yaml files inside) as output of cdk8s application (like we can receive CloudFormation Template from cdk application using cdk synth command) or not?

[mrvisser]

primarily having to do with differences in expressiveness between TS/gotmpl, and the difficulties of cross-compiling an arbitrary TS program.

I'm a little confused as to why it's so important in this discussion to enforce an expectation that if statements, loops, and variable references in your cdk8s TS will be transcribed at deploy-time in your Helm manifests.

As @excavador pointed out, other popular libraries such as AWS CDK are able to function with deploy-time parameters, which feed into conditional resource deployment and arity, and everyone seems OK with the fact that their typescript if statements and for loops aren't the way to achieve those deploy-time objectives.

In AWS CDK if you want deploy-time conditions, you can use FnIf. If you want deploy-time concatenation of strings, you use FnJoin, and so on. There's no expectation that I can just do ${deployTimeVar1}:${deployTimeVar2} and that can be injected into a native resource variable and actually work, and it's still powerful to have that ability.

I'm not arguing that this is trivial, it will no doubt come with a substantial addition of resources and a toolkit for generating a limited set of gotmpl expressions, but declaring it unfeasible on the primary basis that language-native for loops and if statements aren't going to achieve a specific goal would be very unfortunate.

Maybe we can direct the conversation towards what a HelmValues resource that can expose configuration values might need to look like, and then what an API of Gotmpl utilities might look like that can generate some rudimentary, albeit limited, expressions for variable reference (e.g., $.Values...), conditional blocks (e.g., Gotmpl.if('<gotmpl expression>', <resource>)), and so on... what would be the minimum amount of Gotmpl support needed to be useful, and what would be some reasonable escape hatches to bridge gaps?

[github-actions]

This issue is now marked as stale because it hasn't seen activity for a while. Add a comment or it will be closed soon. If you wish to exclude this issue from being marked as stale, add the "backlog" label.

[juanrh]

Translating cdk8s programs to Helm charts certainly would be complex, but I think it would be already useful to consider both cdk8s and Helm as a way of defining functions from Helm values to k8s manifests. Ideally we would be able to distribute cdk8s programs like Helm charts, and deploy them with versioning and rollback support. A simple approach for that could be writing a Helm plugin that:

1. Fetches a cdk program as a docker image. Private repos could be also configured, and this could be a way of distributing cdk8 programs.
2. Renders a cdk manifest, using as input a set of Helm values that could be specified as plugin flags and args, and that would be passed as an props/options argument for the main cdk8s Chart object.
3. Creates a chart that just contains the rendered chart in templates, and some metadata in Chart.yaml, and calls helm install with that chart.

[juanrh]

I wrote a proof of concept for the Helm plugin here, for those interested

[excavador]

Do we have any updates in this direction?

[iliapolo]

We are not currently pursuing this

[excavador]

We are not currently pursuing this

Do you have this feature in roadmap?