split k8s_custom_deploy into multiple resources

[nicks]

Current Behavior

Currently, when you deploy a Helm chart with helm_resource, Tilt tracks it as a single resource. This means:

• there's one KubernetesDiscovery object
• there's one UIResource

This means that if your chart has multiple deployments in the chart:

• there's only one item in the UI
• all deployments log to the same resource
• restarting the item redeploys the whole chart
• tilt picks one pod to determine health check status / readiness
• tilt picks one pod to do port forwarding

Describe the Feature You Want

A way to represent the chart as multiple resources in the Tilt UI

Additional context
I have mixed feelings about this. We may have to break it up into multiple feature requests. The problem is that a "resource" in tilt aggregates many different things, and breaking them up might get really hairy (e.g., if you break a chart into 2 resources, and restart one, does that mean the other one must restart? If the answer is "no", that breaks Helm chart install semantics. If the answer is "yes", that breaks Tilt restart semantics.)

[nicks]

one idea i played around with was to do something like:

k8s_custom_deploy(
  ...,
 port_forwards=[
    port_forward(8080, 8080, selector={"app": "labelA"}),
    port_forward(8081, 8081, selector={"app": "labelB"})
  ]
)

or you could go even further, and represent it as a series of templates, like the underlying api:

https://api.tilt.dev/kubernetes/kubernetes-apply-v1alpha1.html

k8s_custom_deploy(
  discovery_templates=[
    discovery_template(selector={"app": "labelA"}, pod_log_template=..., port_forward_template=...),
    discovery_template(selector={"app": "labelB"}, ui_resource="labelB", pod_log_template=..., port_forward_template=...),
  ])

[nicks]

i also am not sure if there's a way to do this without having to encode a lot of the chart internals in the tiltfile, which feels odd

[rrmistry]

What if we (users of Tilt) add labels to our resources that would "hint" Tilt to split resources within Tilt?

E.g. tilt.dev/resource : "<Tilt Resource Label Value>"

[nicks]

One way to workaround this is to use k8s_attach.

So you might have a chart that creates multiple deployments, and you can use k8s_attach to create separate port-forwards:

load('ext://k8s_attach', 'k8s_attach')
load('ext://helm_resource', 'helm_resource')

helm_resource(
    name="my-name",
    chart='my-chart')

k8s_attach(
    name="deployment-0",
    obj="deployment/deployment-0",
    port_forwards=[
        "8000:8000", 
    ],
    resource_deps=["my-chart"],
)

k8s_attach(
    name="deployment-1",
    obj="deployment/deployment-1",
    port_forwards=[
        "8001:8001",
    ],
    resource_deps=["my-chart"],
)

[nijave]

One way to workaround this is to use k8s_attach.

So you might have a chart that creates multiple deployments, and you can use k8s_attach to create separate port-forwards:

load('ext://k8s_attach', 'k8s_attach')
load('ext://helm_resource', 'helm_resource')

helm_resource(
    name="my-name",
    chart='my-chart')

k8s_attach(
    name="deployment-0",
    obj="deployment/deployment-0",
    port_forwards=[
        "8000:8000", 
    ],
    resource_deps=["my-chart"],
)

k8s_attach(
    name="deployment-1",
    obj="deployment/deployment-1",
    port_forwards=[
        "8001:8001",
    ],
    resource_deps=["my-chart"],
)

Some more context https://kubernetes.slack.com/archives/CESBL84MV/p1692118493208569

If it's ok to have a dependency on k8s_attach in helm_resource, some middle-ground might be creating an attachments keyword argument that accepts a list of Kubernetes Object Selectors. That would leave it up to the user to figure out what should become a resource and what shouldn't.

[rrmistry]

What if we (users of Tilt) add labels to our resources that would "hint" Tilt to split resources within Tilt?

E.g. tilt.dev/resource : "<Tilt Resource Label Value>"

Made some progress with workload_to_resource_function on this, but getting stuck by bug #6349 at the moment.

[rrmistry]

Got this idea semi-working now (though a bit janky) with this workaround: #6349 (comment)

The workaround will use kubectl command in an OS agnostic way to read kubernetes labels tilt.dev/resource and tilt.dev/label and then dynamically set the resource in Tilt UI.

• Label value for tilt.dev/resource will rename that resource in the Tilt UI
• Label value for tilt.dev/label will assign it a tilt group in the Tilt UI

With this feature, addressing the original post, we can handle the shared issues with helm_resource by switching to the Tilt native k8s_yaml(helm(chart_path)) pattern provided that the necessary labels are assigned in YAML manifests. This apporach even works for helm_remote extension provided that the remote chart allows setting resource labels (and we set them of course).

The only janky part of the workaround at the moment is that it requires you to manually re-trigger the Tiltfile resource from the Tilt web UI to reflect the new changes on any label value edits. I'm sure there are many tilt friendly ways to get past even this, but for now, the current workaround is good enough for us to move forward.

If anyone else has any ideas along similar lines then please do share with us!

additional notes

Tested this using helm_remote for both labels by setting them under kube-prometheus-stack.prometheus-node-exporter.podLabels as helm values and observing the Tilt web UI to see reflected changes.

Ideally, would like this to be implemented in Tilt binary itself as natively supported feature. But I can contribute a PR to Tilt extensions after figuring out the janky part and making it tilt friendly.

I suspect it will be difficult to detect and hook onto changes to environment variables with the current workaround unless a file is used as a proxy and a reference to some previous environment variable state. This seems like the Tilt way to deal with this.

I can add my Prometheus example as a test case to the tilt extension. However, it would be ideal to implement this in base Tilt as it seems like a very sensible feature IMHO.