[BUG] CODE_EXECUTOR_INGRESS_DOMAIN uses the unqualified service name and cannot be overridden via .Values.env

[gautier-levert]

[BUG] CODE_EXECUTOR_INGRESS_DOMAIN uses the unqualified service name and cannot be overridden via .Values.env

Description

On backend pods the chart injects the code-executor URL as the unqualified service name:

templates/deployment_backend.yaml (gated by retool.workflows.enabled):

- name: CODE_EXECUTOR_INGRESS_DOMAIN
  value: http://{{ template "retool.codeExecutor.name" . }}

retool.codeExecutor.name resolves to <fullname>-code-executor (e.g. retool-code-executor) — a bare
service name with no namespace, so it relies on the pod's resolv.conf search list being applied by
whatever resolver the caller uses.

The backend's own HTTP client (used for the code-executor health check, and presumably for runtime code
execution) fails to reach this unqualified name. With the chart default it surfaces as:

Could not detect healthy Code Executor. To ignore failures of this check on startup,
set IGNORE_CODE_EXECUTOR_STARTUP_CHECK=true.

and the backend crash-loops unless IGNORE_CODE_EXECUTOR_STARTUP_CHECK=true is set. Setting that flag only
silences the startup check; it does not make the code executor reachable.

What we verified (and what we did not)

From inside a backend pod (namespace retool, chart 6.11.10, image tryretool/backend:4.0.5-stable, on EKS):

• The unqualified name does resolve through the OS resolver:
  • node -e "require('dns').lookup('retool-code-executor', console.log)" → returns the service ClusterIP.
  • A plain node http.get('http://retool-code-executor/api/checkHealth') → 200.
• The fully-qualified name works everywhere:
  http://retool-code-executor.retool.svc.cluster.local/api/checkHealth → 200, and the backend's own
  health check succeeds when pointed at it.

So the observable fact is: the backend's HTTP client does not apply the resolv.conf search domains, so
the unqualified name fails for it while it works for a standard dns.lookup/http.get. We have not
identified the exact resolver/library inside the (compiled) retool_backend binary, so we are not claiming a
specific root cause beyond "the failing path is not search-domain-aware".

Why .Values.env is not a workaround

Overriding via env.CODE_EXECUTOR_INGRESS_DOMAIN does not work. The chart emits its own entry first
(above), and .Values.env is rendered later via include "retool.env" .Values.env. The rendered manifest
therefore contains two CODE_EXECUTOR_INGRESS_DOMAIN entries, and on apply the duplicate collapses (env
list is merged by the name key) keeping the chart's first/unqualified value. The override is silently lost.

Reproduction Steps

1. Render the chart while trying to override the value:

   helm template retool retool/retool --version 6.11.10 \
     --set workflows.enabled=true \
     --set env.CODE_EXECUTOR_INGRESS_DOMAIN=http://retool-code-executor.retool.svc.cluster.local \
     | grep -n -A1 CODE_EXECUTOR_INGRESS_DOMAIN

   Observed: two entries in the backend Deployment — the chart's http://retool-code-executor (first) and the
   .Values.env FQDN (second). On apply, only the first (unqualified) survives.

2. On a backend pod, confirm the unqualified name resolves via the OS resolver but the backend's health check
   still fails, while the FQDN works (see "What we verified" above).

Impact

• Any environment whose backend HTTP client does not apply the resolv.conf search list cannot reach the
  code executor with the chart defaults.
• The only chart-provided mitigation (IGNORE_CODE_EXECUTOR_STARTUP_CHECK=true) hides the startup failure but
  does not restore code-executor connectivity.
• There is no supported way to point the backend at the fully-qualified code-executor address, because
  .Values.env cannot override the chart's own entry.

Proposed Fix

The simplest, root-cause fix is client-side; the chart-level options are fallbacks if the client behavior
cannot change. This repo is the chart, so option 1 likely needs to be routed to the backend team.

1. Make the backend resolve unqualified names the way a standard client does (i.e. apply the
   resolv.conf search list — e.g. resolve via getaddrinfo/dns.lookup). Then the chart's existing
   unqualified value works as-is, with no chart change and no per-deployment override needed. This is the most
   direct fix, since a plain http.get to the same unqualified name already returns 200 from inside the pod.

2. Emit the fully-qualified service name by default in the chart, so it works regardless of the caller's
   search-domain handling:

   - name: CODE_EXECUTOR_INGRESS_DOMAIN
     value: http://{{ template "retool.codeExecutor.name" . }}.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain | default "cluster.local" }}

3. Expose an overridable value (e.g. codeExecutor.ingressDomain) that, when set, replaces the chart's
   computed value in-place (single entry), instead of relying on .Values.env (which cannot win the duplicate
   merge).

Environment

• Chart: 6.11.10
• Retool image: tryretool/backend:4.0.5-stable
• Kubernetes: EKS
• workflows.enabled: true