[Bug] Defined variables not found at Deploy

[crazyelectron-io]

Configuration

• Debian 12 host (w/latest updates)
• Docker version 28.0.1, build 068a01e
• Komodo Core version: v1.16.12 (running in container)
• Periphery 1.16.12 running under systemd

I have currently 11 stacks up and running and defined 49 variables/secrets.

The issue

There were frequent issues with defined variables not being found when (re)deploying a stack but they always 'disappeared' by retrying several times or restarting Komodo Core and/or Periphery. But now I am creating the 12th stack and I just can't get past this issue.

The complete message when doing initial Deploy of this new stack is:

interpolate global variables
Stage 1 of 6
POSTGRES_DB => postgres
HOME_DOMAIN => **REDACTED**
N8N_DATABASE_NAME => n8n

interpolate core secrets
Stage 2 of 6
replaced: N8N_ENCRYPTION_KEY
replaced: POSTGRES_USER
replaced: N8N_DATABASE_USER
replaced: N8N_DATABASE_PASSWORD
replaced: POSTGRES_PASSWORD

git pull
Stage 3 of 6
command: cd /etc/komodo/stacks/n8n && git pull --rebase --force origin main
stdout: Already up to date.
stderr: 
From https://GitHub.com/crazyelectron-io/homeserver
 * branch            main       -> FETCH_HEAD

latest commit
Stage 4 of 6
command: cd /etc/komodo/stacks/n8n && git rev-parse --short HEAD && git rev-parse HEAD && git log -1 --pretty=%B
stdout: 
hash: 5562b39
message: Fix Postgres Cluster dependency

interpolate periphery secrets
Stage 5 of 6
stderr: ERROR: failed to interpolate secrets into environment
TRACE:
	1: did not find any value for variable N8N_DATABASE_NAME

compose up failed
Stage 6 of 6
stderr: ERROR: Failed to write / clone compose file
TRACE:
	1: Stopped after repo pull failure

The toml configuration file for this stack is:

[[stack]]
name = "n8n"
[stack.config]
server = "homeserver"
project_name = "n8n"
poll_for_updates = true
run_directory = "./n8n"
git_provider = "GitHub.com"
git_account = "crazyelectron-io"
repo = "crazyelectron-io/homeserver"
ignore_services = ["postgres-create"]
environment = """
N8N_DATABASE_NAME = [[N8N_DATABASE_NAME]]
N8N_DATABASE_USER = [[N8N_DATABASE_USER]]
N8N_DATABASE_PASSWORD = [[N8N_DATABASE_PASSWORD]]
N8N_ENCRYPTION_KEY = [[N8N_ENCRYPTION_KEY]]
HOME_DOMAIN =[[HOME_DOMAIN]]
POSTGRES_USER = [[POSTGRES_USER]]
POSTGRES_PASSWORD = [[POSTGRES_PASSWORD]]
POSTGRES_DB = [[POSTGRES_DB]]
"""

As can be seen there are 8 variables/secrets defined in the stack's environment section and all of them are interpolated in stage 1 and 2. But still Komodo complains about the first variable or secret defined in the environment section (I tried swapping them).

For completeness a part of the toml output from the variables:

  ...
##

[[variable]]
name = "N8N_DATABASE_NAME"
value = "n8n"
is_secret = false

##

[[variable]]
name = "N8N_DATABASE_USER"
value = "**REDACTED**r"
is_secret = true

##

[[variable]]
name = "N8N_DATABASE_PASSWORD"
  ...

The logging

The logging in Komodo Core when doing the Deploy:

INFO ExecuteRequest{req_id=7909ae8e-64be-4b10-b776-b879a4e15f77 user_id="67c9ca36f55d64ebf731089c" update_id="67d524163542d7031e15d39f" request="DeployStack"}: core::api::execute: /execute request 7909ae8e-64be-4b10-b776-b879a4e15f77 | user: **REDACTED**

And there is no related logging in Komodo Periphery at that moment.

Additional info

Apart from this issue I also noticed that refreshing the repo's gives a warning in the logs for all created stacks (they are GitHub based).
In Periphery an entry for all stacks:

WARN periphery::router: request 8a11987e-e1c5-4fd3-90f3-adc0a2acabc1 | type: ComposePull | error: Stopped after repo pull failure

And in Core for all stacks:

 WARN core::api::write::stack: Failed to pull latest images for Stack postgres | request to periphery failed | 500 Internal Server Error: Stopped after repo pull failure

These are logged at regular intervals but not while trying to deploy the stack of this topic.

BTW, turning on debug for Periphery floods the log with messages of dozens of kB each (listing all Docker details) which I cannot dump here.

[Maltahl]

I have this exact issue on both 1.6.0 and 1.7.0

[crazyelectron-io]

It seems to be a hit/mis scenario in some cases: first Deploy fails because of a missing variable, click Deploy again and everything is fine.

[Maltahl]

If i deploy the same compose.yaml file in UI defined deployment then the interpolation goes through as it is suppose to.
In git deployment via repo (gitea) variable interpolation is not being done.

[Cloudworks99]

I seem to have the same issue:
I am trying to run a docker container with Traefik and Letsencrypt on cloudflare. Using git deployment via repo (forgejo)
The docker compose file contains:

environment:
CF_DNS_API_TOKEN: "[[CF_DNS_API_TOKEN]]"

I also have created a variable called: CF_DNS_API_TOKEN

I think I should be using it without quotes but if i try that the deploy fails with:
validating /etc/komodo/stacks/traefik_internal/stacks/homeserver/traefik_internal/compose.yaml: services.traefik.environment.CF_DNS_API_TOKEN must be a string, number, boolean or null

If I keep the quotes the container starts but gives me an error because the value is invalid.

If I open a console into the container the command 'env' gives me:
CF_DNS_API_TOKEN=[[CF_DNS_API_TOKEN]]

When putting in the actual token instead of using the variable it works.

So to me it looks like the variable replacement using git deployment is failing at this time.

[crazyelectron-io]

With the latest version of Komodo, I have not seen the behaviour I mentioned before. All seems stable now.

[Cloudworks99]

I am running the current latest version 1.17.5 and I do see these issues.

[Cloudworks99]

Also version 1.18 is failing at this. I have created the following simple compose to reproduce this:

services:
  alpine:
    image: alpine:latest
    stdin_open: true
    tty: true
    command: /bin/sh
    environment:
      - MYSECRET=[[MYSECRETVAR]]
    restart: unless-stopped

And I have defined a value for MYSECRETVAR

Now I create 2 stacks using the same compose yaml (changed service name of course:
1 stack is created with mode 'Git Repo'
1 stack is created with mode 'UI Defined'

When deploying the stacks the results differ:
For the 'Git Repo' stack:

Deploy Stack

admin
[alpine](http://192.168.10.XX:9120/stacks/123456123456789123456)
03/06/2025, 10:02:15 2.4 seconds

Set git remote
Stage 1 of 6|0.0 seconds

command
cd /stacks/alpine && git remote set-url origin http://token:<TOKEN>@192.168.1.X:3300/org/myrepository

Checkout branch
Stage 2 of 6|0.0 seconds

command
cd /komodo/stacks/alpine && git checkout -f main

stdout
Your branch is up to date with 'origin/main'.

stderr
Already on 'main'

Git pull
Stage 3 of 6|0.8 seconds

command
cd /komodo/stacks/alpine && git pull --rebase --force origin main

stdout
Already up to date.

stderr
From http://192.168.1.X:3300/org/myrepository
 * branch            main       -> FETCH_HEAD

Latest Commit
Stage 4 of 6|0.0 seconds

command
cd /komodo/stacks/alpine && git rev-parse --short HEAD && git rev-parse HEAD && git log -1 --pretty=%B

stdout
hash: 679efd6
message: [Komodo] admin: Write Compose File: update stacks/homeserver/alpine/compose.yaml

Compose Pull
Stage 5 of 6|0.9 seconds

command
cd /komodo/stacks/alpine/stacks/repository/ && docker compose -p alpine -f compose.yaml pull

stderr
 alpine Pulling 
 alpine Pulled 

Compose Up
Stage 6 of 6|0.2 seconds

command
cd /komodo/stacks/alpine/stacks/repository/alpine&& docker compose -p alpine -f compose.yaml up -d

stderr
 Network alpine_default  Creating
 Network alpine_default  Created
 Container alpine  Creating
 Container alpine  Created
 Container alpine  Starting
 Container alpine  Started

When i go into the container shell I get the following value for MYSECRET when running 'env'

MYSECRET=[[MYSECRETVAR]]

And for the 'UI Defined' stack the deploy results are:

Deploy Stack

admin
[alpine2](http://192.168.10.XX:9120/stacks/987654987654321987654)
03/06/2025, 10:00:59 2.3 seconds

interpolate core secrets
Stage 1 of 3|0.0 seconds

stdout
replaced: MYSECRETVAR

Compose Pull
Stage 2 of 3|0.9 seconds

command
cd /komodo/stacks/alpine2 && docker compose -p alpine2 -f compose.yaml pull

stderr
 alpine2 Pulling 
 alpine2 Pulled 

Compose Up
Stage 3 of 3|1.1 seconds

command
cd /komodo/stacks/alpine2 && docker compose -p alpine2 -f compose.yaml up -d

stderr
 Container alpine2  Creating
 Container alpine2  Created
 Container alpine2  Starting
 Container alpine2  Started

When i go into the container shell I get the following value for MYSECRET when running 'env'

MYSECRET=NotSoSecretAintItQuestionMark

As one can see for the "Git Repo" mode there is no 'interpolate core secrets' stage and MYSECRET is just gives me back what I entered in the compose yaml. In the "UI Defined" mode MYSECRET is properly read from the KOMODO variables.

[eikaramba]

i'm on v1.18.1. Same problem here, i have a ui defined compose.yaml file using environment variables (which are also just defined in the env textarea of the same stack). however pulling image or deploying results in:

Write Stack
Stage 2 of 2|
0.0 seconds
stderr
ERROR: failed to interpolate secrets into file contents
TRACE:
	1: did not find any value for variable PG_PASSWORD

[bengkelgawai]

I have the same issue, for both git and file. The only way interpolation is working is with UI Defined. I will try to reproduce it in with a simpler docker image as above example.

[eikaramba]

it seems this is fixed in 1.18.4! for me it works now

[jptlabs]

This is not fixed. I'm still having this issue on 1.18.4. As others have mentioned above the current workaround is deploying them through the UI.

[joshpl03]

I'm a new user to Komodo, same issue here on v1.18.4: git repo stack does not appear to have a step where interpolation happens (for deploy). I get an sderr that says that the env variable must be a string, number, boolean or null.