Skip to content

Add env/file interpolation for config.toml and APIP_CP_ env prefix#2616

Merged
thivindu merged 11 commits into
wso2:mainfrom
thivindu:config-interpolation
Jul 15, 2026
Merged

Add env/file interpolation for config.toml and APIP_CP_ env prefix#2616
thivindu merged 11 commits into
wso2:mainfrom
thivindu:config-interpolation

Conversation

@thivindu

@thivindu thivindu commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Brings the config-injection model agreed in discussion #1983 to the platform-api, and tightens how secrets are provided cross the sample deployments. Three logical changes:

  1. Config interpolation - any config field can now pull its value from an environment
    variable ({{ env "NAME" }}) or an allowlisted file ({{ file "/path" }}), resolved at
    startup, fail-closed.
  2. APIP_CP_ env-var prefix - config-override environment variables are now namespaced
    APIP_CP_*, matching the gateway's APIP_GW_ and the Developer Portal's APIP_DP_.
  3. Secret hygiene - sample configs/compose files no longer hardcode secrets; they reference
    them via {{ env }} / {{ file }} tokens with values supplied from a .env file or a mounted
    secret file, and docs are updated to teach the preferred pattern.

Related to

1. Config interpolation

  • Consumes the shared common/configinterpolate package (from feat(config): add env/file interpolation for config.toml #2596) in platform-api/config/config.go, wired as an interpolate(k) step after the env+file merge and before unmarshal, mirroring the gateway-controller.
  • File reads are restricted to a per-component allowlist (/etc/platform-api, /secrets/platform-api), overridable via the shared APIP_CONFIG_FILE_SOURCE_ALLOWLIST.
  • Fails closed: a missing/empty required env var, or a missing/disallowed/oversize file, aborts startup. Resolved values are never logged (only reference counts, at info level).
  • Backward compatible: a config value with no {{ … }} token passes through untouched.
  • Adds github.com/knadh/koanf/providers/confmap for the post-interpolation reload.

2. APIP_CP_ environment-variable prefix

Config-override env vars must now be prefixed with APIP_CP_. The prefix is stripped and the
remainder mapped to a config key:

Before After
LOG_LEVEL=debug APIP_CP_LOG_LEVEL=debug
DATABASE_HOST=… APIP_CP_DATABASE_HOST=…
AUTH_JWT_SECRET_KEY=… (direct override) APIP_CP_AUTH_JWT_SECRET_KEY=…

Two variables are intentionally not prefixed:

  • APIP_DEMO_MODE — a standalone runtime flag (read directly, not via the config loader).
  • APIP_CONFIG_FILE_SOURCE_ALLOWLIST — the shared interpolation allowlist.

Note on the two naming contexts: the APIP_CP_ prefix applies only to the direct config-override path. The {{ env "NAME" }} interpolation tokens read bare names via
os.LookupEnv, so {{ env "ENCRYPTION_KEY" }} reads ENCRYPTION_KEY, while the direct override for the same key is APIP_CP_ENCRYPTION_KEY.

Action required for operators: rename any platform-api config env vars to APIP_CP_*. All first-party compose/CI stacks in this repo have been updated.

3. Secret handling

Secrets should never be hardcoded - not as raw values in config.toml, nor as literals in a compose file. Instead they are referenced in the config with a token and resolved at startup:

encryption_key = '{{ env "ENCRYPTION_KEY" }}'                     # from an env var (e.g. a .env file)
# preferred (production) — from a mounted secret file:
# encryption_key = '{{ file "/secrets/platform-api/encryption_key" }}'
  • Both ENCRYPTION_KEY and AUTH_JWT_SECRET_KEY remain required and are never auto-generated.
  • Sample configs (platform-api/config/config.toml, the ai-workspace and developer-portal templates) now use tokens; raw literals removed.
  • Added .env.example templates next to each sample stack (.env is git-ignored).
  • Test/CI compose files keep their throwaway hardcoded values (they are fixtures, not deployments).

Files changed

Code

  • platform-api/config/config.go — interpolation wiring, APIP_CP_ prefix, extracted validateJWTConfig / validateEncryptionKey.
  • platform-api/config/config_test.go — updated env-var names; new TestLoadConfig_EnvPrefix.
  • platform-api/internal/server/server.go — user-facing error/warning strings reference the prefixed names.
  • platform-api/go.mod / go.sum — add confmap.

Docs

  • platform-api/README.md, docs/ai-workspace/features/secrets-management.md, docs/ai-workspace/configuration.md.

Sample deployments

  • distribution/all-in-one/docker-compose.yaml, portals/ai-workspace/docker-compose.yaml, portals/developer-portal/docker-compose*.yaml, plus sample config TOMLs and new .env.example files.

Test/CI stacks (prefix rename only)

  • tests/integration-e2e/docker-compose{,.sqlite,.sqlserver}.yaml
  • portals/developer-portal/it/docker-compose.test{,.postgres}.yaml

Related

@coderabbitai

coderabbitai Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Platform API configuration now uses APIP_CP_ environment values, TOML environment/file interpolation, required-key validation, and updated Compose, tests, examples, documentation, and runtime guidance. Integration configurations and service wiring use the renamed variables, while the LLM provider example adds a context path.

Suggested reviewers: malinthaprasan, tharsanan1, virajsalaka, tharindu1st, rakhitharr

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description has useful details, but it does not follow the required template and omits mandatory sections like Purpose, tests, and Test environment. Rewrite the description to match the template and add the missing sections: Purpose, Goals, Approach, User stories, tests, security checks, samples, related PRs, and Test environment.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately captures the main change: adding config interpolation and the new APIP_CP_ env prefix.
Docstring Coverage ✅ Passed Docstring coverage is 88.24% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

Dependency Validation Results

Dependency name: github.com/knadh/koanf/providers/confmap
Version: v1.0.0
Allowed range: >=v1.0.0
Approved: ✅ Yes

⚠️ Please verify the scope of the dependencies usage is necessary

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
distribution/all-in-one/docker-compose.yaml (1)

54-77: 🗄️ Data Integrity & Integration | 🔴 Critical | ⚡ Quick win

Add the Platform API secrets to platform-api
distribution/all-in-one/docker-compose.yaml doesn’t pass APIP_CP_ENCRYPTION_KEY / APIP_CP_AUTH_JWT_SECRET_KEY, and it doesn’t mount a config.toml either, so the container falls back to defaults and startup validation will reject the missing keys. Add those vars here or inject them via env_file, matching the test compose files.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@distribution/all-in-one/docker-compose.yaml` around lines 54 - 77, Add the
required Platform API secrets to the platform-api service configuration, using
APIP_CP_ENCRYPTION_KEY and APIP_CP_AUTH_JWT_SECRET_KEY with the same injection
approach and values or env_file pattern used by the test compose files. If those
values are defined through config.toml there, mount the corresponding
config.toml into platform-api as well so startup validation receives both keys.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/ai-workspace/features/secrets-management.md`:
- Around line 384-389: Update the secrets setup instructions around the `.env`
example to separate shell key generation from `.env` contents: show `openssl
rand -hex 32` commands whose outputs users paste as `AUTH_JWT_SECRET_KEY` and
`ENCRYPTION_KEY`, rather than documenting command substitution inside the env
file.

In `@portals/ai-workspace/.env.example`:
- Around line 8-11: Update the platform-api service in docker-compose.yaml to
pass AUTH_JWT_SECRET_KEY and ENCRYPTION_KEY into the container via env_file or
explicit environment entries, so the mounted config-platform-api.toml can
resolve both env tokens; do not rely on the host .env file alone.

---

Outside diff comments:
In `@distribution/all-in-one/docker-compose.yaml`:
- Around line 54-77: Add the required Platform API secrets to the platform-api
service configuration, using APIP_CP_ENCRYPTION_KEY and
APIP_CP_AUTH_JWT_SECRET_KEY with the same injection approach and values or
env_file pattern used by the test compose files. If those values are defined
through config.toml there, mount the corresponding config.toml into platform-api
as well so startup validation receives both keys.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 7115bf82-4ec6-4206-ac10-9de0e95a3b55

📥 Commits

Reviewing files that changed from the base of the PR and between 7a66828 and e3e2d61.

⛔ Files ignored due to path filters (1)
  • platform-api/go.sum is excluded by !**/*.sum
📒 Files selected for processing (25)
  • distribution/all-in-one/.env.example
  • distribution/all-in-one/docker-compose.yaml
  • docs/ai-workspace/configuration.md
  • docs/ai-workspace/features/secrets-management.md
  • platform-api/README.md
  • platform-api/config/config.go
  • platform-api/config/config.toml
  • platform-api/config/config_test.go
  • platform-api/go.mod
  • platform-api/internal/server/server.go
  • portals/ai-workspace/.env.example
  • portals/ai-workspace/configs/config-platform-api-template.toml
  • portals/ai-workspace/configs/config-platform-api.toml
  • portals/ai-workspace/docker-compose.yaml
  • portals/developer-portal/.env.example
  • portals/developer-portal/configs/config-platform-api.toml.example
  • portals/developer-portal/distribution/.env.example
  • portals/developer-portal/distribution/docker-compose.yaml
  • portals/developer-portal/docker-compose.platform-api.yaml
  • portals/developer-portal/docker-compose.yaml
  • portals/developer-portal/it/docker-compose.test.postgres.yaml
  • portals/developer-portal/it/docker-compose.test.yaml
  • tests/integration-e2e/docker-compose.sqlite.yaml
  • tests/integration-e2e/docker-compose.sqlserver.yaml
  • tests/integration-e2e/docker-compose.yaml
💤 Files with no reviewable changes (3)
  • portals/developer-portal/docker-compose.platform-api.yaml
  • portals/developer-portal/distribution/docker-compose.yaml
  • portals/developer-portal/docker-compose.yaml

Comment thread docs/ai-workspace/features/secrets-management.md
Comment thread portals/ai-workspace/.env.example Outdated
@github-actions

Copy link
Copy Markdown
Contributor

Dependency Validation Results

Dependency name: github.com/knadh/koanf/providers/confmap
Version: v1.0.0
Allowed range: >=v1.0.0
Approved: ✅ Yes

⚠️ Please verify the scope of the dependencies usage is necessary

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/ai-workspace/configuration.md`:
- Around line 100-101: Update the inline code formatting in the configuration
instructions so the `docker compose --env-file keys.env up` command has a
closing backtick before the alternative sentence. Keep the `{{ file }}` inline
code span separate and preserve the surrounding wording.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 0bb7ab30-345a-4971-af12-cb01ffc92529

📥 Commits

Reviewing files that changed from the base of the PR and between e3e2d61 and 5747829.

📒 Files selected for processing (12)
  • distribution/all-in-one/docker-compose.yaml
  • docs/ai-workspace/configuration.md
  • docs/ai-workspace/features/secrets-management.md
  • platform-api/README.md
  • platform-api/config/config.toml
  • portals/ai-workspace/configs/config-platform-api-template.toml
  • portals/ai-workspace/configs/config-platform-api.toml
  • portals/ai-workspace/docker-compose.yaml
  • portals/developer-portal/configs/config-platform-api.toml.example
  • portals/developer-portal/distribution/docker-compose.yaml
  • portals/developer-portal/docker-compose.platform-api.yaml
  • portals/developer-portal/docker-compose.yaml
🚧 Files skipped from review as they are similar to previous changes (6)
  • portals/ai-workspace/configs/config-platform-api.toml
  • portals/developer-portal/docker-compose.yaml
  • portals/ai-workspace/configs/config-platform-api-template.toml
  • distribution/all-in-one/docker-compose.yaml
  • platform-api/README.md
  • portals/ai-workspace/docker-compose.yaml

Comment thread docs/ai-workspace/configuration.md Outdated
@thivindu
thivindu force-pushed the config-interpolation branch from 5747829 to 7a57fd3 Compare July 13, 2026 05:20
coderabbitai[bot]
coderabbitai Bot previously approved these changes Jul 13, 2026
thivindu and others added 2 commits July 14, 2026 16:40
Port the shared config-interpolation mechanism drafted for the gateway
(PR wso2#2596) to the platform-api config loader. Any config field may now
pull its value from an environment variable ({{ env "NAME" }}) or an
allowlisted file ({{ file "/path" }}), resolved after the env+file merge
and before unmarshal. String leaves without a "{{" token pass through
unchanged, so existing token-free configs are unaffected.

- Vendor common/configinterpolate from wso2#2596 (branch conf-template). This
  is a temporary copy to be reconciled when wso2#2596 merges to main.
- Wire an interpolate() step into config.LoadConfig, mirroring the
  gateway-controller. File reads are restricted to a per-component
  allowlist (/etc/platform-api, /secrets/platform-api), overridable via
  the shared APIP_CONFIG_FILE_SOURCE_ALLOWLIST env var. Fails closed on a
  missing required env var or a disallowed/missing/oversize file; resolved
  values are never logged (only reference counts at info level).
- Add github.com/knadh/koanf/providers/confmap for the reload step.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Dependency Validation Results

Dependency name: github.com/knadh/koanf/providers/confmap
Version: v1.0.0
Allowed range: >=v1.0.0
Approved: ✅ Yes

⚠️ Please verify the scope of the dependencies usage is necessary

1 similar comment
@github-actions

Copy link
Copy Markdown
Contributor

Dependency Validation Results

Dependency name: github.com/knadh/koanf/providers/confmap
Version: v1.0.0
Allowed range: >=v1.0.0
Approved: ✅ Yes

⚠️ Please verify the scope of the dependencies usage is necessary

@Thushani-Jayasekera

Copy link
Copy Markdown
Contributor

@thivindu let's fix the minor comments in a separate PR

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
platform-api/internal/integration/harness_test.go (1)

51-63: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Use a wrapped function to ensure cleanup with defer.

If m.Run() or file writing panics, os.Remove(f.Name()) will not execute. Using an anonymous function allows defer to run reliably before os.Exit() is called, which aligns with best practices for temporary file cleanup.

♻️ Proposed refactor
-	f, err := os.CreateTemp("", "it-config-*.toml")
-	if err != nil {
-		panic(fmt.Sprintf("integration harness: create temp config: %v", err))
-	}
-	if _, err := fmt.Fprintf(f, "encryption_key = %q\n\n[auth.jwt]\nenabled = true\nsecret_key = %q\n", testKey, testKey); err != nil {
-		panic(fmt.Sprintf("integration harness: write temp config: %v", err))
-	}
-	_ = f.Close()
-	config.SetConfigPath(f.Name())
-
-	code := m.Run()
-	_ = os.Remove(f.Name())
-	os.Exit(code)
+	code := func() int {
+		f, err := os.CreateTemp("", "it-config-*.toml")
+		if err != nil {
+			panic(fmt.Sprintf("integration harness: create temp config: %v", err))
+		}
+		defer os.Remove(f.Name())
+
+		if _, err := fmt.Fprintf(f, "encryption_key = %q\n\n[auth.jwt]\nenabled = true\nsecret_key = %q\n", testKey, testKey); err != nil {
+			f.Close()
+			panic(fmt.Sprintf("integration harness: write temp config: %v", err))
+		}
+		_ = f.Close()
+		config.SetConfigPath(f.Name())
+
+		return m.Run()
+	}()
+	os.Exit(code)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@platform-api/internal/integration/harness_test.go` around lines 51 - 63, Wrap
the temporary-config setup, test execution, and cleanup in an anonymous function
so cleanup is registered with defer and runs when file writing or m.Run panics.
Update the flow around m.Run and os.Remove, preserve the returned exit code, and
call os.Exit only after the function completes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@platform-api/internal/integration/harness_test.go`:
- Around line 51-63: Wrap the temporary-config setup, test execution, and
cleanup in an anonymous function so cleanup is registered with defer and runs
when file writing or m.Run panics. Update the flow around m.Run and os.Remove,
preserve the returned exit code, and call os.Exit only after the function
completes.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f7e46f96-1430-4339-b549-8c7ea7003ba3

📥 Commits

Reviewing files that changed from the base of the PR and between b8b9fad and 7b4bd7c.

⛔ Files ignored due to path filters (1)
  • platform-api/go.sum is excluded by !**/*.sum
📒 Files selected for processing (34)
  • .github/workflows/ai-workspace-pr-check.yml
  • distribution/all-in-one/docker-compose.yaml
  • docs/ai-workspace/configuration.md
  • docs/ai-workspace/features/secrets-management.md
  • gateway/examples/llm-provider.yaml
  • platform-api/README.md
  • platform-api/config/config.go
  • platform-api/config/config.toml
  • platform-api/config/config_test.go
  • platform-api/go.mod
  • platform-api/internal/integration/harness_test.go
  • platform-api/internal/repository/api_deployments_test.go
  • platform-api/internal/repository/subscription_repository.go
  • platform-api/internal/server/server.go
  • platform-api/plugins/eventgateway/service/websub_api_hmac_secret.go
  • portals/ai-workspace/README.md
  • portals/ai-workspace/configs/config-platform-api-template.toml
  • portals/ai-workspace/configs/config-platform-api.toml
  • portals/ai-workspace/docker-compose.yaml
  • portals/ai-workspace/production/README.md
  • portals/developer-portal/README.md
  • portals/developer-portal/configs/config-platform-api.toml.example
  • portals/developer-portal/distribution/docker-compose.yaml
  • portals/developer-portal/docker-compose.platform-api.yaml
  • portals/developer-portal/docker-compose.yaml
  • portals/developer-portal/docs/administer/manage-organizations.md
  • portals/developer-portal/it/configs/config-platform-api-it.toml
  • portals/developer-portal/it/docker-compose.test.postgres.yaml
  • portals/developer-portal/it/docker-compose.test.yaml
  • tests/integration-e2e/README.md
  • tests/integration-e2e/docker-compose.sqlite.yaml
  • tests/integration-e2e/docker-compose.sqlserver.yaml
  • tests/integration-e2e/docker-compose.yaml
  • tests/integration-e2e/platform-api-config.toml
🚧 Files skipped from review as they are similar to previous changes (24)
  • .github/workflows/ai-workspace-pr-check.yml
  • portals/ai-workspace/production/README.md
  • gateway/examples/llm-provider.yaml
  • platform-api/plugins/eventgateway/service/websub_api_hmac_secret.go
  • tests/integration-e2e/docker-compose.sqlite.yaml
  • platform-api/config/config.toml
  • tests/integration-e2e/README.md
  • portals/developer-portal/it/docker-compose.test.yaml
  • portals/developer-portal/it/docker-compose.test.postgres.yaml
  • portals/developer-portal/docs/administer/manage-organizations.md
  • docs/ai-workspace/configuration.md
  • platform-api/internal/repository/subscription_repository.go
  • portals/developer-portal/README.md
  • tests/integration-e2e/docker-compose.sqlserver.yaml
  • distribution/all-in-one/docker-compose.yaml
  • portals/developer-portal/configs/config-platform-api.toml.example
  • portals/ai-workspace/configs/config-platform-api-template.toml
  • portals/developer-portal/docker-compose.platform-api.yaml
  • portals/developer-portal/distribution/docker-compose.yaml
  • platform-api/config/config_test.go
  • portals/ai-workspace/configs/config-platform-api.toml
  • platform-api/internal/repository/api_deployments_test.go
  • portals/ai-workspace/docker-compose.yaml
  • platform-api/config/config.go

Krishanx92
Krishanx92 previously approved these changes Jul 14, 2026
@thivindu
thivindu dismissed stale reviews from Krishanx92 and Thushani-Jayasekera via 57c8101 July 14, 2026 15:44
@thivindu
thivindu force-pushed the config-interpolation branch from 7b4bd7c to 57c8101 Compare July 14, 2026 15:44
@github-actions

Copy link
Copy Markdown
Contributor

Dependency Validation Results

Dependency name: github.com/knadh/koanf/providers/confmap
Version: v1.0.0
Allowed range: >=v1.0.0
Approved: ✅ Yes

⚠️ Please verify the scope of the dependencies usage is necessary

@thivindu
thivindu merged commit fbf7fd5 into wso2:main Jul 15, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants