Secret Management Feature for Platform API

[npamudika]

Secrets Management — Platform API

Platform administrators need to configure sensitive credentials across multiple domains — primarily LLM provider API keys — when creating and managing LLM providers through the platform. Currently, there is no mechanism to handle these securely. The problem has two distinct dimensions:

1. No secure storage for secrets - The platform has no dedicated secret storage layer. Sensitive values such as LLM provider API keys (e.g., sk-xxx for OpenAI) have nowhere safe to live. If they were stored today, they would land as plain TEXT in the database — the same way client_cert and client_key are already stored in api_mtls_config and backend_endpoints. A database dump would expose all credentials in full.

2. No safe way to reference secrets in deployment artifacts - When an LLM provider is configured (e.g., auth.header.value: "Bearer sk-xxx"), the raw API key would end up embedded inside the deployment artifact pushed to the gateway controller. This means:

   • Plain-text secrets persist in configuration files and database records at the gateway side
   • Rotating a key requires tracking down every artifact it appears in
   • Any log, debug output, or audit trail touching that configuration leaks the credential

The core gap: there is no placeholder/reference mechanism that keeps the credential out of deployment artifacts while still making it available at runtime. This will be handled as explained below. The proposed architecture would be as,

1. Overview

The Platform API provides administrator-only CRUD endpoints for managing encrypted secrets, scoped to organizations (and optionally projects). Secrets are encrypted at rest and referenced in resource definitions — LLM providers, API backend configs, mTLS configs — via a placeholder syntax. The raw secret value never appears in any stored artifact or API response after the initial creation response.

Placeholder syntax used in resource definitions:

{{ secret "wso2-openai-key" }}

Example — LLM provider stored configuration:

spec:
  upstream:
    auth:
      type: api-key
      header: Authorization
      value: 'Bearer {{ secret "wso2-openai-key" }}'

---

2. API Endpoints

All endpoints are administrator-only and JWT-authenticated. Organization scope is derived from the JWT token — the same pattern used by the existing gateway and LLM provider endpoints.

Method	Path	Description
POST	/api/v1/secrets	Create a secret; value returned once only
GET	/api/v1/secrets	List secrets (metadata only, no values)
GET	/api/v1/secrets/:id	Get a secret by ID (metadata only)
PUT	/api/v1/secrets/:id	Rotate / update the secret value
DELETE	/api/v1/secrets/:id	Delete — blocked (409) if referenced elsewhere

2.1 Create Secret

Request

POST /api/v1/secrets
Authorization: Bearer <JWT>
Content-Type: application/json

{
  "name": "wso2-openai-key",
  "displayName": "WSO2 OpenAI API Key",
  "description": "API key for WSO2 OpenAI integration",
  "type": "API_KEY",
  "value": "sk-xxx",
  "projectId": "proj-uuid-optional"
}

projectId is optional. When omitted, the secret is org-wide. When set, the secret is scoped to that project and the same name can coexist at org-level and project-level independently.

Response — 201 Created (value is returned exactly once)

{
  "id": "a1b2c3d4-...",
  "name": "wso2-openai-key",
  "displayName": "WSO2 OpenAI API Key",
  "description": "API key for WSO2 OpenAI integration",
  "type": "API_KEY",
  "provider": "IN_HOUSE",
  "value": "sk-xxx",
  "createdAt": "2026-01-12T10:00:00Z",
  "createdBy": "admin@wso2.com"
}

After this response, the value field is never included in any response again.

2.2 List Secrets

Response — 200 OK (metadata only)

{
  "list": [
    {
      "id": "a1b2c3d4-...",
      "name": "wso2-openai-key",
      "displayName": "WSO2 OpenAI API Key",
      "type": "API_KEY",
      "provider": "IN_HOUSE",
      "environment": "__ALL_ENV",
      "projectId": null,
      "createdAt": "2026-01-12T10:00:00Z",
      "updatedAt": "2026-01-12T10:00:00Z"
    }
  ],
  "count": 1
}

2.3 Get Secret by ID

Same shape as a single list item — no value field.

2.4 Rotate Secret (PUT)

Re-encrypts and stores the new value. The name and id remain unchanged, so all {{ secret "..." }} placeholders across resources remain valid without modification.

Request

{
  "value": "sk-new-value",
  "displayName": "WSO2 OpenAI API Key (rotated)"
}

Response — 200 OK (new value returned once)

{
  "id": "a1b2c3d4-...",
  "name": "wso2-openai-key",
  "value": "sk-new-value",
  "updatedAt": "2026-06-01T08:00:00Z"
}

2.5 Delete Secret

Before deletion, the service queries all tables that can reference a secret by placeholder:

• llm_providers (configuration JSON)
• api_backend_endpoints
• api_mtls_config

If any reference exists, 409 Conflict is returned:

{
  "error": "secret is referenced by active resources",
  "references": [
    { "type": "llm_provider", "id": "...", "name": "OpenAI Provider" }
  ]
}

A soft alternative is available: set status: DEPRECATED on the secret instead of deleting it. The GW controller will continue to resolve it but new resources cannot reference a deprecated secret.

---

3. Storage and Encryption

3.1 Encryption

Property	Value
Algorithm	AES-GCM-256 (reuses existing EncryptSubscriptionToken / DecryptSubscriptionToken pattern in utils/subscription_token_crypto.go)
Nonce	12-byte random, prepended to ciphertext before storage
Key derivation	DeriveEncryptionKey (existing utility) — 32-byte key from 64-char hex or base64
Key source	PLATFORM_SECRET_ENCRYPTION_KEY env var (consistent with AUTH_JWT_SECRET_KEY)
Fallback	Startup fails if key is absent or invalid — no silent plaintext fallback

3.2 Vault Provider Interface

Encryption is behind a pluggable interface to allow future KMS backends without changing the service layer:

// SecretVault is the pluggable encryption backend.
type SecretVault interface {
    // Encrypt plaintext and return an opaque ciphertext blob.
    Encrypt(ctx context.Context, plaintext string) ([]byte, error)
    // Decrypt returns the original plaintext from a ciphertext blob.
    Decrypt(ctx context.Context, ciphertext []byte) (string, error)
    // ProviderName identifies the backend (e.g. "IN_HOUSE", "AWS_KMS", "VAULT").
    ProviderName() string
}

Implementations to provide at launch:

Provider	provider column value	Notes
In-house AES-GCM	IN_HOUSE	Default. Uses PLATFORM_SECRET_ENCRYPTION_KEY
AWS KMS	AWS_KMS	Research phase — post-MVP
HashiCorp Vault	HASHICORP_VAULT	Research phase — post-MVP

The provider column persists which vault encrypted the secret so rotation and decryption use the correct backend even after a backend migration.

Research items for AWS KMS and HashiCorp Vault:

• Envelope encryption pattern (data key per secret vs. direct KMS call)
• KMS key rotation policy and how it interacts with re-encryption on PUT
• Auth credential injection (IAM role for KMS; AppRole/token for Vault)
• Latency implications on GW controller startup (bulk secret resolution)

3.3 Hashing and Change Detection

Every secret stores a hash of its plaintext value (SHA-256 hex). The GW controller uses this hash to determine whether a secret has changed since its last sync, avoiding unnecessary re-deployment.

hash = hex(SHA-256(plaintext_value))

Hash versioning: The hash column stores a prefixed value to allow algorithm changes:

sha256:e3b0c44298fc1c149afb...

The prefix lets the GW controller (and future code) detect the hashing algorithm in use and handle multi-algorithm transitions during rollouts.

GW controller flow:

1. On startup / reconcile loop, fetch /api/v1/secrets (metadata + hash, no value).
2. Compare each secret's hash against the locally cached hash.
3. Only fetch the decrypted value (via an internal GW-to-platform call) if the hash differs.
4. This minimises decryption calls and network round-trips.

3.4 Database Schema

CREATE TABLE IF NOT EXISTS secrets (
    organization_id TEXT      NOT NULL,
    project_id      TEXT,                          -- NULL = org-wide; set = project-scoped
    handle          TEXT      NOT NULL,             -- identifier in {{ secret "handle" }}
    display_name    TEXT      NOT NULL,
    description     TEXT,
    ciphertext      BLOB      NOT NULL,             -- AES-GCM-256 nonce||ciphertext
    hash            TEXT      NOT NULL,             -- prefixed SHA-256 of plaintext (e.g. "sha256:abc...")
    type            TEXT      DEFAULT 'GENERIC',    -- API_KEY | CERTIFICATE | PRIVATE_KEY | GENERIC
    provider        TEXT      DEFAULT 'IN_HOUSE',   -- IN_HOUSE | AWS_KMS | HASHICORP_VAULT
    status          TEXT      DEFAULT 'ACTIVE',     -- ACTIVE | DEPRECATED
    environment     TEXT      DEFAULT '__ALL_ENV',
    created_at      TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    created_by      TEXT,
    updated_at      TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_by      TEXT,
    PRIMARY KEY (organization_id, handle),          -- name uniqueness within org (org-wide)
    FOREIGN KEY (organization_id) REFERENCES organizations(uuid) ON DELETE CASCADE,
    FOREIGN KEY (project_id)      REFERENCES projects(uuid)      ON DELETE CASCADE
);

-- Efficient polling queries from GW controller
CREATE INDEX IF NOT EXISTS idx_secrets_updated_at ON secrets(updated_at);

-- Project-scoped lookups
CREATE INDEX IF NOT EXISTS idx_secrets_project_id ON secrets(project_id);

Note on uniqueness: The PRIMARY KEY (organization_id, handle) enforces org-level uniqueness. Project-scoped secrets use the same handle but a different scope, so an additional unique constraint covers this:

CREATE UNIQUE INDEX IF NOT EXISTS idx_secrets_scope_handle
    ON secrets(organization_id, COALESCE(project_id, ''), handle);

This allows the same secret name (e.g. openai-key) to coexist as org-wide and within a specific project simultaneously.

---

4. Scope Resolution

4.1 Access Control

All queries include WHERE organization_id = ? extracted from the JWT token — the same pattern as gateways and LLM providers. Cross-organization requests return 404 Not Found (not 403) to avoid leaking existence.

4.2 Project vs Org Scope

A secret can be org-wide (project_id IS NULL) or project-scoped (project_id = <uuid>). Scope resolution for placeholder lookup follows a most-specific-wins rule:

1. Look for a secret matching (organization_id, project_id, handle).
2. If not found, fall back to (organization_id, NULL, handle) (org-wide).
3. If still not found, reject with 400 Bad Request at resource save time.

This allows teams to override org-level secrets per project without renaming placeholders.

4.3 Placeholder Validation at Resource Save

When creating or updating any resource that contains {{ secret "..." }} references (LLM providers, APIs, backend configs), the service:

1. Extracts all placeholder names from the configuration JSON/YAML.
2. For each name, checks that a secret exists in the same org (respecting scope resolution order).
3. Rejects the request with 400 Bad Request and the list of unresolvable placeholders if any are missing.

---

5. Key Behaviours

5.1 Write-once Value Semantics

The ciphertext column is never mapped to any GET or LIST response DTO. This is enforced at the DTO struct level (no value field on SecretResponse), not by convention. Only SecretCreateResponse and SecretRotateResponse carry the value field.

5.2 Rotation

PUT /api/v1/secrets/:id re-encrypts and stores the new value. The handle and id are immutable, so no resource definition changes are required after rotation. The hash is also updated, triggering GW controller reconciliation on next poll.

5.3 Delete Protection

See §2.5. Reference scanning covers all tables that store configuration blobs. The implementation uses SQLite JSON functions to query for placeholder patterns within configuration columns.

5.4 Soft Delete / Deprecation

status: DEPRECATED allows an admin to mark a secret as retired without breaking existing GW deployments. The GW controller continues resolving deprecated secrets but the platform API rejects new resource definitions that reference a deprecated secret.

---

6. Gateway Controller Integration

6.1 Background — Compatibility

GW Release	Behaviour
1.1.0 (current)	No encryption support. Placeholders passed as-is if platform API not present.
New release	Must resolve {{ secret "..." }} placeholders to plaintext before generating Envoy/Nginx config. Encrypts where applicable.

The new GW release should be backward-compatible: if a resource definition contains no {{ secret "..." }} placeholders, it behaves identically to 1.1.0.

6.2 Secret Resolution Flow at Deploy Time

GW Controller reconcile loop
│
├── Fetch resource definitions (APIs, LLM providers, backends) from Platform API
├── Extract all {{ secret "..." }} placeholders from config blobs
├── Call GET /api/v1/secrets (metadata + hash) — org-scoped
├── For each referenced secret:
│   ├── If hash unchanged since last sync → use cached plaintext
│   └── If hash changed or not cached → call internal endpoint to get plaintext
│                                        (authenticated with GW service account JWT)
├── Substitute placeholders in config blobs
└── Generate Envoy/xDS config — plaintext values never written to disk

Internal decryption endpoint (not in the public API surface — GW-to-platform internal):

GET /internal/v1/secrets/:id/value

Authenticated via GW service account JWT. Returns decrypted plaintext. Audited separately.

6.3 Hash-driven Incremental Updates

The GW controller polls /api/v1/secrets?updatedAfter=<timestamp>. The platform API returns only secrets whose updated_at > updatedAfter. If no secrets changed, no decryption calls are made.

---

7. Configuration

Env Var	Description	Required
PLATFORM_SECRET_ENCRYPTION_KEY	32-byte AES-256 key (64 hex chars or base64)	Yes (startup fails if absent)
PLATFORM_SECRET_VAULT_PROVIDER	IN_HOUSE | AWS_KMS | HASHICORP_VAULT	No (default: IN_HOUSE)

For AWS KMS and HashiCorp Vault, additional provider-specific env vars will be specified in the respective research deliverables.

---

8. Error Responses

Scenario	HTTP Status	Detail
Secret name already exists in scope	409	"secret with this name already exists in scope"
Unresolvable placeholder at resource save	400	List of missing secret names
Delete blocked by active references	409	List of referencing resources
Cross-org access attempt	404	Generic not-found (no existence leak)
Invalid encryption key at startup	Fatal	Server refuses to start

---

9. AI Workspace — Auto-Encryption on LLM Provider Creation

9.1 Overview

When an LLM provider is added or updated via the AI Workspace UI, any plain-text API key value in the upstream auth configuration is automatically encrypted via the Platform API secrets endpoint before the provider is persisted. The raw API key never reaches the stored LLM provider artifact.

9.2 Flow

User fills in API key in the Add Provider form
  ↓
handleCreateProvider() in ServiceProviderNew.tsx
  → Constructs upstream.main.auth.value with plain-text key
  ↓
createProvider(payload) in LLMProvidersContext.tsx
  │
  ├── Detect plain-text credential
  │   (skip if value already contains {{ secret " — idempotent guard)
  │
  ├── Generate secret handle
  │   generateSecretHandle(providerId, 'api-key')
  │   → e.g. "wso2-openai-provider-api-key"
  │
  ├── POST /secrets  ← Platform API
  │   {
  │     name:        "wso2-openai-provider-api-key",
  │     displayName: "<Provider Name> API Key",
  │     value:       "sk-xxx",           ← plain-text sent here only
  │     type:        "API_KEY"
  │   }
  │   Response includes value once — discarded after substitution
  │
  ├── Substitute placeholder in payload
  │   upstream.main.auth.value = '{{ secret "wso2-openai-provider-api-key" }}'
  │
  └── POST /llm-providers  ← Platform API
      upstream.main.auth.value now contains only the placeholder

The same flow applies to updateProvider (PUT) — if the user provides a new plain-text key the same handle is used, effectively rotating the secret.

9.3 Secret Handle Naming Convention

Secret handles are derived deterministically from the provider ID:

<providerId>-api-key

Examples:

Provider ID	Secret Handle
wso2-openai-provider	wso2-openai-provider-api-key
anthropic-claude	anthropic-claude-api-key
azure-openai-east	azure-openai-east-api-key

The handle is lowercased and non-alphanumeric characters are collapsed to hyphens. This makes the secret discoverable by handle if an admin needs to rotate it independently via the secrets API.

9.4 Idempotency Guard

Before calling the secrets API, the context checks whether the auth value already contains a {{ secret " substring. If it does (e.g. the provider config is being re-submitted with the existing placeholder), the secret creation step is skipped entirely and the payload is passed through unchanged. This prevents duplicate secrets being created on re-save.

9.5 Error Behaviour

If the POST /secrets call fails, the entire provider creation is aborted — no partial state is written. The error propagates to the existing snackbar error handler in ServiceProviderNew.tsx, which displays the Platform API error message to the user.

---

For more details refer https://docs.google.com/document/d/1I-ls5QUvDpGLmYPe39ai_i3AEL_awv8vQbi83jOnoRg/edit?usp=sharing

[RakhithaRR]

We support secured MCP Proxies through the AI Workspace. The scope of this feature should extend to cover the MCP related scenarios as well.

[renuka-fernando]

I have some problems with the design.

1. Do we really need type in the request body? Maybe we can introduce this type later, but I feel we don't need it now.
2. The existing behavior of the gateway is not to apply the secret immediately, but to wait for a redeploy or a gateway restart (gateway controller), which is not good. So I think we can apply the secret immediately when it is updated. So, even from the Platform API secret update event, we can just trigger the same behavior as the gateway controller does.