Skip to content

v0.1.5

Latest
Latest

Choose a tag to compare

View all tags
@ThomasK33 ThomasK33 released this 14 Oct 23:41
v0.1.5
57ee69a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

This release turns on two kinds: CoderAIGatewayKey and CoderMCPServer. The operator now creates, rotates and deletes Coder AI Gateway keys, and it manages MCP server configs that Coder chats use. It also cleans up better. A CoderControlPlane now waits for the keys, AI providers and MCP servers that use it while you delete them, so their deletes reach Coder. A cleanup record is a ConfigMap that the operator writes before each AI provider or key create. It lets the operator delete what a namespace deletion left in Coder, when a control plane on the same Coder database comes back. Read the upgrade notes before you upgrade. The first step needs an action from you.

Upgrade notes

  1. Apply the CRDs and RBAC before or together with the new image (#346, #374, #381). This release adds the CoderAIGatewayKey and CoderMCPServer CRDs, new ClusterRole rules, and two new objects in coder-system: the Role manager-role and the RoleBinding coder-k8s-cleanup-records. From a clone of v0.1.5, run:

    kubectl apply -f config/crd/bases/ -f config/rbac/

    Or apply dist/install.yaml from v0.1.5. The operator watches every kind that it reconciles, so the manager does not start without the two new CRDs. Without the Role and the RoleBinding, the operator cannot write cleanup records. Then it creates no AI provider and no key, and existing CoderAIProvider objects stop syncing. Check both objects:

    kubectl -n coder-system get role manager-role
kubectl -n coder-system get rolebinding coder-k8s-cleanup-records

    If you run the operator in another namespace, change the namespace of both objects.

  2. Delete the AI providers that v0.1.4 left in Coder by hand (#374, #375). In v0.1.4, a namespace deletion or a provider deleted after its control plane left the provider in Coder in some cases. v0.1.4 wrote no cleanup records, so this release cannot find those providers. Search the v0.1.4 operator logs for Releasing the finalizer without a delete in Coder: the provider can stay in the Coder database, which names the provider ID. Then open AI settings > Providers (/ai/settings/providers) as an owner, and delete each provider that no CoderAIProvider object uses. A CoderAIProvider that still exists gets its record at its first reconcile that reaches Coder.

  3. Do not give other users write access to ConfigMaps in coder-system (#374). A cleanup record names the Coder provider ID or the key name prefix that a sweep deletes. A user who can write these ConfigMaps can point a sweep at other Coder resources. Find the records with kubectl -n coder-system get configmaps -l coder.com/cleanup-record.

  4. Expect slower control plane deletes (#362, #384). A CoderControlPlane delete can now take up to about 15 minutes while its keys, providers and MCP servers delete. Before, it went at once. See "Changed behavior".

  5. Delete keys, providers and MCP servers before their namespace. This advice does not change. A namespace deletion removes Coder's pods at once, so the operator cannot delete these objects in Coder at that time.

New

  • CoderAIGatewayKey (#341, #346, #370). This is the first release that runs this kind. One object keeps one Coder AI Gateway key in a Secret that the object owns. The operator creates the key, rotates it (every 30 days by default), deletes replaced keys after a grace period (1 hour by default), and deletes every key of the object when you delete the object. The control plane must opt in with spec.aiGateway.allowKeys: true. Without it, the object stalls with NotAllowed. An object that never created a key makes no Coder call. You need a Coder Premium license with the AI Gateway enabled, and the operator user must be a site owner. A gateway key is a full-trust credential. Grant write access to coderaigatewaykeys, and read access to the output Secrets, only to principals that can hold that power. The guide is Manage AI Gateway keys.
    • Key names. The operator names its keys ck8s-<UID>-<serial>, where <UID> is the object's metadata.uid without dashes. The operator and the sweep delete keys with this prefix. Do not give your own keys names that start this way.
    • Only Coder's own 404 proves that a key is gone. A 404 from a proxy makes the operator retry (#341).
  • CoderMCPServer (#330, #356, #360, #363, #366, #368, #381). This is the first release that runs this kind. One object manages one MCP server config in a Coder organization (AI settings > MCP servers). Coder chats then offer its tools to the model. The control plane must list the organizations in spec.mcpServers.organizations. Without it, the object stalls with MCPServersNotAllowed. You need Coder v2.37.2 or later, and the operator user must be a site owner. The guide is Manage MCP servers.
    • Kill switches. The operator disables the server in Coder when the credential Secret withdraws consent, or when the control plane withdraws its opt-in. A disable in the dashboard stalls the object with DisabledExternally. A spec edit enables the server again.
    • Access. spec.access lists groups. When a listed group does not exist in Coder, the operator removes the access of unlisted groups and of users, and adds no group.
    • Deletion. Deleting the object deletes the config in Coder.
  • Opt-in status admission policy for CoderMCPServer (#377). config/admission/status-operator-only.yaml holds a ValidatingAdmissionPolicy and its binding. Only the operator ServiceAccount can then write codermcpservers/status, and cluster admins are denied too. The install bundle does not apply it. Apply it with kubectl apply -f config/admission/. It needs Kubernetes v1.30 or later, where ValidatingAdmissionPolicy is GA. It hard-codes the user system:serviceaccount:coder-system:coder-k8s. If the operator runs as another ServiceAccount or in another namespace, edit the expression first, or the operator cannot write status.
  • GitOps health rules for CoderMCPServer (#383). config/gitops/argocd-health-codermcpserver.lua is for Argo CD. config/gitops/flux-healthcheck-codermcpserver.yaml is for a Flux v2.5+ Kustomization with spec.wait: true.
  • Cleanup records and the sweep (#374, #375, #385, #386). Before each AI provider or key create, the operator writes a ConfigMap ck8s-cleanup-<object UID> in its own namespace (coder-system). The label coder.com/cleanup-record holds the kind (CoderAIProvider or CoderAIGatewayKey). The record holds no secret. It names the object, the Coder deployment ID from /api/v2/buildinfo, and the provider ID or the key name prefix. If the operator cannot write the record, it sends no create.
    • When an object goes without a delete in Coder, for example in a namespace deletion, the record stays. The leftovers stay valid in Coder until the sweep deletes them.
    • The sweep runs when a ready CoderControlPlane in the same namespace reaches the same Coder deployment again, which means the same Coder database. It matches by deployment ID, never by URL or name. The operator user of that control plane must be a site owner.
    • A new Coder database behind the same URL is never swept. A control plane in another namespace never sweeps a record either.
    • The counter coder_k8s_cleanup_records_swept_total{kind} counts swept records. When no ready control plane reaches a record's deployment, the gauge coder_k8s_cleanup_records_unmatched{kind} counts the record. The operator also writes a NoMatchingDeployment Warning Event on the ConfigMap every 10 minutes.
  • New metric coder_k8s_cleanup_released_without_delete_total{kind} (#362). It counts each CoderAIGatewayKey or CoderAIProvider finalizer that went without a delete in Coder.

Changed behavior

  • A deleted CoderControlPlane waits for the objects that use it (#362, #384, #385). This applies while CoderAIGatewayKey, CoderAIProvider or CoderMCPServer objects that reference the control plane are being deleted.
    • The control plane keeps its finalizer for up to 10 minutes, so Coder keeps running. The objects delete in Coder as usual, with retries.
    • After the 10 minutes, the remaining objects get at most 5 more minutes for one last delete attempt. So a control plane delete can take up to about 15 minutes. Before, it went at once.
    • A key or MCP server delete attempt that runs during the wait stops at the end of the wait.
    • The operator removes the control plane's workspace RBAC first, so Coder loses its Kubernetes access at once.
    • An object that you do not delete holds the control plane for only 30 seconds. Delete the objects together with the control plane, or within 30 seconds.
    • The control plane does not wait when its namespace is being deleted, or when you delete it in the foreground.
    • The operator log shows Waiting for the objects that use this control plane and their names.
  • A change of Coder database stalls keys and providers (#374, #385). When a control plane reaches another Coder deployment, for example after its database Secret changed, the object stalls with CoderDeploymentChanged and creates nothing in the new deployment. A delete of the object then removes the finalizer without a delete in Coder. Delete the old keys or providers in the old deployment by hand, then delete the ConfigMap.
  • Each AI provider or key create sends one more request (#374, #385). The operator reads GET /api/v2/buildinfo before the create. If that read fails, the create waits and retries after 30 seconds.
  • New RBAC rules (#346, #374, #375, #381). The ClusterRole adds rules for the two new kinds and their status and finalizers. It also adds create and patch on events.k8s.io events. The Role manager-role gives the operator ConfigMap write access in coder-system only. The ClusterRole keeps ConfigMap read only.

Fixes

  • CoderTemplateTest workspace lookups now trust only Coder's own 404. Before, a proxy's 404 page or Route not found. counted as "no workspace". In some cases the test then failed with CreateOutcomeUnknown and left a ktt- workspace in Coder, or sent a create without a real name check. Now the test waits with CoderUnavailable (#352).
  • Before, a deleted CoderMCPServer left its config enabled in Coder in some cases, with its stored credential, when its control plane was deleted too. Now the control plane waits for it (#384, fixes #382).
  • Internal: CoderWorkspacePool work that still does not run (its CRDs stay in config/crd/dormant/), Kind E2E phases for CoderAIGatewayKey and CoderMCPServer, a Terraform provider mirror for the Kind E2E, RBAC pin tests, and test fixes (#343, #344, #357, #359, #361, #364, #367, #370, #371, #373, #376, #379, #380).

Known limits

  • A namespace deletion still leaves resources in Coder (#353). Leftover keys and providers stay valid in Coder until a ready control plane comes back in the same namespace on the same Coder database. If that never happens, delete them by hand, then delete the ConfigMap. The NoMatchingDeployment Event names the object.
  • MCP servers have no cleanup records (#353). A namespace deletion can leave an MCP server config in Coder, and no sweep deletes it. Delete CoderMCPServer objects before their namespace. MCP server releases without a delete do not count in coder_k8s_cleanup_released_without_delete_total.
  • Coder IDs in status have no ownership proof (#372). A subject who can write an object's status can point the object at another Coder resource. Do not grant write access to the status of these kinds. For CoderMCPServer, apply the opt-in admission policy.
  • A CoderMCPServer with only a create marker can wait forever (#378). This happens when its Coder organization is deleted while its control plane still runs.

Install

kubectl apply -f https://raw.githubusercontent.com/coder/coder-k8s/v0.1.5/dist/install.yaml
kubectl -n coder-system set image deployment/coder-k8s coder-k8s=ghcr.io/coder/coder-k8s:0.1.5

dist/install.yaml uses ghcr.io/coder/coder-k8s:latest. The second command pins the image to ghcr.io/coder/coder-k8s:0.1.5.

Assets 5
Loading