Repository navigation
KubeBolt 2.7.0 — an out-of-date agent says so, a cloud operation stops without a release, and OpenCost's cost arrives whole
Drop-in over 2.6.7. No schema change and no migration. The agent moves to
1.4.10: OpenCost's cost arrives whole and once, and the agent's default
limits rise to 256Mi / 500m. Its chart has one new rule: it refuses two
sources of cost (see Upgrading).
An out-of-date agent says so, with the command
KubeBolt now says when a cluster's agent is out of date, why, and the command
that upgrades it. It never upgrades the agent itself: the agent is your Helm
release, often under GitOps, and upgrading it from KubeBolt would take write
access to its DaemonSet.
- A cluster is judged by its oldest live agent: current, update
available (a newer release exists, nothing to fix) or update recommended
(a newer release fixes something it gets wrong). Today that is any agent
below 1.4.9: below 1.4.8 each pod's network is stored twice (up to three
times the series), and below 1.4.9, with Hubble flows on, flow keys are never
forgotten. - Clusters puts a chip on the cluster's card, amber when an upgrade is
recommended and neutral when one is available. It opens the reasons and the
helm upgrade … --reset-then-reuse-valuescommand to copy, with what to
check before running it:- kubectl's current context must be that cluster (or add
--kube-context); - another release name or namespace changes the command;
- a release installed with a full copy of the chart's values file keeps the
old defaults as settings (helm get valuesshows them); - with ArgoCD or Flux, set the chart version in the Application or
HelmRelease instead.
- kubectl's current context must be that cluster (or add
- Fleet tags a cluster whose upgrade is recommended, and Home lists it
under Needs your attention. - Kobi answers whether a cluster's agent is out of date, and gives the same
command. It never runs it. - Agents before 1.4.4 all report
1.0.0. They show as before 1.4.4, never
as 1.0.0; Applications shows the chart actually installed.
Stop a cloud operation without a release
What Kobi's cloud connectors may run is a list in the code, so stopping one
operation (a read later found to expose data, or one Kobi keeps reaching for)
took a release. Administration › AI › Blocked operations is the emergency
switch:
- One list per connector: for AWS
service.operation(s3.getobject), for
Azure the command name, for GCP the tool name.s3.*blocks a whole service
or group, and*pauses the connector. - It applies from the next call. A blocked call is never sent: Kobi's card
reads Not sent, and the access trail records it with resultblocked. - An entry that would block nothing (an AWS operation without its service, a
connector this install does not have) is refused when you save it, with the
reason under the connector. - It fails closed: when the list cannot be read, every connector call is
refused. - Admin only, and audited with the other settings
(GET/PUT /admin/settings/blocked-operations).
An alert when Kobi keeps asking for what it may not do
Every call the connector gateway refuses is stopped and audited, but a run of
them reached nobody. Kobi asking again and again for a tool its set does not
list, or sending input the connector refuses, is the shape of a prompt
injection.
- New counter
kubebolt_kobi_mcp_refusals_total{consumer, connector, reason}: what the gateway refuses before the provider (tool_not_allowed,
input_refused). A provider's own denial is not counted. Thetenant_id
label is always empty in this edition. - New alert
KobiKeepsAskingForWhatItMayNotDoin thekubebolt-kobi
group: 5 or more refusals in an hour on one connector. It ships with the
other rules (metrics.alertRules.enabled). The series start at 0, so the
first burst alerts too.
A cloud call that did not read says why
- A call the per-minute budget held back reads as a limit, not a failure.
Kobi's cloud calls are held to 120 a minute, a quarter of them at once. A
call held back now shows amber, with a clock: Not read — cloud
call limit: 120/min, 30 at once. It counts among the reads that could not be
made, and a reopened conversation reads the same. Kobi is told the same
numbers. - Refused and held-back calls are not errors. They count as
not_sentand
rate_limitedinkubebolt_kobi_copilot_tool_results_total, so
Administration › AI › Health no longer shows them as failing tools. - A role STS will not let KubeBolt assume reads STS AssumeRole refused
KubeBolt (AccessDenied) on the card. The AWS SDK's text stays in the log,
and the call counts asunauthorized. - The access trail names the operation of a call that reached the provider
and failed, and of a call that was held back or blocked. - Kobi's AWS investigation procedure now comes with the first call that reaches
the cloud, not with one that was refused or held back.
OpenCost's cost arrives whole, and once
OpenCost's cost reached KubeBolt with pieces missing, sometimes in the wrong
namespace or counted twice, and with about as many unread series as read ones.
One list of 13 cost families is now what every path ships, asks for and keeps.
- Pieces were missing on two paths. Reading from your Prometheus
(promRead.cost) never asked for volume claims (pod_pvc_allocation), spot
nodes, load balancers or the four network egress costs. The agent's scrape
mode dropped volume, spot, load-balancer and egress cost in its keep regex. - Half of what arrived was read by nothing. The OpenCost exporter
forwarded everything OpenCost publishes. Agent 1.4.10 forwards the cost
families and OpenCost'skube_*only, and 2.7.0 drops the rest from older
agents at the door, counted as
kubebolt_agent_grpc_dropped_series_total{reason="opencost_unread"}. On a
test cluster, OpenCost's series went from 1,213 to about 500. - Every container's cost could land in OpenCost's namespace. A Prometheus
that scrapes OpenCost withouthonor_labels(a kube-prometheus
ServiceMonitor's default) puts the OpenCost pod'snamespace,pod,
containerandinstanceon every cost series and renames OpenCost's own to
exported_*. Read through promread, every container's cost was charged to
the namespace OpenCost runs in. promread now gives cost families their
exported_*labels back; the cluster and tenant labels it stamps are never
replaced. - Two sources doubled every total. With the exporter and
promRead.cost
both on, every cost family was stored twice, and the Cost page read $1,485 a
month for a test cluster that costs $743. The agent chart now refuses that
combination.
The API's drop works with any agent; the rest needs agent 1.4.10.
Smaller fixes
- The action history's class filter fills its page.
GET /admin/actions?class=…read the newestlimitrecords and then dropped the
other class, so?class=mutationbehind a busy hour of terminals and cloud
calls came back empty. The class now applies before the limit. A record
written before records carried a class reads as a mutation. - Each row of Home's Needs your attention says where it comes from: a
labelled icon, Cluster or Security, in a column of its own, so a mixed
list reads by kind before any row is read. - Cluster cards take their columns from the width the page has. Next to
Kobi's panel they ran past their edge.
Agent 1.4.10
- OpenCost: the exporter forwards only the 13 cost families and OpenCost's
kube_*; promread asks for all 13 (and no longer for the two GPU families
nothing reads); scrape mode keeps them; promread gives cost families
OpenCost's own labels back. See OpenCost's cost arrives whole, and once. - The chart refuses two sources of cost.
agent.promRead.cost.enabled
together with the OpenCost exporter (collectors.exporters.opencostor
opencost.enabled) now fails the render, next to the existingscrape+
promReadrule. Pick one:promRead.costreads OpenCost from your
Prometheus, the exporter reads OpenCost directly. - Default limits 256Mi / 500m (were 128Mi / 100m). Requests are unchanged,
so scheduling does not move. The agent relays the API server's responses for
a cluster connected through it and holds each one whole: at 128Mi it was
OOMKilled on a cluster with thousands of ReplicaSets, and at 100m a large
cluster's first connect was throttled. The promread Deployment keeps its own
limits. - A response too large for the agent's memory is an error, not an OOMKill.
One relayed body is capped at a quarter of the container's memory limit, and
never above the channel's 63 MiB. Past it, the agent answers with an error
that names the cap, the limit andresources.limits.memory. - Go 1.26.11 (CVE-2026-97054, Go standard library).
Upgrading
# The backend
helm upgrade kubebolt oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt \
--version 2.7.0 -n kubebolt --reset-then-reuse-values
# Then the agent, in every cluster that runs one (with that cluster's kubectl context)
helm upgrade kubebolt-agent oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt-agent \
--version 1.4.10 -n kubebolt-system --reset-then-reuse-valuesUse the namespaces you installed into: helm list -A | grep kubebolt shows
them. --reset-then-reuse-values needs Helm 3.14+; passing your values file
(-f) works too. Do not use --reuse-values: Helm keeps the previous chart's
values, defaults included, so the chart's pinned image tags stay old and the
agent keeps 128Mi / 100m and the scrape regex that drops volume and network
cost. Step by step, what to expect, rollback, and what each version asks of
you when you skip some:
Upgrade.
What this version may ask of you:
- OpenCost series outside the list that the exporter forwarded (the GPU
families, OpenCost's own HTTP metrics,pod_info, …) are no longer stored
from the backend's upgrade on, whatever the agent's version. A query of
your own that reads them comes back empty.
The agent's upgrade may ask:
- One source of cost. An agent release with both
agent.promRead.cost.enabledand the OpenCost exporter
(collectors.exporters.opencost, oropencost.enabledfor the bundled
OpenCost) failshelm upgradeto 1.4.10 with a message naming both. Turn one
off; until then the release stays on its previous version. - The agent's limits. The new defaults apply unless your values set
resources(a full copy of the chart's values file does). If you raised the
memory limit after 2.6.5, keep it. A ResourceQuota on limits in the agent's
namespace needs room for the higher limits. - Your own
scrape.metricRelabelConfigsreplaces the chart's keep regex:
add the cost families the chart's default lists, or scrape mode keeps
dropping volume, spot, load-balancer and egress cost.
2.7.0 also runs with agent 1.4.9, and Clusters shows it as update
available. Agents below 1.4.9 show as update recommended.
Install
Helm (recommended for Kubernetes)
helm install kubebolt oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt \
--namespace kubebolt --create-namespace
kubectl -n kubebolt port-forward svc/kubebolt 3000:80Upgrading an existing install: see Upgrading above and the chart's
Upgrade guide.
Single Binary (download below)
# macOS Apple Silicon
curl -LO https://github.com/clm-cloud-solutions/kubebolt/releases/download/v2.7.0/kubebolt-darwin-arm64
chmod +x kubebolt-darwin-arm64 && mv kubebolt-darwin-arm64 /usr/local/bin/kubebolt
# Linux amd64
curl -LO https://github.com/clm-cloud-solutions/kubebolt/releases/download/v2.7.0/kubebolt-linux-amd64
chmod +x kubebolt-linux-amd64 && sudo mv kubebolt-linux-amd64 /usr/local/bin/kubebolt
kubebolt --kubeconfig ~/.kube/configAll platform binaries are attached below. Verify with sha256sum -c CHECKSUMS.txt.
Docker (single container)
docker run -p 3000:3000 \
-v ~/.kube/config:/kubeconfig:ro -e KUBECONFIG=/kubeconfig \
ghcr.io/clm-cloud-solutions/kubebolt:2.7.0Homebrew (macOS, Linux)
brew install clm-cloud-solutions/tap/kubeboltkubectl krew plugin
kubectl krew index add clm https://github.com/clm-cloud-solutions/krew-index.git
kubectl krew install clm/kubebolt
kubectl kubeboltDocker Compose
git clone https://github.com/clm-cloud-solutions/kubebolt.git
cd kubebolt/deploy && docker compose up -dContainer Images
- Single container (binary + frontend):
ghcr.io/clm-cloud-solutions/kubebolt:2.7.0 - API only:
ghcr.io/clm-cloud-solutions/kubebolt/api:2.7.0 - Web only:
ghcr.io/clm-cloud-solutions/kubebolt/web:2.7.0
Changelog
Features
- feat(agents): say when a cluster's agent is out of date, why, and the command to upgrade it ()
- feat(home): each attention row says where it comes from ()
- feat(mcp): block a connector operation without a release ()
- feat(connectors): a call the rate limit held back is «not read», not failed ()
- feat(kobi): a limit on questions reads as a limit, not as Kobi failing ()
- feat(mcp): ask the per-plan rate even when the install has no plan ()
- feat(mcp): connector rate limit per org and consumer ()
- feat(mcp): alert when Kobi keeps asking for what it may not do ()
Fixes
- fix(opencost): promread keeps OpenCost's own labels, and the chart refuses two sources of cost ()
- fix(agents): the upgrade instructions warn about kubectl's context and full-values installs ()
- fix(clusters): cards take their columns from the width the page has, and nothing runs past their edge ()
- fix(agents): an agent that reports 1.0.0 is «before 1.4.4», and Kobi answers whether the agent is out of date ()
- fix(opencost): one list of cost families for every path, and OpenCost's unread series dropped ()
- fix(audit): the action history applies the class before the limit ()
- fix(mcp): a cloud call that did not read says why in KubeBolt's words, and the trail names it ()
- fix(kobi): the connector's procedure goes with the first call that reaches the cloud ()
- fix(mcp): the refusal series start at 0, so the first burst alerts ()
- fix(kobi): a connector call that reached nothing is not counted as an error ()
- fix(connectors): a blocked entry that matches nothing is refused ()
- fix(kobi): strip a connector's procedure by its header, not a removed map ()
- fix(kobi): a connector's first call reads by its own answer, not by the procedure in front of it ()
Other
- Merge pull request #352 from clm-cloud-solutions/develop ()
- Merge pull request #351 from clm-cloud-solutions/chore/release-2.7.0 ()
- chore(release): 2.7.0, agent 1.4.10 ()
- Merge pull request #350 from clm-cloud-solutions/fix/opencost-cost-families ()
- Merge pull request #349 from clm-cloud-solutions/feat/agent-update-notice ()
- Merge pull request #348 from clm-cloud-solutions/fix/audit-class-before-limit ()
- Merge pull request #347 from clm-cloud-solutions/fix/mcp-live-test-findings ()
- refactor(web): Kobi's panel follows the server's queue instead of retrying on its own ()
- Merge pull request #309 from clm-cloud-solutions/fix/connector-blocked-operations ()
- Merge pull request #308 from clm-cloud-solutions/feat/kobi-limit-notices ()
- Merge pull request #307 from clm-cloud-solutions/feat/mcp-rate-per-plan-and-consumer ()
- Merge pull request #306 from clm-cloud-solutions/feat/mcp-refusal-alert ()
- Merge develop into fix/connector-blocked-operations ()
- Merge develop into feat/kobi-limit-notices ()
- Merge develop into feat/mcp-rate-per-plan-and-consumer ()
- Merge develop into feat/mcp-refusal-alert ()
- Merge pull request #346 from clm-cloud-solutions/feat/home-attention-source-labels ()
- Merge remote-tracking branch 'origin/develop' into fix/connector-blocked-operations ()
- test(mcp): the refusal-counter tests live in their own file ()