Repository navigation
v2.6.0
KubeBolt 2.6.0 — the Pods tab pages, VictoriaMetrics shows its work, and pod network counts once
Drop-in over 2.5.x. No schema change and no migration. The agent moves to
1.4.8, a metric-correctness fix: each pod's network is now stored once.
A workload's Pods tab pages its pods
A Deployment, StatefulSet, DaemonSet or Job with hundreds of pods used to load
every one into its Pods tab at once. The tab now pages them, 50 at a time,
sorted by name, and the backend reads them from the workload's own namespace
and carries only the fields the list draws — so a big workload opens quickly
instead of hauling the whole pod set across the wire.
- The pickers that need the pod list (logs, terminal, files) poll it once a
minute rather than on every render, and count the pods without fetching rows. - A
hostNetworkpod's Monitor tab now points at its node's network. A
pod that shares the node's network namespace has no network of its own to
chart, so its Monitor reads the node'snode_network_*instead of an empty
per-pod series.
VictoriaMetrics shows what it does with your metrics
Administration › System › Health gains a VictoriaMetrics section: the store
measuring itself — ingestion, active series, what it accepts and what it drops,
and how it is sized against what it receives. The panel explains the rows the
store keeps rather than leaving the metric store a black box.
VictoriaMetrics is also configured to keep series of up to 60 labels, so the
agent's enriched network and workload series are stored in full instead of being
dropped for carrying one label too many. The chart and the compose file set it;
a VictoriaMetrics you run yourself (metrics.storage.embedded.enabled=false,
KUBEBOLT_METRICS_STORAGE_URL, the single-container image) needs
-maxLabelsPerTimeseries=60 added to its own flags — its default drops those
series, and only the new section shows it.
Pod network counts once
Every pod's network traffic was stored twice. The agent's stats collector
(/stats/summary) and its cAdvisor collector (/metrics/cadvisor) both sent
container_network_* for each pod interface, with label sets that never matched,
so VictoriaMetrics kept two series and every sum by (pod) or by (namespace)
came out doubled — and both copies counted against active series. Present since
the agent's 0.0.5.
- The agent (1.4.8) now runs cAdvisor right after stats, in the same pass,
and skips what stats already sent — while still sending what stats leaves out
(*_packets_dropped_total, short-lived pods, a second interface like Multus
net1). Dropped-packet series gain the workload labels. - Pods with
hostNetwork: trueno longer send pod network at all — they
share the node's interfaces, so that traffic now lives innode_network_*,
once. On a Cilium kind cluster, 11 hostNetwork pods had been producing 82% of
thecontainer_network_*series. - The API, the pod charts, Copilot and Kobi count each pod interface once.
- After the upgrade the duplicate (
pod_uid-less) copies stop; queries that
selectpod_uid!=""or the workload labels are unchanged, and a plainsum
falls to the real value. Stored history keeps both copies until retention
drops them.
Right-sizing waits until it has real data
On a freshly-connected cluster, right-sizing used to flag every workload
over-provisioned and recommend shrinking it to almost nothing — a workload with
no usage samples yet read as one measured at zero. It now holds its
recommendations until it has about two days of history: below that, the
Capacity and Cost surfaces show a collecting samples note instead of advice, or
a false "well sized", built on a cluster that has barely run. A workload the
engine has genuinely never sampled is left alone; a genuinely idle one still
shows as over-provisioned. Only a completed evaluation says "well sized" now: a
metrics store that does not answer reads as such on every surface (and is no
longer taken for missing history), and the Capacity card shows — with the
hours collected so far instead of a 0.
Reaching a metrics-only cluster
A cluster connected with a metrics-only agent (rbac.mode=metrics) or a
Prometheus-reading (Mode C) agent — one that was never an agent-proxy — returned
"waiting for agent to register" on every cluster-scoped view, even though Fleet
listed it as live. Its metrics dashboards are now reachable. Folded in:
- Switching to a metrics-only cluster from a global page (Home / Fleet /
Security) opens its Overview on the first click, not the second. - Renaming a metrics-only (or any agent-proxy) cluster no longer fails with
context "agent%3A…" not found. - Deleting a metrics-only cluster removes it. The delete answered 200 and
the cluster stayed listed until the API restarted. - A Prometheus-reading agent's demotion no longer overwrites a DaemonSet agent
registering at the same instant, nor brings back a cluster deleted within its
30-second grace window.
Insights bursts follow the cluster you're looking at
The Bursts tab showed every cluster's bursts (they are grouped org-wide, by
shared cause, on purpose), so a burst made entirely of another cluster's episodes
appeared under the one you had open — and its "See the N episodes" link then
opened this cluster's episodes in the burst's time window rather than the burst's
real members, so the count never matched.
- The tab now defaults to the selected cluster, with a This cluster / All
clusters toggle that mirrors Active and History. - "See the N episodes" opens the burst's exact members now — the drill-down
fetches the burst's member episodes by id, so the count matches, the
severity classifiers and pagination at the top work over that set, and back
to bursts returns to the window and scope you came from.
Home's "while you were away" summary stays org-wide — it answers "what needs me,
anywhere?", a different question from the one the cluster's own Bursts tab answers.
Smaller fixes
- A reconnecting Prometheus-reading agent no longer demotes its cluster. A
Mode C (promread) pod coming back a second after the DaemonSet used to delete
the agent-proxy context and take every resource view down with it. - The Monitor charts draw each step's peak on long ranges, and Kobi's
get_workload_metricsfinds the same peaks the charts show and says when each
one happened — instead of an averaged value that hid the spike. The stats
beside a peaked chart say so (last,avg peak,min peak), and Ask Kobi is
told not to size requests from them. - A recurring OOM stays open until the container outlasts its restart
interval, rather than flapping resolved between kills. - A Deployment's first rollout no longer trips the zero-replicas insight —
it waits the rollout out. - Agent samples reach VictoriaMetrics with their labels in one order, so a
series is not split by label ordering.
Security
- Kobi's chat renders Markdown images as text. A model answer could embed a
Markdown image whose URL the browser would fetch on render; chat now shows the
image markup as text and the page carries an image content-security policy. - VictoriaMetrics and vmagent → v1.156.0-scratch, clearing their images'
dependency reports. - Two Alpine CVEs in the pinned
azure-mcpsidecar image are waived with an
owner and a remove-by date (no fix available upstream yet; the sidecar is
loopback-only, read-only, with no ServiceAccount token).
Upgrading
# The backend
helm upgrade kubebolt oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt \
--version 2.6.0 -n kubebolt --reset-then-reuse-values
# Then the agent, in every cluster that runs one (with that cluster's kubectl context)
helm upgrade kubebolt-agent oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt-agent \
--version 1.4.8 -n kubebolt-system --reset-then-reuse-valuesUse the namespaces you installed into: helm list -A | grep kubebolt shows
them. --reset-then-reuse-values needs Helm 3.14+; passing your values file
(-f) works too. Do not use --reuse-values: it keeps the previous chart's
defaults, so VictoriaMetrics and vmagent would stay on your previous version's
image instead of v1.156.0. Step by step, what to expect, rollback, and what
each version asks of you when you skip some:
Upgrade.
2.6.0 also runs with agent 1.4.7; 1.4.8 stops the duplicate pod-network series —
upgrade it to halve the container_network_* cardinality and get correct
sum by (pod) / by (namespace) network totals.
Install
Helm (recommended for Kubernetes)
helm install kubebolt oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt \
--namespace kubebolt --create-namespace
kubectl -n kubebolt port-forward svc/kubebolt 3000:80Upgrading an existing install: see Upgrading above and the chart's
Upgrade guide.
Single Binary (download below)
# macOS Apple Silicon
curl -LO https://github.com/clm-cloud-solutions/kubebolt/releases/download/v2.6.0/kubebolt-darwin-arm64
chmod +x kubebolt-darwin-arm64 && mv kubebolt-darwin-arm64 /usr/local/bin/kubebolt
# Linux amd64
curl -LO https://github.com/clm-cloud-solutions/kubebolt/releases/download/v2.6.0/kubebolt-linux-amd64
chmod +x kubebolt-linux-amd64 && sudo mv kubebolt-linux-amd64 /usr/local/bin/kubebolt
kubebolt --kubeconfig ~/.kube/configAll platform binaries are attached below. Verify with sha256sum -c CHECKSUMS.txt.
Docker (single container)
docker run -p 3000:3000 \
-v ~/.kube/config:/kubeconfig:ro -e KUBECONFIG=/kubeconfig \
ghcr.io/clm-cloud-solutions/kubebolt:2.6.0Homebrew (macOS, Linux)
brew install clm-cloud-solutions/tap/kubeboltkubectl krew plugin
kubectl krew index add clm https://github.com/clm-cloud-solutions/krew-index.git
kubectl krew install clm/kubebolt
kubectl kubeboltDocker Compose
git clone https://github.com/clm-cloud-solutions/kubebolt.git
cd kubebolt/deploy && docker compose up -dContainer Images
- Single container (binary + frontend):
ghcr.io/clm-cloud-solutions/kubebolt:2.6.0 - API only:
ghcr.io/clm-cloud-solutions/kubebolt/api:2.6.0 - Web only:
ghcr.io/clm-cloud-solutions/kubebolt/web:2.6.0
Changelog
Features
- feat(web): a workload's Pods tab pages its pods, 50 at a time, sorted by name ()
- feat: a hostNetwork pod's Monitor tab points at its node's network ()
- feat(deploy): VictoriaMetrics keeps series of up to 60 labels ()
- feat(web): the VictoriaMetrics section shows what the store does with the rows it receives ()
Fixes
- fix(web): VictoriaMetrics memory chart unstacked; Pods tab page per workload ()
- fix(cluster): metrics-only clusters can be deleted; demotion checks and writes under one lock ()
- fix(insights): a burst's drill-down says when it is gone and keeps its way back ()
- fix(web): Monitor stats over step peaks say so, and Ask Kobi is told ()
- fix(rightsizing): a failed read or a cluster still collecting never reads as well sized ()
- fix(insights): cluster-scope the bursts tab and drill into a burst's exact members ()
- fix(rightsizing): don't recommend resizing before there's enough usage history ()
- fix(cluster): URL-decode the {context} path param in rename/delete handlers ()
- fix(cluster): switching to a metrics-only cluster no longer reports a spurious error ()
- fix(api): reach a metrics-only cluster that was never agent-proxy ()
- fix(agent): a reconnecting promread no longer demotes its agent-proxy cluster ()
- fix(web): render chat Markdown images as text ()
- fix(copilot): get_workload_metrics says when the peak happened ()
- fix(copilot): get_workload_metrics finds the same peaks as the Monitor charts ()
- fix(insights): a recurring OOM stays open until the container outlasts its interval ()
- fix(insights): zero-replicas waits out a Deployment's first rollout ()
- fix(web): Monitor charts draw each step's peak on long ranges ()
- fix(agent): hostNetwork pods send no pod network ()
- fix(agent): each pod interface reaches the store once ()
- fix: the pod chart, Copilot and Kobi count each pod interface once ()
- fix(api): agent samples reach VictoriaMetrics with their labels in one order ()
Performance
- perf(api): a workload's pod list carries only the fields its readers use ()
- perf: a workload's pods are read from its namespace, counted without rows, and polled once a minute by pickers ()
Documentation
- docs(release): 2.6.0 review — agent README to 1.4.8, own-VictoriaMetrics label limit, CLAUDE.md ()
- docs: renumber metrics-only finding #72 to #73 ()
Other
- Merge pull request #296 from clm-cloud-solutions/develop ()
- Merge pull request #295 from clm-cloud-solutions/chore/release-2.6.0 ()
- Merge pull request #294 from clm-cloud-solutions/chore/release-2.6.0 ()
- chore(release): 2.6.0 (agent 1.4.8) ()
- Merge pull request #293 from clm-cloud-solutions/fix/rightsizing-insufficient-history ()
- Merge pull request #292 from clm-cloud-solutions/fix/metrics-only-cluster-unreachable ()
- Merge pull request #291 from clm-cloud-solutions/fix/reconnect-race-demotes-cluster ()
- Merge pull request #281 from clm-cloud-solutions/feat/vm-health-panels ()
- chore(security): waive two Alpine CVEs in the pinned azure-mcp sidecar image ()
- Merge pull request #290 from clm-cloud-solutions/fix/chat-renders-markdown-images ()
- deps(deploy): bump VictoriaMetrics and vmagent to v1.156.0-scratch ()
- Merge pull request #289 from clm-cloud-solutions/fix/monitor-peaks-and-insights ()
- Merge pull request #288 from clm-cloud-solutions/feat/workload-pods-tab-pages ()
- Merge pull request #287 from clm-cloud-solutions/perf/workload-pods-list-rows ()
- Merge pull request #286 from clm-cloud-solutions/feat/hostnetwork-pods-use-node-network ()
- Merge pull request #285 from clm-cloud-solutions/perf/workload-pods-namespace-index ()
- Merge pull request #284 from clm-cloud-solutions/feat/agent-hostnetwork-pod-network ()
- Merge pull request #283 from clm-cloud-solutions/fix/pod-network-counted-twice ()
- Merge pull request #282 from clm-cloud-solutions/fix/agent-pod-network-single-source ()
- Merge pull request #280 from clm-cloud-solutions/fix/agent-ingest-writes-labels-in-order ()