Large clusters

Large clusters

How Sextant behaves on clusters with tens of thousands of pods and hundreds of CRDs, what it does to stay responsive, and where its limits are (v0.15.0, issue #157).

In short

• Big tables show their first rows while the rest is still loading, and keep at most 20 000 objects.
• Before listing more than 10 000 objects of a kind in all namespaces, the table asks.
• Label and field terms you type in the search box are applied by the API server, so only matching objects are downloaded.
• The sidebar shows how many objects each kind has, within a request budget.
• Views that show the same list share one watch.

To see fewer objects: choose namespaces in the namespace filter, or type app=web, tier!=db, status.phase=Pending or spec.nodeName=node-1 in the search box. See Using Sextant for the search syntax.

What changed in 0.15

Area	Before	Now
First rows of a big table	Shown only after the whole list was downloaded and parsed	The list arrives in parts while the API server pages through it (500 objects a page): the first rows show after the first page, with "Loading… N so far"
Memory of a table	Every object of the kind, however many	At most 20 000 objects per table; the rest are counted, not kept, and the table says "Showing the first 20000 of M objects. Narrow the list by namespace or label to see the rest."
Opening a kind with very many objects in all namespaces	Listed everything	Counted first (one metadata request of one object, at most 10 s; the table does not wait more than 3 s for it); above 10 000 the table asks before listing ("Show anyway" is remembered until restart)
Label and field terms in the search box	Filtered in the window after everything was listed	Sent to the API server 400 ms after you stop typing: app=web, tier!=db, app= (has the label) become a label selector; status.phase=Running, spec.nodeName=node-1 (pods), metadata.name=… (any kind) and the other fields the API server supports become a field selector. The previous rows stay ("Updating…") until the narrower list is in
Selected namespaces	One watch per namespace	Unchanged (verified): with namespaces selected, nothing is listed cluster-wide
Sidebar	No counts	A count next to each kind of an open built-in section and next to the open entry, from one metadata request of one object per namespace (cached for a minute, "999+" above that); closed sections, installed tools' sections and custom resources ask nothing, and a cluster gets at most 20 count requests a minute
Same list in two views	A table and the resource map (or the RBAC views) each opened their own watch	One shared watch per (cluster, kind, namespaces, selectors); a view that needs every object lifts a table's cap
Command palette	Every object of the searched kinds (metadata only)	At most 50 000 objects per kind; the footer says when a kind was searched only partly
Sorting and search	localeCompare with options per comparison, the search parsed per object	One collator, the search compiled once per query

Measurements

Backend: watch of N pods from a fake API server

Release build, a fake API server paging realistic pods (≈2.5 KB of JSON each) 500 per page; "page delay" is how long each page takes, like a busy API server. kept estimates the heap of the objects the watch keeps.

Pods	Page delay	Mode	First rows	Complete	Objects sent	Kept
10 000	0	before (one snapshot)	803 ms	803 ms	10 000	96 MB
10 000	0	parts	24 ms (500 rows)	527 ms	10 000	96 MB
10 000	50 ms	before	1.6 s	1.6 s	10 000	96 MB
10 000	50 ms	parts	74 ms	1.5 s	10 000	96 MB
50 000	0	before	3.6 s	3.6 s	50 000	481 MB
50 000	0	parts + cap 20 000	26 ms	2.5 s	20 000	193 MB
50 000	50 ms	before	7.2 s	7.2 s	50 000	481 MB
50 000	50 ms	parts + cap 20 000	72 ms	7.6 s	20 000	193 MB

The complete time is the API server's paging; the window now has rows after the first page. The cap halves the backend memory of a 50 000-pod table and the data sent to the window drops from 50 000 to 20 000 objects.

Backend: the kind test cluster with 1 943 pods

What	Time
Count of all pods (resource_count)	3 ms
Plain watch, first rows = complete	197–245 ms
Watch in parts, first rows	60–98 ms

Window (jsdom, Node 22)

What	10 000 before	10 000 now	50 000 before	50 000 now
Snapshot to first table rows (whole table render)	216 ms	197 ms	577 ms	316 ms
Sort by name	49 ms	2.7 ms	258 ms	14.8 ms
Search "web-1 team"	11.3 ms	4.3 ms	50.5 ms	23.8 ms
Search with a label term "app=web-3" (in the window)	9.5 ms	0.4 ms	53 ms	2.2 ms
Palette query "w" (every object matches)	5.9 ms	4.1 ms	42.5 ms	12.9 ms
Palette query "web-4242"	8.2 ms	9.7 ms	18.5 ms	16.2 ms
Applying a snapshot to the watch state	1.7 ms	2.1 ms	9.7 ms	8.4 ms
Applying a delta of 10 objects	2.1 ms	1.9 ms	9.7 ms	8.0 ms
A list in parts (all parts)	—	21 ms	—	422 ms (100 parts, spread over the load)

The window keeps a pod as parsed JSON: about 1.5× its JSON size (≈1.6 KB for a 1 KB pod).

The sidebar with 150 / 500 custom kinds paints in 36–75 ms; opening a CRD group re-renders it in ≈40–55 ms. The per-entry counts add about 5 % to that and one request per shown kind per minute (3 ms each on kind).

Limits

• A table keeps at most 20 000 objects. Choose namespaces, type a label (app=web) or a field term (status.phase=Pending) to see the rest; the terms are applied by the API server.
• Above 10 000 objects in all namespaces, a table asks before listing them all. The count comes from the API server's remainingItemCount, which may be approximate; with a label selector the API server does not count, and Sextant counts metadata pages up to 10 000 ("at least …").
• Field selectors exist only for the fields the API server supports (pods: spec.nodeName, spec.restartPolicy, spec.schedulerName, spec.serviceAccountName, spec.hostNetwork, status.phase, status.podIP, status.nominatedNodeName; events: involvedObject.*, reason, reportingComponent, source, type; secrets: type; namespaces: status.phase; nodes: spec.unschedulable; ReplicaSets and ReplicationControllers: status.replicas; Jobs: status.successful; CertificateSigningRequests: spec.signerName; every kind: metadata.name, metadata.namespace). Any other word stays a text search in the window.
• Each change of a table's 20 000-object list copies its index (≈4 ms per batch, at most four batches a second).
• The palette searches at most 50 000 objects per kind and lists at most 6 kinds at once.
• The backend keeps a watched object as a JSON tree (≈4× its JSON size); keeping it as raw JSON would cut that further and is a possible next step.

Running the measurements (for contributors)

# Backend, fake API server with 10k and 50k pods
cargo test --release -p sextant-core --lib scale_watch -- --ignored --nocapture
# Backend, kind (load more pods first: scripts/fixtures/load/load.sh apply; delete afterwards)
SEXTANT_TEST_KUBECONFIG=$(scripts/test-cluster.sh kubeconfig) cargo test --release -p sextant-core --test watch_scale -- --ignored --nocapture
# Window
SEXTANT_SCALE=1 SEXTANT_SCALE_OUT=/tmp/scale.txt npx vitest run --maxWorkers=1 scale