feat(kb): add kibana functional tests by margaretjgu · Pull Request #279 · elastic/cli

margaretjgu · 2026-05-04T19:57:16Z

Summary

Closes #219

Adds hand-authored functional tests for 5 Kibana API namespaces: data-views, spaces, alerting, connectors, saved-objects
Each test exercises the full CRUD lifecycle (create, get, list, delete / export, import) using the same set -euo pipefail + elastic --json + jq assertion pattern as ES functional tests
Fixes two pre-existing bugs in kibana-client.ts (introduced in feat(kb): Add Kibana API support with generated command definitions #195) required to make saved-objects work:
- application/x-ndjson responses are now parsed line-by-line into a JSON array instead of throwing a JSON parse error
- multipart/form-data requests are now sent via FormData (file upload) instead of application/json, fixing the 415 Unsupported Media Type error from Kibana
Adds requestType and responseType fields to KbApiDefinition so the request builder and client know which transport mode to use (depends on elastic/elastic-client-generator-js feat/kb-ndjson-multipart)
Adds a Buildkite CI step (run-kb-tests.sh) that starts ES + Kibana Docker containers and runs the suite on Node 22 and 24
Adds test:functional:kb npm script for local execution

Future optimization: #280

Test plan

All 5 tests pass locally against a live Kibana 9.3.0 (npm run test:functional:kb: 5 passed, 0 failed)
Generator PR (elastic-client-generator-js feat/kb-ndjson-multipart) merged first
Buildkite KB functional tests step passes in CI

github-actions · 2026-05-04T20:01:03Z

✅⚠️MegaLinter analysis: Success with warnings

Descriptor	Linter	Files	Errors	Warnings	Elapsed time
⚠️ BASH	shellcheck	8	1	0	0.46s
✅ COPYPASTE	jscpd	yes	no	no	5.26s
✅ REPOSITORY	gitleaks	yes	no	no	92.27s
✅ REPOSITORY	git_diff	yes	no	no	0.46s
✅ REPOSITORY	secretlint	yes	no	no	17.72s
✅ REPOSITORY	trivy	yes	no	no	15.94s
✅ TYPESCRIPT	eslint	7	0	0	6.07s
✅ YAML	yamllint	1	0	0	0.78s

Detailed Issues

⚠️ BASH / shellcheck - 1 error

In .buildkite/run-kb-tests.sh line 65:
  docker load < "$(ls "$ES_CACHE_DIR/elasticsearch-$STACK_VERSION"*.tar.gz | head -1)"
                   ^-- SC2012 (info): Use find instead of ls to better handle non-alphanumeric filenames.

For more information:
  https://www.shellcheck.net/wiki/SC2012 -- Use find instead of ls to better ...

See detailed reports in MegaLinter artifacts
Set VALIDATE_ALL_CODEBASE: true in mega-linter.yml to validate all sources, not only the diff

MegaLinter is graciously provided by OX Security
Show us your support by starring ⭐ the repository

…loyments

JoshMock · 2026-05-05T17:34:51Z

Not sure what's up with Megalinter, but otherwise this LGTM.

…ished

…rray)

margaretjgu · 2026-05-05T18:21:19Z

Not sure what's up with Megalinter, but otherwise this LGTM.

yea it was flagging some dummy keys as a leak. updatged the linter to ignore these specific keys

…download timeout

…ld phase

…ith n2-standard-4

margaretjgu · 2026-05-05T19:42:27Z

ES startup timeout: root cause and fixes

Getting ES running in CI for the Kibana tests turned out to be surprisingly tricky. Here is what happened and how we fixed it.

Why ES kept timing out

When you run docker run --detach, Docker does not just start the process. It first has to unpack the image layers into the container filesystem (via overlay2). For the 695MB Elasticsearch image, this alone took 7+ minutes on cold CI agents. Our health check timer started immediately after docker run returned, so we were timing out before ES had even booted its JVM.

The script was doing things in this order:

npm install > npm build > docker pull > docker run ES > wait 6 min > timeout

Meanwhile, ES was not even alive yet.

What we changed

1. Start ES before the build

npm install + npm build takes about 15 minutes. We now start the ES and Kibana containers before that, so they boot in the background while the build runs. By the time the build finishes, both services are ready and waiting.

docker pull > docker run ES > docker run Kibana > npm install > npm build > check health (instant)

2. Better agent image

Switched from family/core-ubuntu-2204 to family/kibana-ubuntu-2404 (same image the Kibana team uses) with machineType: n2-standard-4. The newer N2 hardware has faster local storage I/O, which is the main bottleneck for Docker image unpacking.

3. Two-phase ES health check

Even after /_cluster/health returns 200, the .security-7 index may not exist yet. Kibana's alerting and connectors plugins need it to store API keys. We now explicitly probe POST /_security/api_key to confirm the security index is bootstrapped before starting Kibana. This is borrowed from Kibana's own wait_for_security_index.ts.

4. ES snapshot cache

The kibana-ubuntu-2404 agent image may have ES Docker layers pre-cached on disk. The script now checks $ES_CACHE_DIR first and loads from there before falling back to a registry pull, which is the same approach Kibana uses.

…lth checks Ubuntu 24.04 agents use nftables which can break Docker --publish port forwarding. Resolve container IPs on the Docker bridge network immediately after docker run and use those for all health checks and CLI config. The Linux host can always reach bridge network container IPs directly without going through port publishing.

Docker bridge/port-publishing both fail on kibana-ubuntu-2404 agents (Ubuntu 24.04 uses nftables which breaks Docker NAT rules). --network host puts both containers directly on the host network stack so localhost:9200 and localhost:5601 work unconditionally — equivalent to how Kibana's own CI runs ES as a native process via node scripts/es snapshot.

…unner container On kibana-ubuntu-2404 agents, all host→container networking is broken: --network host is blocked (user namespace remapping), --publish doesn't work (nftables replaces iptables), and direct bridge IPs are not routed to the host. The only reliable networking is inter-container communication on a custom bridge network. This matches how Kibana's own CI works: Kibana runs ES natively so localhost always works; we achieve the same by running a dedicated test-runner container (node:NODE_VERSION-bookworm-slim) on the same network as ES and Kibana, using Docker DNS aliases (elasticsearch:9200, kibana:5601). The built workspace is mounted read-only so node dist/cli.js works without rebuilding.

The test runner container couldn't reach ES via Docker DNS (elasticsearch:9200) in build 268 for an unknown reason. This adds: - Network diagnostics in the runner (resolv.conf, routes, first verbose curl) so the next failure gives us the exact error (DNS failure, TCP refused, etc.) - IP fallback: docker inspect fetches ES/Kibana container IPs on the host and passes them as ES_IP / KB_IP; the runner uses these if DNS lookup fails - docker logs for ES and Kibana in the cleanup trap for visibility - Lower Node.js heap limit (6GB -> 4GB) to reduce memory pressure during build

ES 8.0+ auto-enables HTTPS on the HTTP layer when ELASTIC_PASSWORD is set. This caused "Empty reply from server" errors because the test runner and Kibana were connecting via http:// to a port expecting TLS. Kibana then crashed and its DNS entry was removed, explaining the secondary "kibana DNS failed" symptom. Setting xpack.security.http.ssl.enabled=false keeps security (auth, RBAC, API keys) enabled while allowing plain HTTP access, which is fine for CI.

Two fixes: - Kibana was crashing silently because --rm auto-removed the container before the cleanup trap could collect logs. Removed --rm so docker logs always works. - Kibana was likely crashing because it tried to connect to ES before ES finished bootstrapping. Moved Kibana startup to after the npm build (~3 min buffer), so ES is fully ready when Kibana first connects. ES still starts early. - Also adds xpack.security.transport.ssl.enabled=false to ES for consistency with the http.ssl flag (aligns with elastic/start-local reference setup).

Kibana 9.x explicitly forbids ELASTICSEARCH_USERNAME=elastic with a fatal config validation error. We must use kibana_system instead. Since the host cannot reach ES directly on this agent, a one-shot Node.js container (setup-kibana.js) runs on the same Docker network, waits for ES cluster health and the security index, sets the kibana_system password, then exits. Kibana is then started with ELASTICSEARCH_USERNAME=kibana_system.

The repo has "type": "module" in package.json so .js files are treated as ESM, causing "require is not defined in ES module scope". Renaming to .cjs forces Node to treat it as CommonJS regardless of package.json. Also adds the missing SPDX-License-Identifier header to pass the test:spdx check.

The /api/actions/connector_types and /api/alerting/rules/_find health checks were timing out because the Fleet plugin's retry loop (FleetEncryptedSaved ObjectEncryptionKeyRequired for agent binary source) was causing those endpoints to return non-200 responses. Fleet's issue is unrelated to our tests. Replace the 30-retry polling loop with a 15-second sleep after Kibana reports "available". By that point all essential plugins (alerting, actions) are initialised as part of the "available" state.

… check Three root causes addressed: 1. `params` (alerting create) and `config`/`secrets` (connector create/update) were typed as "string" in the generated API definitions. The CLI factory only JSON-parses flag values for "object"/"array" typed params, so these were sent as raw string literals instead of JSON objects, producing 400 errors from the Kibana API. Fixed in the generator (elastic-client-generator-js#174) and reflected here. 2. The previous `sleep 15` after Kibana's "available" status was not reliable. Kibana's actions plugin serves 403 "license information is not available" until its license subscription fires after connecting to ES. Replaced with an active poll on GET /api/actions/connector_types which directly confirms the license is loaded and the actions API is ready. 3. Added stderr capture (2>/tmp/cli-err.txt + cat on failure) to the first CLI call in alerting.sh and connectors.sh so the actual HTTP error is visible in the Buildkite log if any future failure occurs.

Polling GET /api/actions/connector_types directly was causing repeated 500 Server Errors in Kibana's HTTP access log (the actions plugin HTTP context is not yet wired when Kibana first reports 'available', so early requests get a 500). This looked like the Fleet error resurfacing. Switch to polling /api/status and checking .status.plugins.actions.level == 'available' .status.plugins.alerting.level == 'available' The status endpoint always returns 200 and never causes log noise. Fleet degradation appears only in plugins.fleet and does not affect plugins.actions or plugins.alerting.

Kibana's Docker entrypoint only processes environment variables in SCREAMING_SNAKE_CASE format (e.g. XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY). Dotted-notation names (e.g. xpack.encryptedSavedObjects.encryptionKey) are not picked up, so encryptedSavedObjects.canEncrypt stayed false in CI. Every call to getActionsClient() checks canEncrypt and throws 'Unable to create actions client because the Encrypted Saved Objects plugin is missing encryption key' causing a 500 on both POST /api/alerting/rule and GET /api/actions/connector_types. Confirmed: local Kibana (start-local) sets XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY and all 5 functional tests pass (5/5 locally).

margaretjgu added 4 commits May 4, 2026 15:54

test: add KB data-views functional tests

68be249

test: add KB spaces functional tests

1cc0758

ci: add KB functional tests step to Buildkite pipeline

2bdf9f0

chore: add test:functional:kb npm script

b24359b

margaretjgu changed the title ~~Feat/kb functional tests~~ feat(kb): add kibana functional tests May 4, 2026

margaretjgu and others added 12 commits May 4, 2026 16:13

feat(kb): add requestType and responseType to KbApiDefinition

7dba2f0

fix(kb): handle ndjson response and multipart/form-data request

447bf4a

fix(kb): route multipart body params to form fields in request builder

bee9d66

chore(kb): regenerate saved-objects with requestType and responseType

7dd5841

test(kb): add saved-objects export/import functional test

abbeb90

Merge branch 'main' into feat/kb-functional-tests

b2c7c2f

fix(kb): make auth optional in KibanaClient for security-disabled dep…

33e7e21

…loyments

test(kb): update unit tests for optional auth in KibanaClient

15a8ef5

fix(test): use stackAlerts consumer and Node UUID in KB functional tests

620a99d

fix(ci): wait for alerting and actions plugins before running KB tests

e690f8e

fix(ci): enable ES/Kibana security so actions and alerting plugins work

98bd3e0

fix(ci): publish ES port 9200 and use docker exec for health check

72ea637

JoshMock approved these changes May 5, 2026

View reviewed changes

margaretjgu added 4 commits May 5, 2026 13:52

fix(ci): use host curl for ES health check now that port 9200 is publ…

24c600b

…ished

fix(ci): suppress gitleaks false positives for dummy encryption keys

7dcf50b

fix(ci): allowlist dummy Kibana encryption key in gitleaks config

7896fce

fix(ci): correct gitleaks allowlist syntax (singular table, regexes a…

80e0254

…rray)

margaretjgu added 5 commits May 5, 2026 14:27

fix(ci): increase ES health timeout to 6 min, add progress logging

3eda510

fix(ci): wait for ES security index before starting Kibana

3042673

fix(ci): pull Docker images before health check timer to avoid image …

2403829

…download timeout

fix(ci): start ES before npm build so it boots during the ~15 min bui…

f1afbd2

…ld phase

fix(ci): start ES+Kibana before build, use kibana-ubuntu-2404 agent w…

060a36d

…ith n2-standard-4

JoshMock approved these changes May 5, 2026

View reviewed changes

margaretjgu added 13 commits May 5, 2026 16:03

fix(ci): add 5s HTTP timeout to setup-kibana requests

756570a

chore(ci): remove verbose debug logging from kb test runner

a1ce6a9

margaretjgu enabled auto-merge (squash) May 5, 2026 23:08

margaretjgu merged commit adb6cc7 into main May 5, 2026
20 of 35 checks passed

margaretjgu deleted the feat/kb-functional-tests branch May 5, 2026 23:19

elastic-vault-github-plugin-prod Bot mentioned this pull request May 4, 2026

chore: release main #246

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(kb): add kibana functional tests#279

feat(kb): add kibana functional tests#279
margaretjgu merged 39 commits into
mainfrom
feat/kb-functional-tests

margaretjgu commented May 4, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 4, 2026 •

edited

Loading

Uh oh!

JoshMock commented May 5, 2026

Uh oh!

margaretjgu commented May 5, 2026

Uh oh!

margaretjgu commented May 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

margaretjgu commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

github-actions Bot commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅⚠️MegaLinter analysis: Success with warnings

Detailed Issues

Uh oh!

JoshMock commented May 5, 2026

Uh oh!

margaretjgu commented May 5, 2026

Uh oh!

margaretjgu commented May 5, 2026

ES startup timeout: root cause and fixes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

margaretjgu commented May 4, 2026 •

edited

Loading

github-actions Bot commented May 4, 2026 •

edited

Loading