[mongodb_atlas] Fix CEL state wipe and missing group_id on error paths

[shmsr]

Proposed commit message

WHAT: Replace {} terminal returns with explicit reset states in the
process, disk, hardware, mongod_database, and mongod_audit CEL
programs. Preserve group_id and pagination keys on all non-200 measurement
error paths. Fix an invalid nested ternary in the disk terminal (a CEL
syntax error that no static check catches). Replace the committed mock-server
binary with Go source. Remove old pre-compiled mock binary.

WHY: Two related bugs caused integration runs to fail with
"unexpected missing events array from evaluation" or
"failed eval: ERROR: no such key: group_id". Both share the same root cause:
state keys written by one evaluation are lost, so the next evaluation crashes
trying to read them.

How the CEL input loop works

The Filebeat CEL input runs the program in a loop. Each evaluation receives
the previous return value as state, plus one always-re-injected key:
state.url (from resource.url). Every other key — group_id, page_num,
query, pagination cursors — must be present in the program's return value
to survive into the next evaluation. Keys absent from the return are gone.

---

Bug 1 — empty process list wipes all state

Trigger: Atlas /processes returns an empty results array (valid, not an
error — the project has no monitored processes).

Root cause: The terminal branch (state.next >= size(state.hostlist))
returned {}.

Initial state:
  { "url": "…", "group_id": "abc123", "want_more": false, "page_num": 1, "query": "…" }

Eval N — process-list fetch, results: []:
  → state.with({ "hostlist": [], "next": 0, "page_num": 1 })
  → state.next (0) >= size(state.hostlist) (0)  →  terminal branch
  → returns {}                    ← BUG: group_id, page_num, query all gone

Eval N+1 input:
  { "url": "…" }                  ← only url is re-injected
  → tries to build process-list URL: state.url + "…" + state.group_id
  → ERROR: "no such key: group_id"
  → no events[] returned → "unexpected missing events array from evaluation"

Fix: terminal branch returns all required keys:

Eval N (fixed):
  → returns {
      "events": [], "group_id": "abc123",
      "want_more": false, "page_num": 1, "query": "…"
    }

Eval N+1:
  want_more == false → loop stops cleanly ✓

---

Bug 2 — non-200 measurement response drops group_id

Trigger: Atlas measurement endpoint returns a non-200 (temporary throttle,
unavailability, auth expiry).

Root cause: The error branch returned only {events, want_more}.

Initial state:
  { "url": "…", "group_id": "abc123", "want_more": true,
    "hostlist": ["…/processes/host1:27017/measurements?…"], "next": 0,
    "page_num": 1, "query": "…" }

Eval N — measurement fetch returns HTTP 503:
  resp body: {"error": "SERVICE_UNAVAILABLE", "detail": "Atlas unavailable"}
  → error branch returns {
      "events": [{"error": {"code": "SERVICE_UNAVAILABLE", "message": "Atlas unavailable"}}],
      "want_more": false
    }   ← BUG: group_id, page_num, query absent

Eval N+1 input:
  { "url": "…", "events": […], "want_more": false }
  → want_more == false, next interval starts:
  { "url": "…" }                  ← group_id still gone
  → ERROR: "no such key: group_id"

Fix: error branch preserves all required keys:

Eval N (fixed):
  → returns {
      "events": [{"error": {"code": "SERVICE_UNAVAILABLE", "message": "Atlas unavailable"}}],
      "want_more": false,
      "group_id": "abc123", "page_num": 1, "query": "…"
    }   ← group_id survives ✓

---

Additional fixes

disk stream — nested pagination terminal: The disk stream has two
pagination levels (host list → disk partitions per host). The disk-partition
terminal (state.disk_next >= size(state.disk_list)) originally returned {},
wiping state. The fix preserves all keys and correctly handles multi-page disk
lists: if disk_page_num != 1 there are more partition pages for the current
host, so the host cursor must not advance yet.

mongod_database / mongod_audit — terminal branch drops cursor: When
the cluster list is empty the terminal branch was missing
cursor.last_timestamp, so the next scheduled run fell back to the 30-minute
default lookback window instead of continuing from the last timestamp. Fixed by
preserving cursor.last_timestamp: state.endDate.

disk stream — invalid nested ternary (syntax): The disk terminal used
a nested conditional in the true-branch of a ternary without parentheses
(A ? B ? C : D : E). CEL's grammar defines a ternary's true-branch as a
ConditionalOr, not a full Expr, so this is a syntax error — the program
fails to compile at runtime in the Filebeat CEL input and collects zero disk
events. elastic-package build/check/lint do not compile the CEL
program, so it passed every static check. Fixed by wrapping the inner ternary
in parentheses: A ? (B ? C : D) : E. All five programs now compile with
cel-go (see the live-Atlas section below).

---

End-to-end data flow for each test scenario

All scenarios use the same mock server
(_dev/deploy/docker/mock_server/main.go) with Digest auth (user admin,
password MongoDB@123). Group-ID routing selects the response path.

---

Scenario A — happy path (test-default-config.yml, group_id: mongodb-group1)

Step 1 — CEL fetches process list:

GET /api/atlas/v2/groups/mongodb-group1/processes?pageNum=1&itemsPerPage=100

← 200 {"results": [{"id": "hostname-1:27017", "hostname": "hostname-1", "port": 27017,
                    "typeName": "REPLICA_PRIMARY", "version": "7.0.6", ...}],
        "totalCount": 1, "links": [...]}

Step 2 — CEL fetches measurements for hostname-1:27017:

GET /api/atlas/v2/groups/mongodb-group1/processes/hostname-1:27017/measurements?…

← 200 {"processId": "hostname-1:27017", "granularity": "PT1M",
        "measurements": [
          {"name": "CONNECTIONS",                   "dataPoints": [{"value": 38.0,  "timestamp": "…"}]},
          {"name": "ASSERT_REGULAR",                "dataPoints": [{"value": 0.332, "timestamp": "…"}]},
          {"name": "PROCESS_CPU_USER",              "dataPoints": [{"value": 1.07,  "timestamp": "…"}]},
          {"name": "PROCESS_CPU_KERNEL",            "dataPoints": [{"value": 0.237, "timestamp": "…"}]},
          {"name": "PROCESS_NORMALIZED_CPU_USER",   "dataPoints": [{"value": 0.654, "timestamp": "…"}]},
          {"name": "PROCESS_NORMALIZED_CPU_KERNEL", "dataPoints": [{"value": 0.073, "timestamp": "…"}]}
        ], "links": [...]}

Step 3 — CEL zip(names, values) flattens measurements; returns to Filebeat:

{
  "events": [{
    "processId": "hostname-1:27017",
    "granularity": "PT1M",
    "response": {
      "CONNECTIONS": 38.0, "ASSERT_REGULAR": 0.332,
      "PROCESS_CPU_USER": 1.07, "PROCESS_CPU_KERNEL": 0.237,
      "PROCESS_NORMALIZED_CPU_USER": 0.654, "PROCESS_NORMALIZED_CPU_KERNEL": 0.073
    }
  }],
  "hostlist": [], "next": 0, "want_more": false, "page_num": 1,
  "group_id": "mongodb-group1", "query": "…"
}

Step 4 — Ingest pipeline maps fields to mongodb_atlas.* ECS fields.
Pipeline test input/output for the full happy-path field set:

• Input: test-process-metrics.json
• Expected indexed document: test-process-metrics.json-expected.json

---

Scenario B — non-200 measurement (test-non200-meas-config.yml, group_id: non200-meas-group)

Step 1 — CEL fetches process list: same as Scenario A (mock returns
hostname-1:27017 for any group other than empty-processes-group /
non200-processes-group).

Step 2 — CEL fetches measurements → mock returns HTTP 503:

GET /api/atlas/v2/groups/non200-meas-group/processes/hostname-1:27017/measurements?…

← 503 {"error": "SERVICE_UNAVAILABLE", "detail": "mock: measurements unavailable"}

Step 3 — CEL error branch returns (after fix):

{
  "events": [{"error": {"code": "SERVICE_UNAVAILABLE", "message": "mock: measurements unavailable"}}],
  "want_more": false,
  "group_id": "non200-meas-group",
  "page_num": 1,
  "query": "/measurements?granularity=PT10m&period=PT10m"
}

Step 4 — Ingest pipeline receives {"error": {"code": "…", "message": "…"}}.
The set event.kind: pipeline_error if ctx.error?.message != null processor
fires. Indexed document:

{
  "@timestamp": "…",
  "ecs": {"version": "8.11.0"},
  "event": {"kind": "pipeline_error", "module": "mongodb_atlas", "category": ["process"], "type": ["info"]},
  "error": {"code": "SERVICE_UNAVAILABLE", "message": "mock: measurements unavailable"}
}

Note: pipeline tests for error events cannot exist — elastic-package treats any
error.message in a pipeline test output as a test failure by design. The
error-path document format is validated end-to-end by the system test instead.
error.code (ECS keyword) and error.message (ECS match_only_text) come
from the imported ECS v8.11.0 definitions, so the document passes field
validation. This system test passes locally today.

Step 5 — Next eval input still carries group_id: "non200-meas-group",
so the process-list fetch on the following interval succeeds instead of
crashing with "no such key: group_id" ✓.

---

Scenario C — empty process list (Bug 1, local testing only)

group_id: empty-processes-group is supported by the mock server for local
verification. CI system tests cannot cover zero-event scenarios because
elastic-package waits for events and times out. The fix (explicit terminal
return with all keys) is verified by inspection of the CEL code.

GET /api/atlas/v2/groups/empty-processes-group/processes?…
← 200 {"results": [], "totalCount": 0, "links": [...]}

CEL terminal branch returns (after fix):

{
  "events": [], "group_id": "empty-processes-group",
  "want_more": false, "page_num": 1, "query": "…"
}

To test locally: configure a policy with groupId: empty-processes-group and
observe that the agent collects zero events per interval without crashing.

Note on the diff: an earlier revision of this PR added
test-empty-processes-config.yml system tests for this scenario. They were
removed because the scenario produces zero events, and elastic-package test system waits for events and times out (~10 min) — which failed CI. The
test-non200-meas-config.yml configs were also removed — they always failed
CI (see commit 633493e). Bug 2 is verified by live Atlas testing only.

---

Mock server

Group ID	Behaviour
mongodb-group1 (default)	Normal responses for all endpoints
non200-meas-group	/processes normal; measurements return HTTP 503
empty-processes-group	/processes returns results: []
non200-processes-group	/processes returns HTTP 503

Before this PR the mock server was a pre-compiled x86-64 Linux binary committed
at _dev/deploy/docker/mongodb_atlas/test with no source. Routing was
hard-coded and could not be extended for the new error-path scenarios. The
Dockerfile now builds from Go source (_dev/deploy/docker/mock_server/main.go)
so the routing is readable, testable, and extensible.

---

Verified against a live Atlas API

Beyond the mock, every modified program was run unmodified against a real
MongoDB Atlas project (a live 3-node replica set) to confirm the fix on real
response shapes. Because mito (the CEL evaluator used to replicate the
Filebeat CEL input loop) has no Digest-auth support, the program's requests
were sent to a tiny local proxy that re-issued them to cloud.mongodb.com
with Digest auth and forwarded the responses verbatim. The Filebeat loop
semantics were reproduced exactly: publish result.events, retain the rest as
next state, re-inject url, repeat while want_more; then one extra
evaluation to prove the retained state survives the next interval (this is the
eval that used to crash). Error paths (non-200 measurement, empty process list)
were reproduced by toggling fault injection in the proxy while the process list
stayed real.

Result for all five streams (process, disk, hardware, mongod_database,
mongod_audit):

Stream	Happy path (real data)	Bug 1: empty process list	Bug 2: non-200 measurement
process	✓ 3 hosts, 39 metric fields	old crashes / fixed ok	old crashes / fixed ok
disk	✓ 3 hosts, 21 metric fields	old crashes / fixed ok	old crashes / fixed ok
hardware	✓	old crashes / fixed ok	old crashes / fixed ok
mongod_database	✓ (no-data path)	old crashes / fixed ok + cursor kept	logs handle non-200 gracefully
mongod_audit	✓ (no-data path)	old crashes / fixed ok + cursor kept	logs handle non-200 gracefully

On the old code the error paths reproduce the exact runtime failure from the
issue — failed eval: ERROR: no such key: group_id (and no such key: page_num for the mongod_* streams) on the evaluation that follows the error.
The fixed code preserves group_id/page_num/cursor and the loop continues
cleanly.

This live run is also what caught the disk nested-ternary syntax bug
described under Additional fixes above — the program failed to compile in
cel-go, which no static check surfaces. All five programs were then confirmed
to compile cleanly with cel-go.

---

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• elastic-package lint and elastic-package build pass
• All five modified CEL programs compile cleanly with cel-go
• Bug 2 (non-200 measurement → missing group_id) verified by live Atlas testing; test-non200-meas-config.yml system tests were removed (always failed CI)
• Bug 1 (empty process list → {} terminal) verified by CEL code review and local mock-server testing; no CI system test possible (zero-event scenario times out elastic-package)
• All five programs run unmodified against a live Atlas API — happy path, empty-list, and non-200 error paths; old code reproduces the exact runtime crash, fixed code preserves group_id/page_num/cursor and continues cleanly

How to test

Pipeline tests (no stack required — covers happy-path ingest pipeline):

cd packages/mongodb_atlas
elastic-package test pipeline

System tests (requires elastic-package stack up -d --version 9.x.x):

cd packages/mongodb_atlas
elastic-package test system --data-streams process,disk,hardware,mongod_database,mongod_audit

To reproduce Bug 1 locally, run a system test with groupId: empty-processes-group
(or configure the policy in Kibana with that value). The agent should collect
zero events per interval without logging "no such key: group_id".

Related issues

• Closes [mongodb_atlas] mongodb_atlas.process: failed to run: unexpected missing events array from evaluation errors #17927

Screenshots

N/A — no UI changes.

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[Copilot]

Pull request overview

This PR fixes state-handling bugs in the MongoDB Atlas integration’s CEL-based data streams that could wipe required state keys (notably group_id) and cause subsequent evaluations to crash, and it adds dedicated regression/system test coverage plus an updated mock Atlas server to reliably exercise the previously broken paths.

Changes:

• Replace {} terminal returns with explicit reset states that always include events and preserve required state keys across process, disk, hardware, mongod_database, and mongod_audit.
• Preserve group_id (and related pagination keys) on non-200 measurement error paths to prevent follow-on evaluation failures.
• Add Go-based CEL regression tests and refresh the Docker mock server + system test configs to reproduce the bugs deterministically.

Reviewed changes

Copilot reviewed 38 out of 38 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
packages/mongodb_atlas/manifest.yml	Bumps integration version to 1.3.1.
packages/mongodb_atlas/changelog.yml	Documents the bugfix release and links the tracked issue.
packages/mongodb_atlas/data_stream/process/agent/stream/input.yml.hbs	Ensures terminal/error states preserve events and required keys (incl. group_id).
packages/mongodb_atlas/data_stream/hardware/agent/stream/input.yml.hbs	Same state preservation fixes for the hardware CEL program.
packages/mongodb_atlas/data_stream/disk/agent/stream/input.yml.hbs	Fixes terminal state wipes and improves host/disk cursor progression behavior.
packages/mongodb_atlas/data_stream/mongod_database/agent/stream/input.yml.hbs	Replaces {} terminal return with explicit reset state for log stream.
packages/mongodb_atlas/data_stream/mongod_audit/agent/stream/input.yml.hbs	Same terminal reset-state change for audit log stream.
packages/mongodb_atlas/data_stream/process/_dev/test/system/test-empty-processes-config.yml	Adds system test config to exercise empty-process list path.
packages/mongodb_atlas/data_stream/process/_dev/test/system/test-non200-meas-config.yml	Adds system test config to exercise non-200 measurement responses.
packages/mongodb_atlas/data_stream/hardware/_dev/test/system/test-empty-processes-config.yml	Adds empty-processes system test config for hardware stream.
packages/mongodb_atlas/data_stream/hardware/_dev/test/system/test-non200-meas-config.yml	Adds non-200 measurement system test config for hardware stream.
packages/mongodb_atlas/data_stream/disk/_dev/test/system/test-empty-processes-config.yml	Adds empty-processes system test config for disk stream.
packages/mongodb_atlas/data_stream/disk/_dev/test/system/test-non200-meas-config.yml	Adds non-200 measurement system test config for disk stream.
packages/mongodb_atlas/data_stream/mongod_database/_dev/test/system/test-empty-processes-config.yml	Adds empty-processes system test config for mongod_database stream.
packages/mongodb_atlas/data_stream/mongod_audit/_dev/test/system/test-empty-processes-config.yml	Adds empty-processes system test config for mongod_audit stream.
packages/mongodb_atlas/_dev/scripts/go.mod	Introduces a Go module for CEL regression testing scripts.
packages/mongodb_atlas/_dev/scripts/cel_eval_test.go	Adds a Go test suite that runs mito against an in-process mock server and asserts state invariants.
packages/mongodb_atlas/_dev/scripts/mito-config.yaml	Adds local mito configuration documentation for running CEL programs manually.
packages/mongodb_atlas/_dev/scripts/process/process.cel	Adds a checked-in CEL program copy used by the regression tests.
packages/mongodb_atlas/_dev/scripts/process/state-normal.json	Adds baseline state fixture for process scenarios.
packages/mongodb_atlas/_dev/scripts/process/state-empty-processes.json	Adds empty-processes state fixture for process scenario.
packages/mongodb_atlas/_dev/scripts/process/state-non200-meas.json	Adds non-200 measurement state fixture for process scenario.
packages/mongodb_atlas/_dev/scripts/hardware/hardware.cel	Adds a checked-in CEL program copy used by the regression tests.
packages/mongodb_atlas/_dev/scripts/hardware/state-normal.json	Adds baseline state fixture for hardware scenarios.
packages/mongodb_atlas/_dev/scripts/hardware/state-empty-processes.json	Adds empty-processes state fixture for hardware scenario.
packages/mongodb_atlas/_dev/scripts/hardware/state-non200-meas.json	Adds non-200 measurement state fixture for hardware scenario.
packages/mongodb_atlas/_dev/scripts/disk/disk.cel	Adds a checked-in CEL program copy used by the regression tests.
packages/mongodb_atlas/_dev/scripts/disk/state-normal.json	Adds baseline state fixture for disk scenarios.
packages/mongodb_atlas/_dev/scripts/disk/state-empty-processes.json	Adds empty-processes state fixture for disk scenario.
packages/mongodb_atlas/_dev/scripts/disk/state-non200-meas.json	Adds non-200 measurement state fixture for disk scenario.
packages/mongodb_atlas/_dev/scripts/mongod_database/mongod_database.cel	Adds a checked-in CEL program copy used by the regression tests.
packages/mongodb_atlas/_dev/scripts/mongod_database/state-normal.json	Adds baseline state fixture for mongod_database scenario.
packages/mongodb_atlas/_dev/scripts/mongod_audit/mongod_audit.cel	Adds a checked-in CEL program copy used by the regression tests.
packages/mongodb_atlas/_dev/scripts/mongod_audit/state-normal.json	Adds baseline state fixture for mongod_audit scenario.
packages/mongodb_atlas/_dev/deploy/docker/mock_server/main.go	Rewrites the mock Atlas API server in Go and adds special group IDs to trigger error paths.
packages/mongodb_atlas/_dev/deploy/docker/mock_server/go.mod	Adds a Go module for the Docker mock server build.
packages/mongodb_atlas/_dev/deploy/docker/mock_server/.gitignore	Ignores the built mock_server binary.
packages/mongodb_atlas/_dev/deploy/docker/Dockerfile	Updates Docker build to compile and run the new Go-based mock server (Go 1.21).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

packages/mongodb_atlas/data_stream/mongod_database/agent/stream/input.yml.hbs:104

• The terminal branch that runs when state.next >= size(state.hostlist) drops cursor from the returned state. If the process list is empty for a period, this will wipe the persisted cursor and cause the next scheduled run to fall back to the default lookback window again (duplicating data). Preserve the existing cursor in the reset state (and keep it empty when not present).

      {
        "events": [],
        "group_id": state.group_id,
        "want_more": false,
        "page_num": 1,
        "cursor": {

packages/mongodb_atlas/data_stream/mongod_audit/agent/stream/input.yml.hbs:104

• The terminal branch that runs when state.next >= size(state.hostlist) drops cursor from the returned state. If the process list is empty for a period, this wipes the persisted cursor and makes the next scheduled run fall back to the default lookback window again (duplicating data). Preserve the existing cursor in the reset state (or leave it empty when not present).

      {
        "events": [],
        "group_id": state.group_id,
        "want_more": false,
        "page_num": 1,
        "cursor": {

[stefans-elastic]

Just a small comment (and CI failure needs resolving)

[stefans-elastic]

Code Review

The core state-preservation fixes and the nested-ternary CEL syntax catch are correct and well-reasoned. The disk stream's two-level pagination terminal (advance host vs. stay for more disk pages, gated on disk_page_num) is sound on trace-through, and the CEL grammar analysis (ternary true-branch is a ConditionalOr, so the inner ternary needs parens) is accurate — a real latent bug that no static check surfaces.

A few things to address before merge:

🔴 The fix is incomplete: process-list / disk-list non-200 branches still drop group_id

This PR fixes the measurement (second/third do_request) error branches, but the process/host-list error branch — the first do_request in every stream — still returns bare {events, want_more: false} with no group_id:

• process/…/input.yml.hbs (process-list error branch)
• hardware/…/input.yml.hbs (process-list error branch)
• disk/…/input.yml.hbs (process-list and disk-list error branches)
• mongod_database/…/input.yml.hbs and mongod_audit/…/input.yml.hbs (process-list error branch)

This is the identical bug class the PR sets out to fix. By the PR's own state-persistence model, on a non-200 from /processes:

Eval N: /processes → 503 → error branch → {events, want_more:false}   ← group_id dropped
outer .as(state,…): !has(state.next) → returns state unchanged
Next interval: {url} re-injected only → build /processes URL with state.group_id
→ ERROR: no such key: group_id

Tellingly, the new mock server explicitly defines non200-processes-group (main.go → handleProcessList) to return HTTP 503 on /processes — but that path lands in the unfixed branch and reproduces the exact crash the PR is about. Recommend preserving group_id/page_num/query (and cursor for mongod_*) in these first-level error branches too, so the fix is complete and consistent.

🟠 Old ~7.9 MB mock binary not removed

The description says "Replace the committed mock-server binary with Go source," but _dev/deploy/docker/mongodb_atlas/test (7,925,733 bytes) is still tracked. The Dockerfile's CMD now points at /mock_server, so this binary is dead weight — and it's still COPY'd into /data in the image. It should be git rm'd. (The two .log fixtures in that dir are still needed by handleLog and should stay.)

🟠 Description & Author's Checklist claim test coverage that no longer exists

Both the body and the checklist assert Bug 2 is "covered by test-non200-meas-config.yml system tests for disk, hardware, and process." No such files exist in the PR, and the latest commit on the branch is remove non200-meas system tests that always fail. Net effect: neither bug has automated CI coverage — Bug 1 (empty list) times out elastic-package, and the Bug 2 tests were removed. The mock server supports the scenarios, but nothing exercises them. Please reconcile the description with reality, and consider whether a pipeline test on the error-event document (or a non-timing-out system assertion) can lock in Bug 2's fix.

Minor

• Mock error shape {"error": code, "detail": detail} correctly matches the CEL branches reading body.error/body.detail. 👍
• handleLog synthetic-fallback line is a nice robustness touch.
• Digest auth checks only scheme + username substring — appropriate for a mock, and the comment says so.
• changelog.yml / manifest.yml bump to 1.3.1 (bugfix, patch) is correct.

[shmsr]

🟠 Binary and description — fixed

Agreed on both.

• Old binary (_dev/deploy/docker/mongodb_atlas/test, 7.9 MB) removed in commit 38db4aad.
• PR description updated: corrected the stale note claiming test-non200-meas-config.yml was kept (removed in 633493e), updated the WHAT summary and Author's Checklist to match.

---

🔴 Process-list / disk-list error branches — pushback

the process/host-list error branch — the first do_request in every stream — still returns bare {events, want_more: false} with no group_id

These branches are structured differently from the measurement error branches — they are wrapped in state.with(), which changes the semantics entirely.

Measurement error branch — result bound directly in the outer .as(state, …), replacing state outright (Bug 2, now fixed):

.as(state,
  request(meas_url).do_request().as(resp,
    200 ? { success }
        : { events, want_more }   // replaces state — group_id must be explicit
  )
)

Process-list error branch — result is the argument to state.with(), not a direct state replacement:

state.with(
  request(proc_url).do_request().as(resp,
    200 ? { hostlist, next, page_num }
        : { events, want_more }   // merged INTO state — group_id survives
  )
)

map.with(other) is a full merge: every key in state absent from other is preserved. state.with({events, want_more: false}) produces {group_id, page_num, query, want_more: false, events: […]}.

Eval N: /processes → 503 → error branch → {events, want_more:false} ← group_id dropped
outer .as(state,…): !has(state.next) → returns state unchanged
Next interval: {url} re-injected only → build /processes URL with state.group_id → ERROR: no such key: group_id

The step "group_id dropped" is where the trace diverges. Because the error branch is inside state.with(), the merge happens before .as(state, …) ever sees it:

Eval N — initial state: {group_id: "non200-processes-group", want_more: false, page_num: 1, query: "…"}
  has(state.hostlist) → false → fetch /processes
  /processes → 503
  error branch:             {events: [{error: {…}}], want_more: false}
  state.with(error branch): {group_id, page_num, query, want_more: false, events}  ← MERGE, not replace
  !has(state.next) → true → return merged state

Eval N+1 — state: {group_id: "non200-processes-group", page_num: 1, query, want_more: false, events}
  state.group_id present → no crash ✓

Tellingly, the new mock server explicitly defines non200-processes-group (main.go → handleProcessList) to return HTTP 503 on /processes — but that path lands in the unfixed branch and reproduces the exact crash the PR is about.

Ran this path with mito v1.27.0 against the exact control-flow from process/agent/stream/input.yml.hbs. Output from eval N:

{
  "events": [{"error": {"code": "SERVICE_UNAVAILABLE", "message": "mock: process list unavailable"}}],
  "group_id": "non200-processes-group",
  "page_num": 1,
  "query": "/measurements?granularity=PT1M&period=PT1M",
  "want_more": false
}

Eval N+1 using that state as input produces identical output — group_id present, no crash.

Same result for disk (/disks/ → 503 with host_list already populated) and mongod_database/mongod_audit (/processes → 503, group_id + cursor both preserved).

---

Why the measurement fix was necessary but the process-list branches aren't

state.with() is defined in github.com/elastic/mito@v1.27.0/lib/collections.go as:

func withAll(dst, src ref.Val) ref.Val {
    new, other, err := with(dst, src)  // new = full copy of dst
    for k, v := range other {
        new[k] = v                      // src keys overlay; dst-only keys survive
    }
    return types.NewRefValMap(...)
}

The measurement error branch sits outside any state.with() — its result is the new state verbatim. Before the fix, mito output:

{"events": [{"error": {"code": "SERVICE_UNAVAILABLE", "message": "mock: measurement unavailable"}}], "want_more": false}

group_id absent — confirmed the bug. After the fix, group_id is present. The process-list branches never had this problem; adding explicit group_id there would be redundant.

[stefans-elastic]

lgtm

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: 35786c1

History

• 💚 Build #46393 succeeded 38db4aa
• 💚 Build #46339 succeeded 633493e
• 💔 Build #46289 failed e0eb58d
• 💔 Build #46271 failed c8e85ee
• 💔 Build #46246 failed cee6bdf
• 💔 Build #46228 failed 80ff210

cc @shmsr

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[elastic-vault-github-plugin-prod]

Package mongodb_atlas - 1.3.1 containing this change is available at https://epr.elastic.co/package/mongodb_atlas/1.3.1/