[sentinel_one] Migrate Threat Data stream to CEL by mohitjha-elastic · Pull Request #19881 · elastic/integrations

mohitjha-elastic · 2026-06-30T10:15:38Z

Proposed commit message

sentinel_one: migrate threat data stream to CEL

This change migrates the SentinelOne threat data stream to run under the 
Common Expression Language (CEL) engine by enabling the run_as_cel option. 
With this configuration, the existing httpjson input is transparently executed by the
CEL input, while preserving the current input type.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Clone integrations repo.
Install the elastic package locally.
Start the elastic stack using the elastic package.
Move to integrations/packages/sentinel_one directory.
Run the following command to run tests.

elastic-package test -v

infra-vault-gh-plugin-prod · 2026-06-30T10:15:43Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

github-actions · 2026-06-30T10:17:24Z

Elastic Docs Style Checker (Vale)

Summary: 2 suggestions found

💡 Suggestions (2): Optional style improvements. Apply when helpful.

File	Line	Rule	Message
packages/sentinel_one/data_stream/threat/manifest.yml	29	Elastic.WordChoice	Consider using 'deactivate, deselect, hide, turn off' instead of 'Disable', unless the term is in the UI.
packages/sentinel_one/data_stream/threat/manifest.yml	38	Elastic.WordChoice	Consider using 'can, might' instead of 'may', unless the term is in the UI.

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

mohitjha-elastic · 2026-06-30T10:17:25Z

Hold off on merging the PR until the public release of 8.19.18.

vera-review-bot · 2026-06-30T10:19:16Z

👀 I have started reviewing the PR

vera-review-bot · 2026-06-30T10:37:07Z

+              }
+            :
+              {
+                "events": [


🟡 MEDIUM data_stream/threat/.../httpjson.yml.hbs:167

Array-shaped timeline errors indexed as malformed threat docs

On a timeline fetch that returns a status other than 200 or 404, the CEL program emits an array-shaped error event ("events": [{"error": {"code", "id", "message"}}]). Array-shaped errors use advance semantics: the event is passed to the ingest pipeline and indexed. The threat pipeline has no terminate processor (ES 8.16.0+) and no top-of-pipeline ctx.error guard, so this error event flows through as a normal document: it has no message/json, so the parsing processors are skipped, and it lands in the threat data stream as a document carrying only error.code/error.id/error.message, event.kind: alert, and ecs.version. The pipeline-level on_failure does not catch it because no processor throws. This is a new code path introduced by the CEL migration — the previous httpjson stream never delivered transport errors as documents. The single-object list-fetch error path is correct (retry/halt); only the array timeline-error path needs pipeline support.

Recommendation:

Add a guard near the top of the pipeline (after event.original is set) so CEL-emitted error events are surfaced and stop processing instead of being indexed as threat docs:

- terminate: tag: terminate_on_cel_error if: ctx.error?.message != null && ctx.json == null description: Stop processing CEL-emitted error events so they are not indexed as malformed threat documents.

Alternatively, if these timeline failures should not advance past the threat, switch the non-200/non-404 timeline branch to the single-object error shape ("events": {"error": {...}}, "want_more": false) so the agent retries from the last good cursor rather than emitting an indexable error document.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

…sentinel_one-2.9.0

efd6

I think this probably should have been two PRs, one for the run_as_cel and one for the timeline enrichment.

efd6 · 2026-07-01T06:46:49Z

Please rename this file to httpjson_as_cel.yml.hbs

efd6 · 2026-07-01T06:47:32Z

+    )
+  )
 tags:
 {{#if preserve_original_event}}


Suggested change

{{#if preserve_original_event}}

- run_as_cel

{{#if preserve_original_event}}

efd6 · 2026-07-01T06:49:21Z

      field: message
      value: 'Threat Detected: {{{sentinel_one.threat.name}}} ({{{sentinel_one.threat.confidence_level}}})'
-      if: ctx.sentinel_one?.threat.name != null && ctx.sentinel_one?.threat?.confidence_level != null
+      if: ctx.sentinel_one?.threat?.name != null && ctx.sentinel_one?.threat?.confidence_level != null


Is this a bug-fix?

Yes, I'll separate this with the threat timeline event enrichment into a follow-up PR.

elastic-vault-github-plugin-prod · 2026-07-01T07:05:46Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

github-actions · 2026-07-03T10:21:06Z

TL;DR

Check integrations sentinel_one is failing in the new script test: upgrade in sentinel_one.threat. The upgraded stream does not ingest threat-...0003 because the phase-2 mock only matches updatedAt__gte=2024-01-15T10:01.*, while the preserved 2.8.2 cursor should advance past the second phase-1 threat (2024-01-15T10:06:00Z).

Remediation

In packages/sentinel_one/data_stream/threat/_dev/test/scripts/upgrade.txt, change the phase-2 mock expectation/comment from 2024-01-15T10:01.* to the actual preserved cursor value after the two ascending phase-1 events, 2024-01-15T10:06.*.
Tighten the post-upgrade assertion so an indexed CEL error event cannot satisfy get_docs -want 3; for example, assert three docs where sentinel_one.threat.threat_id is present and assert no error.message documents.
Re-run the SentinelOne package script/system tests for the threat data stream.

Investigation details

Root Cause

The failing assertion is in packages/sentinel_one/data_stream/threat/_dev/test/scripts/upgrade.txt:49: it expects the post-upgrade IDs to include threat-00000000000000000000000000000003.

The phase-2 mock in the same new script only serves that third threat when the request has this query param:

packages/sentinel_one/data_stream/threat/_dev/test/scripts/upgrade.txt:102-107
# Cursor preserved if the agent sends updatedAt__gte
# from the prior last_update_at (10:01Z).
query_params:
  updatedAt__gte: "{updatedAtGte:2024-01-15T10:01.*}"

That expected cursor is inconsistent with the phase-1 data and the previous httpjson cursor behavior. Phase 1 returns two threats sorted by updatedAt ascending: 10:01Z and 10:06Z. The pre-CEL template sorts ascending and stores [[.last_event.threatInfo.updatedAt]], so after both events the cursor should be the second event (2024-01-15T10:06:00Z), not 10:01Z.

The run then gets to three total documents only because CEL indexes a 404 error event from the unmatched phase-2 mock. The final jq output confirms the third document is not a threat document: it prints a leading empty/null value followed by only the first two threat IDs.

Evidence

Build: https://buildkite.com/elastic/integrations/builds/45459
Job/step: Check integrations sentinel_one
Key log excerpt:

script test: upgrade in sentinel_one.threat
upgrade.txt:49: no match for `"threat-00000000000000000000000000000001,threat-00000000000000000000000000000002,threat-00000000000000000000000000000003"` found in stdout
...
error.message: "GET (s1mock/redacted) 404 Not Found (404)"
...
> exec jq '[.hits.hits[]._source.sentinel_one.threat.threat_id] | unique | sort | join(",")' got_docs_post_upgrade.json
[stdout]
",threat-00000000000000000000000000000001,threat-00000000000000000000000000000002"

Verification

Confirmed the Buildkite JUnit annotation reports one failure: script test: upgrade in sentinel_one.threat.
Compared the new upgrade fixture with the old threat httpjson cursor template semantics.
Not run locally: this script requires the Docker-backed elastic-package stack, which is unavailable in this environment.

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

mohitjha-elastic · 2026-07-05T17:15:21Z

I think this probably should have been two PRs, one for the run_as_cel and one for the timeline enrichment.

@efd6 The PR has been updated to include only the changes required to migrate the threat data stream to CEL. I'll raise a follow-up PR for the threat timeline event enrichment changes.

vera-review-bot · 2026-07-05T17:19:08Z

No issues across the latest commits c7b8296.

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

efd6

nit, but LGTM

efd6 · 2026-07-05T22:21:51Z

+          "sortBy": ["updatedAt"],
+          "sortOrder": ["asc"],
+          "updatedAt__gte": [list_updated_at_gte],
+          ?"siteIds": state.?site_ids.optMap(v, [string(v)]),


Do we need the string conversion give that you coerce to string in the YAML?

I can see its not required. Thanks!

mergify · 2026-07-05T22:23:19Z

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

Queue this pull request

elastic-vault-github-plugin-prod · 2026-07-06T06:34:22Z

✅ All changelog entries have the correct PR link.

infra-vault-gh-plugin-prod · 2026-07-06T07:16:57Z

💚 Build Succeeded

Buildkite Build
Commit: 58490d2

History

💚 Build #45489 succeeded c7b8296
💔 Build #45459 failed 58928f9
💚 Build #45292 succeeded 0bb2935
💔 Build #45291 failed 4436ef9
💔 Build #45248 failed 567268c

cc @mohitjha-elastic

elastic-vault-github-plugin-prod · 2026-07-06T08:45:44Z

Package sentinel_one - 2.9.0 containing this change is available at https://epr.elastic.co/package/sentinel_one/2.9.0/

mohitjha-elastic · 2026-07-06T10:22:55Z

Here is the follow up PR for optional threat timeline collection - #19990

Migrate the threat data stream to CEL to support threat timeline

779a6c8

mohitjha-elastic self-assigned this Jun 30, 2026

mohitjha-elastic requested review from a team as code owners June 30, 2026 10:15

Update changelog entry

567268c

vera-review-bot Bot reviewed Jun 30, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

mohitjha-elastic added 2 commits July 1, 2026 11:47

Fix policy test

4436ef9

Merge branch 'main' of github.com:mohitjha-elastic/integrations into …

0bb2935

…sentinel_one-2.9.0

This comment has been minimized.

Sign in to view

efd6 reviewed Jul 1, 2026

View reviewed changes

elastic deleted a comment from vera-review-bot Bot Jul 1, 2026

Remove threat timeline events changes

58928f9

mohitjha-elastic changed the title ~~[sentinel_one] Migrate Threat Data stream to CEL and Enrich Timeline Events~~ [sentinel_one] Migrate Threat Data stream to CEL Jul 3, 2026

andrewkroh removed the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jul 3, 2026

fix ci failure

c7b8296

mohitjha-elastic requested a review from efd6 July 5, 2026 17:15

efd6 approved these changes Jul 5, 2026

View reviewed changes

Removed string conversion from siteID filter

58490d2

mohitjha-elastic merged commit cce3b37 into elastic:main Jul 6, 2026
10 checks passed

mohitjha-elastic deleted the sentinel_one-2.9.0 branch July 6, 2026 08:27

	{{#if preserve_original_event}}
	- run_as_cel
	{{#if preserve_original_event}}

Uh oh!

Conversation

mohitjha-elastic commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jun 30, 2026

Uh oh!

github-actions Bot commented Jun 30, 2026

Elastic Docs Style Checker (Vale)

Uh oh!

mohitjha-elastic commented Jun 30, 2026

Uh oh!

vera-review-bot Bot commented Jun 30, 2026

Uh oh!

vera-review-bot Bot Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

efd6 Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

efd6 Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

efd6 Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

mohitjha-elastic Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Uh oh!

github-actions Bot commented Jul 3, 2026

TL;DR

Remediation

Root Cause

Evidence

Verification

Uh oh!

mohitjha-elastic commented Jul 5, 2026

Uh oh!

vera-review-bot Bot commented Jul 5, 2026

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

efd6 Jul 5, 2026

Choose a reason for hiding this comment

Uh oh!

mohitjha-elastic Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Jul 5, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jul 6, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jul 6, 2026

💚 Build Succeeded

History

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jul 6, 2026

Uh oh!

mohitjha-elastic commented Jul 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

mohitjha-elastic commented Jun 30, 2026 •

edited

Loading

elastic-vault-github-plugin-prod Bot commented Jul 1, 2026 •

edited

Loading