[Neon Cyber] New Elastic integration neon_cyber for the Neon Cyber platform #15725

codypierce · 2025-10-22T18:09:44Z

Proposed commit message

Changes made

Initial package boilerplate
Readme updated to spec
Changelog updated to spec
Event data stream for CEL processing from Neon events API
Field definitions for events
Pipeline processor definitions
Sample log included for supported events
Detections data stream for CEL processing from Neon detections API
Field definitions for detections
Sample log included for supported detections
Pipeline, policy, and system tests for new data streams

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

[ ]

How to test this PR locally

Run pipeline and system tests

Related issues

N/A

Screenshots

cla-checker-service · 2025-10-22T18:09:49Z

💚 CLA has been signed

elasticmachine · 2025-10-27T15:05:17Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6 · 2025-10-28T23:22:45Z

packages/neon_cyber/LICENSE.txt

We don't need to add this file, it will be automatically generated.

packages/neon_cyber/_dev/build/build.yml

efd6 · 2025-10-28T23:24:04Z

packages/neon_cyber/_dev/build/docs/README.md

@@ -0,0 +1,45 @@
+{{- generatedHeader }}


What is this for?

It was in the template and adds the auto generation "Do not edit" in the rendered README.md

We don't need it.

packages/neon_cyber/changelog.yml

packages/neon_cyber/data_stream/events/elasticsearch/ingest_pipeline/default.yml

packages/neon_cyber/data_stream/detections/elasticsearch/ingest_pipeline/default.yml

packages/neon_cyber/manifest.yml

efd6

.github/CODEOWNERS needs to be updated to include this package with the owner matching the owner in the package's manifest.

packages/neon_cyber/manifest.yml

update dep ref version Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

…ering url

packages/neon_cyber/data_stream/detections/elasticsearch/ingest_pipeline/default.yml

efd6 · 2025-11-02T21:29:50Z

packages/neon_cyber/data_stream/detections/elasticsearch/ingest_pipeline/default.yml

+    - date:
+          field: json.detection_timestamp
+          target_field: neon_cyber.detections.detection_timestamp
+          tag: date_detection_timestamp
+          formats:
+              - ISO8601
+    - date:
+          field: json.detection_timestamp
+          target_field: '@timestamp'
+          tag: date_timestamp
+          formats:
+              - ISO8601


For these we might want to have on_failure handlers? How much can we tolerate the timestamps being absent?

Added on_failure just in case but these are guaranteed to be there

Sorry, I should be clearer (your interpretation is entirely reasonable, but not what I meant).

Let's say there is an invalid timestamp so the parse fails, is the document now useless (because it will not have the timestamp) or is it something that we can salvage some information from? If it's the former, we don't need to have an on_failure handler since we can just say definitively that the document had an error and we didn't try to do any further work, but if it's the latter, we can say, "there was a problem with this field, but I kept going and be aware of the error when you look at the document".

It's no doubt unlikely that there would be a corrupted timestamp, but we live in the real world, so it is always possible.

The latter would be correct. Especially for detections we wouldn't want to ignore any information. In that case is the on_failure I added correct, or is there another suggestion?

No, if that's the semantics, what you've done is right.

I would say though that the json processor failure would be unrecoverable; if that fails there's no point in doing any further work since we can guarantee that the fields will be absent.

Got it, committed a change to remove json on_failure

efd6 · 2025-11-02T21:30:53Z

packages/neon_cyber/manifest.yml

+                - name: enable_request_tracer
+                  type: bool
+                  title: Enable request tracing
+                  default: false
+                  multi: false
+                  required: false
+                  show_user: false
+                  description: The request tracer logs requests and responses to the agent's local file-system for debugging configurations. Enabling this request tracing compromises security and should only be used for debugging. See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-cel.html#_resource_tracer_filename) for details.


This should be at the data stream level.

moved to DS

efd6 · 2025-11-02T21:34:51Z

packages/neon_cyber/data_stream/detections/manifest.yml

+                i.e. scheme://host:port/path
+            required: true
+            show_user: true
+            default: https://api.neoncyber.io/v1/detections


For the sake of user ergonomics, is it the case that the detections endpoint is always at /v1/detections? (similarly for events at /v1/events). If it is, I would suggest hard coding the endpoint path into the CEL program and making the base URL a package level configuration next to the API token field. This would mean that that user only needs to configure it once (assuming that the base URL is almost constant for a user).

Moved URL to package and hardcoded and removed resource_url

efd6 · 2025-11-02T21:37:55Z

packages/neon_cyber/data_stream/detections/agent/stream/cel.yml.hbs

+              ),
+            },
+          },
+          "want_more": false,


This is not necessary.

Any time I remove this "want_more" line "elastic_package test system" fails to get a hit on the mock server and I don't know why.

OK. I will take a look and see what is going on.

Thank you for your help! I couldn't figure out how that works since on paper it seem extraneous but the system test definitely prevents it from completing if that single line is removed.

The issue is identified in the error message that unfortunately only gets shown in the logs

{ "log.level": "error", "@timestamp": "2025-11-05T22:21:36.369Z", "log.origin": { "function": "github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange", "file.name": "coordinator/coordinator.go", "file.line": 800 }, "message": "Unit state changed cel-default-cel-neon_cyber-50f5cedf-f9ea-481d-af64-5a5da5a63c74 (STARTING->FAILED): failed to check program: failed compilation: ERROR: <input>:12:54: found no matching overload for '_?_:_' applied to '(bool, map(string, list(map(string, string))), map(string, map(string, map(string, string))))'\n | ).do_request().as(resp, (resp.StatusCode == 200) ?\n | .....................................................^ accessing config", "log": { "source": "elastic-agent" }, "component": { "id": "cel-default", "state": "HEALTHY" }, "unit": { "id": "cel-default-cel-neon_cyber-50f5cedf-f9ea-481d-af64-5a5da5a63c74", "type": "input", "state": "FAILED", "old_state": "STARTING" }, "ecs.version": "1.6.0" }

extracting the relevant part

found no matching overload for '_?_:_' applied to '(bool, map(string, list(map(string, string))), map(string, map(string, map(string, string))))' | ).do_request().as(resp, (resp.StatusCode == 200) ? | .....................................................^ accessing config

What this is saying is that the ternary has branches that are not the same type, map(string, list(map(string, string))) and map(string, map(string, map(string, string))). They need to agree, and we can achieve this by making them both dyn types so that the compile-time types are the same even though the runtime types are not identical.

This is done by (with some other cosmetic changes):

diff --git a/packages/neon_cyber/data_stream/detections/agent/stream/cel.yml.hbs b/packages/neon_cyber/data_stream/detections/agent/stream/cel.yml.hbs index ea6c2524d4..efa226430e 100644 --- a/packages/neon_cyber/data_stream/detections/agent/stream/cel.yml.hbs +++ b/packages/neon_cyber/data_stream/detections/agent/stream/cel.yml.hbs @@ -34,31 +34,34 @@ program: |- ).do_request().as(resp, (resp.StatusCode == 200) ? resp.Body.decode_json().as(body, { - "events": (has(body.data) && body.data.size() > 0) ? - body.data.map(e, - { - "message": e.encode_json(), - } - ) - : - [] + "events": dyn( + (has(body.data) && body.data.size() > 0) ? + body.data.map(e, + { + "message": e.encode_json(), + } + ) + : + [] + ) } ) : { - "events": { - "error": { - "code": string(resp.StatusCode), - "id": string(resp.Status), - "message": "GET " + state.url + ( - (size(resp.Body) != 0) ? - string(resp.Body) - : - string(resp.Status) + " (" + string(resp.StatusCode) + ")" - ), - }, - }, - "want_more": false, + "events": dyn( + { + "error": { + "code": string(resp.StatusCode), + "id": string(resp.Status), + "message": "GET " + state.url.trim_right("/") + "/v1/detections: " + ( + (size(resp.Body) != 0) ? + string(resp.Body) + : + string(resp.Status) + " (" + string(resp.StatusCode) + ")" + ), + }, + } + ) } ) ) diff --git a/packages/neon_cyber/data_stream/events/agent/stream/cel.yml.hbs b/packages/neon_cyber/data_stream/events/agent/stream/cel.yml.hbs index 40341dcf6b..54cb70b0ac 100644 --- a/packages/neon_cyber/data_stream/events/agent/stream/cel.yml.hbs +++ b/packages/neon_cyber/data_stream/events/agent/stream/cel.yml.hbs @@ -34,31 +34,34 @@ program: |- ).do_request().as(resp, (resp.StatusCode == 200) ? resp.Body.decode_json().as(body, { - "events": (has(body.data) && body.data.size() > 0) ? - body.data.map(e, - { - "message": e.encode_json(), - } - ) - : - [] + "events": dyn( + (has(body.data) && body.data.size() > 0) ? + body.data.map(e, + { + "message": e.encode_json(), + } + ) + : + [] + ) } ) : { - "events": { - "error": { - "code": string(resp.StatusCode), - "id": string(resp.Status), - "message": "GET " + state.url + ( - (size(resp.Body) != 0) ? - string(resp.Body) - : - string(resp.Status) + " (" + string(resp.StatusCode) + ")" - ), - }, - }, - "want_more": false, + "events": dyn( + { + "error": { + "code": string(resp.StatusCode), + "id": string(resp.Status), + "message": "GET " + state.url.trim_right("/") + "/v1/events: " + ( + (size(resp.Body) != 0) ? + string(resp.Body) + : + string(resp.Status) + " (" + string(resp.StatusCode) + ")" + ), + }, + } + ) } ) )

Thanks! Updated to your suggestion and system tests are passing now.

…t_pipeline/default.yml Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

…g to DS

efd6

Minor changes only, otherwise looking good.

efd6 · 2025-11-05T22:39:26Z

packages/neon_cyber/data_stream/detections/_dev/test/system/test-default-config.yml

+service: neon_cyber
+vars:
+    api_token: xxxx
+    enable_request_tracer: true


This needs to be moved to match the new manifest layout. Same in the other data stream.

efd6 · 2025-11-05T22:44:08Z

packages/neon_cyber/data_stream/detections/_dev/test/system/test-default-config.yml

+    enable_request_tracer: true
+data_stream:
+    vars:
+        resource_url: http://{{Hostname}}:{{Port}}/v1/detections


This will need to be moved to the package level and changed to url (at the moment the test is reaching out to api.neoncyber.io and failing (correctly) with a 401).

{ "log.level": "debug", "@timestamp": "2025-11-05T22:40:13.138Z", "message": "HTTP request", "transaction.id": "S6QK57BN7LQHG-1", "url.original": "https://api.neoncyber.io/v1/detections", "url.scheme": "https", "url.path": "/v1/detections", "url.domain": "api.neoncyber.io", "url.port": "", "url.query": "", "http.request.method": "GET", "http.request.header": { "Authorization": [ "xxxx" ], "User-Agent": [ "Elastic-Filebeat/8.19.3 (linux; amd64; 7a508104ea6b6ddae7456dff8b61423e59a73962; 2025-08-26 01:12:39 +0000 UTC)" ] }, "user_agent.original": "Elastic-Filebeat/8.19.3 (linux; amd64; 7a508104ea6b6ddae7456dff8b61423e59a73962; 2025-08-26 01:12:39 +0000 UTC)", "http.request.body.content": "", "http.request.body.truncated": false, "http.request.body.bytes": 0, "http.request.mime_type": "", "ecs.version": "1.6.0" } { "log.level": "debug", "@timestamp": "2025-11-05T22:40:14.281Z", "message": "HTTP response", "transaction.id": "S6QK57BN7LQHG-1", "http.response.status_code": 401, "http.response.body.content": "{\"error\":{\"code\":401,\"message\":\"Unauthorized\"},\"data\":null}", "http.response.body.truncated": false, "http.response.body.bytes": 59, "http.response.mime_type": "application/json; charset=utf-8", "http.response.header": { "Apigw-Requestid": [ "Tl3MPgXKIAMEJbw=" ], "Content-Length": [ "59" ], "Content-Type": [ "application/json; charset=utf-8" ], "Date": [ "Wed, 05 Nov 2025 22:40:14 GMT" ], "Etag": [ "W/\"3b-B6EKjFN6TdKYf7gDSWml69abDvE\"" ], "Server": [ "envoy" ], "X-Envoy-Upstream-Service-Time": [ "159" ] }, "ecs.version": "1.6.0" }

Good catch. My bad.

efd6 · 2025-11-05T22:51:50Z

/test

efd6 · 2025-11-05T22:52:23Z

/test

efd6

You will need to run

elastic-package test system -g
elastic-package test pipeline -g
elastic-package test policy -g
elastic-package build

using a v8.18.0 stack (the version that is the minimum specified in the manifest)

efd6 · 2025-11-06T01:33:33Z

packages/neon_cyber/data_stream/events/elasticsearch/ingest_pipeline/default.yml

+          field: json.url
+          target_field: neon_cyber.events.url
+          ignore_missing: true
+          tag: rename_events_url


- uri_parts: field: neon_cyber.events.url ignore_missing: true tag: uri_parts_events_url

efd6 · 2025-11-06T01:33:53Z

packages/neon_cyber/data_stream/detections/elasticsearch/ingest_pipeline/default.yml

+          field: json.url
+          target_field: neon_cyber.detections.url
+          ignore_missing: true
+          tag: rename_detection_url


- uri_parts: field: neon_cyber.detections.url ignore_missing: true tag: uri_parts_detections_url

…parts

codypierce · 2025-11-06T04:26:28Z

Switched manifest version to match testing config version 8.17.0. Regenerated expected and samples

efd6 · 2025-11-06T04:30:59Z

/test

elastic-vault-github-plugin-prod · 2025-11-06T04:54:21Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2025-11-06T04:54:24Z

💚 Build Succeeded

Buildkite Build
Commit: 44ff5cb

History

💔 Build #33744 failed 800f8fe

efd6 · 2025-11-06T05:03:13Z

Quick query: where were the test sample obtained from? Live API with modification? Constructed from documentation?

codypierce · 2025-11-06T05:05:52Z

Quick query: where were the test sample obtained from? Live API with modification? Constructed from documentation?

Live API for a test account and modifications

efd6

Thanks

elastic-vault-github-plugin-prod · 2025-11-06T05:29:42Z

Package neon_cyber - 0.1.0 containing this change is available at https://epr.elastic.co/package/neon_cyber/0.1.0/

…atform (elastic#15725) The initial release includes detections and events data streams. Neon Cyber fields are mapped to their corresponding ECS fields where possible. Test samples were derived from a live API using a test account.

codypierce added 2 commits October 21, 2025 21:17

Initial commit

91c3a1b

Merge branch 'elastic:main' into main

8ff92a6

codypierce requested a review from a team as a code owner October 22, 2025 18:09

codypierce added 2 commits October 22, 2025 11:12

pr link

d94c264

Merge branch 'main' of github.com:neoncyberinc/elastic-integrations

da92af3

andrewkroh added documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. New Integration Issue or pull request for creating a new integration package. labels Oct 22, 2025

narph added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Oct 27, 2025

Merge branch 'main' into main

6fb3e49

efd6 reviewed Oct 29, 2025

View reviewed changes

packages/neon_cyber/manifest.yml Outdated Show resolved Hide resolved

codypierce and others added 6 commits October 31, 2025 10:40

Update packages/neon_cyber/_dev/build/build.yml

ed1b5d4

update dep ref version Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Update packages/neon_cyber/changelog.yml

f050be7

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

pr fixes

99eef00

Merge branch 'main' of github.com:neoncyberinc/elastic-integrations

f1b1027

added back want_more for system test. Generate expected. Cleanup ling…

8a5efed

…ering url

Merge branch 'main' into main

b9641b8

efd6 reviewed Nov 2, 2025

View reviewed changes

codypierce and others added 4 commits November 5, 2025 10:34

Update packages/neon_cyber/data_stream/detections/elasticsearch/inges…

61c5374

…t_pipeline/default.yml Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

added back package url, removed DS resource_url, moved request_tracin…

28ab687

…g to DS

Merge branch 'main' into main

54c7adf

cast to dyn map and remove want_more

ad80196

efd6 reviewed Nov 5, 2025

View reviewed changes

moved config var resource_url to url and moved tracer to DS vars

bc047e3

remove json on_failure

800f8fe

efd6 reviewed Nov 6, 2025

View reviewed changes

changed manifest version, generate tests samples for 8.17.0, add uri_…

44ff5cb

…parts

efd6 approved these changes Nov 6, 2025

View reviewed changes

efd6 merged commit 1ca737d into elastic:main Nov 6, 2025
7 checks passed

andrewkroh added the Integration:neon_cyber Neon Cyber (Partner supported) label Nov 6, 2025

[Neon Cyber] New Elastic integration neon_cyber for the Neon Cyber platform #15725

[Neon Cyber] New Elastic integration neon_cyber for the Neon Cyber platform #15725

Uh oh!

Conversation

codypierce commented Oct 22, 2025

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

cla-checker-service bot commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Oct 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efd6 Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

cla-checker-service bot commented Oct 22, 2025 •

edited

Loading

efd6 Nov 5, 2025 •

edited

Loading

codypierce commented Nov 6, 2025 •

edited

Loading