network_traffic: add ML module for packetbeat anomaly detection - #20825
network_traffic: add ML module for packetbeat anomaly detection#20825efd6 wants to merge 1 commit into
Conversation
efd6
added
Integration:network_traffic
✅ Elastic Docs Style Checker (Vale)No issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
The Kibana security_packetbeat ML module's datafeed queries for packetbeat_dns_tunneling, packetbeat_rare_dns_question, and packetbeat_rare_user_agent filter on event.dataset. The network_traffic compatibility pipelines remove that field, so those jobs match no documents. Add a kibana/ml_module defining all five packetbeat anomaly detection jobs with corrected datafeed queries: - packetbeat_dns_tunneling_ea: filter on data_stream.dataset instead of event.dataset; fix runtime mapping script to reference network_traffic.dns.question.etld_plus_one - packetbeat_rare_dns_question_ea: filter on data_stream.dataset instead of event.dataset - packetbeat_rare_user_agent_ea: filter on data_stream.dataset instead of event.dataset - packetbeat_rare_server_domain_ea: no query change needed - packetbeat_rare_urls_ea: no query change needed Fixes elastic#18111
43e6215 to
561383a
Compare
|
✅ All changelog entries have the correct PR link. |
🚀 Benchmarks reportTo see the full report comment with |
💚 Build Succeeded
cc @efd6 |
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
![vera-review-bot[bot]](https://avatars.githubusercontent.com/in/3297022?s=60&v=4){kind=small}
| } | ||
| }, | ||
| { | ||
| "id": "packetbeat_rare_server_domain_ea", |
There was a problem hiding this comment.
Severity: 🟠 High confidence: medium path: packages/network_traffic/kibana/ml_module/network_traffic-ml.json:114
Three of the five job IDs in this module (packetbeat_rare_server_domain_ea, packetbeat_rare_urls_ea, packetbeat_rare_user_agent_ea) are the exact IDs Kibana's built-in security_packetbeat module already installs, so whichever installer runs first wins and the corrected queries may never be applied. Drop those three from this module and keep only the two DNS jobs Kibana no longer ships.
Details
Kibana main still ships the security_packetbeat data-recognizer module (x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/security_packetbeat/manifest.json). Its manifest declares exactly three jobs: packetbeat_rare_server_domain_ea, packetbeat_rare_urls_ea, and packetbeat_rare_user_agent_ea - the same IDs defined here at lines 114, 148 and 181.
Anomaly detection job IDs are cluster-global, not module-scoped. On a cluster where an operator has already run the built-in Security: Packetbeat module, setting up this package's module will hit resource_already_exists_exception for those three jobs, and the pre-existing job keeps its original datafeed. The practical effect is that the datafeed corrections this PR exists to deliver (for example the data_stream.dataset: network_traffic.http filter at line 431, which replaces the built-in module's event.dataset should-clause that cannot match this package's documents because the dns/http compatibility pipelines remove event.dataset) silently do not take effect. In the reverse order the built-in module setup fails instead.
By contrast packetbeat_dns_tunneling_ea and packetbeat_rare_dns_question_ea are NOT in that manifest, yet prebuilt rules in packages/security_detection_engine still reference them (for example 746edc4c-c54c-49c6-97a1-651223819448_208.json -> packetbeat_rare_dns_question_ea). Those two are genuinely orphaned and this module is the right place to restore them.
Note that simply renaming the three overlapping IDs is not a fix on its own: security_detection_engine rules bind to them by literal ID (17e68559-b274-4948-ad0b-f8415bb31126_211.json -> packetbeat_rare_server_domain_ea, 91f02f01-969f-4167-8d77-07827ac4cee0_208.json -> packetbeat_rare_user_agent_ea), so renaming would orphan those rules instead.
Recommendation:
Scope this module to the two jobs Kibana no longer owns, and land the data_stream.dataset datafeed corrections for the other three in the Kibana security_packetbeat module so there is a single owner per job ID:
"jobs": [
{ "id": "packetbeat_dns_tunneling_ea", "config": { "...": "unchanged" } },
{ "id": "packetbeat_rare_dns_question_ea", "config": { "...": "unchanged" } }
],
"datafeeds": [
{ "id": "datafeed-packetbeat_dns_tunneling_ea", "job_id": "packetbeat_dns_tunneling_ea", "config": { "...": "unchanged" } },
{ "id": "datafeed-packetbeat_rare_dns_question_ea", "job_id": "packetbeat_rare_dns_question_ea", "config": { "...": "unchanged" } }
]If the intent really is for this package to supersede the built-in module, say so in the PR description and pair this change with a Kibana PR removing the three jobs from security_packetbeat/manifest.json, so the two never coexist.
🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| "id": "datafeed-packetbeat_dns_tunneling_ea", | ||
| "job_id": "packetbeat_dns_tunneling_ea", | ||
| "config": { | ||
| "indices": [ |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/network_traffic/kibana/ml_module/network_traffic-ml.json:219
None of the five datafeed configs set max_empty_searches, which the Kibana module these queries were copied from sets to 10. Add "max_empty_searches": 10 to each datafeed config so real-time datafeeds stop instead of searching forever on clusters with no packetbeat data.
Details
Every datafeed config in this file (lines 218, 275, 328, 366 and 415) omits max_empty_searches. The Kibana security_packetbeat datafeeds these were derived from set "max_empty_searches": 10 - datafeed_packetbeat_rare_server_domain_ea.json, datafeed_packetbeat_rare_urls_ea.json and datafeed_packetbeat_rare_user_agent_ea.json all carry it, and apart from this field and the dataset filter the queries here are otherwise identical to upstream.
When max_empty_searches is unset the setting defaults to no limit, so a started real-time datafeed keeps issuing searches indefinitely even when the index never yields a document. That is the common case here: a user can install network_traffic and enable only, say, the DNS protocol, leaving the http/tls-backed jobs permanently empty. Upstream sets 10 specifically so those datafeeds stop themselves.
Recommendation:
Add the setting to each of the five datafeed configs, matching the upstream module:
"config": {
"max_empty_searches": 10,
"indices": [
"INDEX_PATTERN_NAME"
],
"job_id": "packetbeat_dns_tunneling_ea",
"query": {
"...": "unchanged"
}
}🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| - version: "1.34.4" | ||
| changes: | ||
| - description: Add ML modules with corrected datafeed queries for network traffic anomaly detection jobs. | ||
| type: bugfix |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/network_traffic/changelog.yml:5
This PR adds a brand-new asset file with five new ML jobs but records it as type: bugfix with a patch bump to 1.34.4. Change the entry to type: enhancement and bump to 1.35.0 in both changelog.yml and manifest.yml.
Details
packages/network_traffic/kibana/ml_module/network_traffic-ml.json is a new file (the diff shows it as added, mode new file), and it ships five anomaly detection jobs the package did not previously install. That is new user-visible functionality, not a fix to shipped behaviour.
This package's own changelog follows the repo convention consistently: enhancements take a minor bump (1.33.0 "Add 9.0.0 constraint", 1.34.0 "Use links panel in Dashboards") while bugfixes take a patch (1.34.1, 1.34.2). Labelling a new-asset addition bugfix also makes the release notes misleading for anyone deciding whether 1.34.4 is a safe patch upgrade.
The description text also says "Add ML modules" (plural) while a single module is added, and "with corrected datafeed queries" reads as though existing queries were being fixed in this package - nothing in this package previously defined datafeed queries.
Recommendation:
Retitle the entry, switch the type, and take a minor bump in both files:
# packages/network_traffic/changelog.yml
- version: "1.35.0"
changes:
- description: Add an ML module providing packetbeat anomaly detection jobs.
type: enhancement
link: https://github.com/elastic/integrations/pull/20825# packages/network_traffic/manifest.yml
version: "1.35.0"🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
Review summaryIssues found across the latest commits 561383a — 1 high, 2 medium
🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
|
1 participant
Proposed commit message
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Related issues
Screenshots