[axonius][exposure] Add Axonius Exposure datastream#16599
Conversation
muskan-agarwal26
changed the title
andrewkroh
added
New Integration
muskan-agarwal26
force-pushed
the
datastream-exposure
branch
from
df30389 to
a3d90f1
Compare
ShourieG
left a comment
ShourieG
left a comment
There was a problem hiding this comment.
🤖 AI-Generated Review | Elastic Integration PR Review Bot
⚠️ This is an automated review generated by an AI assistant. Please verify all suggestions before applying changes. This review does not represent a human reviewer's opinion.
🔍 Pipeline Review Summary
Files Reviewed:
packages/axonius/data_stream/exposure/elasticsearch/ingest_pipeline/default.ymlpackages/axonius/data_stream/exposure/elasticsearch/ingest_pipeline/pipeline-vulnerability-instance.yml
Integration: axonius
Data Stream: exposure
✅ Compliant Items
- Description: Both pipelines include a clear
descriptionfield. - Logical Processor Order: Parsing, extraction, transformation, and cleanup steps are present and ordered logically.
- Temporary Field Cleanup: Temporary fields (e.g.,
message,json) are removed at the end of the pipeline. - ECS Categorization:
event.kindis set toevent.event.categoryis set usingappendas an array (e.g.,vulnerability).event.typeis set usingappendas an array (e.g.,info).
- @timestamp Extraction: Timestamp is extracted and set from parsed date fields.
- Error Handling:
- Most critical processors (date, convert) use
on_failurehandlers to append error messages. ignore_missingis used on mostrenameandconvertprocessors.
- Most critical processors (date, convert) use
- Field Mapping: Axonius fields are mapped to ECS fields where possible (e.g.,
vulnerability.id,vulnerability.description). - No CDR Requirements: Axonius is not a cloud security integration, so CDR fields are not required and not flagged.
⚠️ Issues Found
Issue 1: Missing Pipeline-Level on_failure Handler
Severity: 🔴 High
Location: packages/axonius/data_stream/exposure/elasticsearch/ingest_pipeline/default.yml line 875
Problem: The pipeline does not include a top-level on_failure handler. This is required to ensure that any unhandled errors are captured and categorized as pipeline errors.
Recommendation:
on_failure:
- set:
field: error.message
value: 'Pipeline processing failed: {{{_ingest.on_failure_message}}}'
- set:
field: event.kind
value: pipeline_errorIssue 2: event.category and event.type Should Be Arrays
Severity: 🟡 Medium
Location: packages/axonius/data_stream/exposure/elasticsearch/ingest_pipeline/default.yml line ~20
Problem: The pipeline uses append to set event.category and event.type, but if the field is not initialized as an array, this can result in a string value instead of an array, violating ECS requirements.
Recommendation:
- set:
field: event.category
value: ['vulnerability']
tag: set_event_category
- set:
field: event.type
value: ['info']
tag: set_event_typeAlternatively, ensure the field is always an array after append.
Issue 3: Some convert and date Processors Missing ignore_failure
Severity: 🟡 Medium
Location: packages/axonius/data_stream/exposure/elasticsearch/ingest_pipeline/pipeline-vulnerability-instance.yml various lines
Problem: While most processors use on_failure, some convert and date processors do not include ignore_failure: true, which is best practice to prevent pipeline errors from type mismatches.
Recommendation:
- convert:
field: json.event.data.axonius_risk_score
tag: convert_event_data_axonius_risk_score_to_double
target_field: axonius.exposure.event.data.axonius_risk_score
type: double
ignore_missing: true
ignore_failure: true
on_failure:
- append:
field: error.message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'Add ignore_failure: true to all convert and date processors.
Issue 4: No Null Checks in Some Conditionals
Severity: 🟡 Medium
Location: packages/axonius/data_stream/exposure/elasticsearch/ingest_pipeline/default.yml various lines
Problem: Some if conditions use ctx.json.field without null checks, which can cause errors if the field is missing.
Recommendation:
if: ctx.json?.event?.data?.added != null && ctx.json.event.data.added != ''Ensure all conditionals use safe navigation (?.) and null checks.
💡 Suggestions for Improvement
- Use
setfor ECS Arrays: Prefersetwith array values for ECS fields overappendunless you are merging multiple values. - Field Namespacing: Continue to use the
axonius.exposure.*namespace for custom fields to avoid ECS conflicts. - Performance: Consider using native processors over scripts wherever possible (currently compliant).
- Documentation: Add comments in the pipeline for complex transformations for future maintainers.
📋 Checklist Status
- ECS categorization complete
- Missing pipeline-level
on_failurehandler - Temporary fields cleaned up
- @timestamp extraction present
- Error handling on most critical processors
- Some processors missing
ignore_failure - No CDR requirements (not a cloud security integration)
- Processor order logical
- Field mapping ECS-compliant
Summary:
The Axonius exposure ingest pipelines are well-structured and mostly ECS-compliant, with good error handling and field mapping. The main issues are the lack of a pipeline-level on_failure handler, potential ECS array violations for categorization fields, and missing ignore_failure on some processors. Addressing these will ensure robust, maintainable, and ECS-compliant pipelines.
| - append: | ||
| field: tags | ||
| value: preserve_original_event | ||
| allow_duplicates: false |
There was a problem hiding this comment.
🤖 AI Review: 🔴 Missing Pipeline-Level on_failure Handler
The pipeline does not include a top-level on_failure handler. This is required to ensure that any unhandled errors are captured and categorized as pipeline errors.
Suggested fix:
on_failure:
- set:
field: error.message
value: 'Pipeline processing failed: {{{_ingest.on_failure_message}}}'
- set:
field: event.kind
value: pipeline_error🤖 AI-generated comment by Elastic Integration PR Review Bot
There was a problem hiding this comment.
I have already added one at the bottom, line 861
Please check @ShourieG
andrewkroh
removed
the
New Integration
efd6
left a comment
efd6
left a comment
There was a problem hiding this comment.
Suggested commit message:
axonius: add exposure data stream with vulnerability and asset data
This new data stream collects vulnerability instances, vulnerabilities,
and vulnerability repository data from the Axonius platform via the
/api/v2/assets/ endpoint. Axonius fields are mapped to their
corresponding ECS fields where possible.
Test samples were derived from live data samples, which were subsequently
sanitized.
API documentation: https://docs.axonius.com/docs/axonius-rest-api
1. Adding new line. 2. Remove empty_event message. 3. Add suggestions in cel.
|
Package axonius - 0.1.0 containing this change is available at https://epr.elastic.co/package/axonius/0.1.0/ |
5 participants
Proposed commit message
The release includes exposure data stream and associated dashboard.
Axonius fields are mapped to their corresponding ECS fields where possible.
Test samples were derived from live data samples, which were subsequently
sanitized.
Checklist
changelog.ymlfile.How to test this PR locally
To test the axonius package:
Related issues
Screenshots