[TI_Anomali] Add a field to easily filter transform's source and destination indices #6344
Conversation
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
🤖 GitHub commentsExpand to view the GitHub comments
To re-run your PR in the CI, just comment with:
|
| # This field is deleted inside destination ingest pipeline run by the transform. | ||
| # The field is added to efficiently filter out destination indices containing unexpired indicators. | ||
| - set: | ||
| field: is_ioc_transform_source |
There was a problem hiding this comment.
The ECS labels.* field could be a good place for this data. The value should be a string to use labels.
https://www.elastic.co/guide/en/ecs/8.5/ecs-base.html#field-labels
There was a problem hiding this comment.
Added labels ECS field, but also had to define inside fields.yml due to errors.
More details here - #6344 (comment)
🌐 Coverage report
|
|
Pinging @elastic/security-external-integrations (Team:Security-External Integrations) |
packages/ti_anomali/elasticsearch/transform/latest_ioc/transform.yml
Outdated
Show resolved
Hide resolved
| field: error.message | ||
| value: '{{{_ingest.on_failure_message}}}' | ||
| - append: | ||
| field: event.kind |
There was a problem hiding this comment.
AFAIK event.kind is a scalar in ECS. Using append is incompatible with the type. How about a set?
There was a problem hiding this comment.
Used set processor. I also noticed quite a few integrations have been using append processor which probably needs fixing as well.
There was a problem hiding this comment.
Is it required to increment fleet_transform_version in order to have this new feature installed on upgrade?
There was a problem hiding this comment.
Removed the destination index pipeline since constant_keyword with default value of true worked. So, doesn't require any transform changes. But yeah it was a required change if we went with adding destination ingest pipeline
| @@ -100,3 +100,6 @@ | |||
| type: date | |||
| description: >- | |||
| Date when IOC was deleted/expired. | |||
| - name: labels.is_ioc_transform_source | |||
| type: keyword | |||
There was a problem hiding this comment.
What about if this became a constant_keyword field with a static value? Then we would not need to modify the ingest pipeline to add it. Consequently we would not need to add a transform pipeline to remove the value because it would not be present in _source.
| type: keyword | |
| type: constant_keyword | |
| value: "true" |
There was a problem hiding this comment.
Updated the field with constant_keyword and removed destination ingest pipeline. No change to transform is required and verified that the field is not present in the destination index.
|
Package ti_anomali - 1.12.0 containing this change is available at https://epr.elastic.co/search?package=ti_anomali |
…ination indices (elastic#6344) * Add field for seperating ioc indices * update pr num * Address PR comment., Add to labels * update README * update package version prefix * update tests * Remove dest pipeline
What does this PR do?
Bakground:
Adding indicator rules is a bit complex after IOC expiration support is launched. The destination indices (unexpired indicators) are needed to be queried using
_indexfield and also using explicitly added Recorded Future and Anomali rules separately.This PR:
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Related issues
Screenshots