ti_misp: harmonise object fields between datastreams #5917

efd6 · 2023-04-19T06:15:33Z

What does this PR do?

Brings field definitions into agreement. This is the second part of addressing #5834,

Note that if the event added in tests here is add to the threat datastream test, it fails due to a failure of the uri_parts processor to handle \ in the host part of a URL.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Fixes [misp] Field conflicts between threat and threat_attributes #5834.

Screenshots

elasticmachine · 2023-04-19T06:18:40Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-04-28T08:16:56.814+0000
Duration: 15 min 23 sec

Test stats 🧪

Test	Results
Failed	0
Passed	15
Skipped	0
Total	15

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-04-19T06:32:12Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`2/2`)	💚
Files	100.0% (`2/2`)	💚
Classes	100.0% (`2/2`)	💚
Methods	100.0% (`30/30`)	💚 10.596
Lines	86.792% (`598/689`)	👎 -4.555
Conditionals	100.0% (`0/0`)	💚

elasticmachine · 2023-04-19T12:08:26Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

kcreddy

Need clarification about sample log

kcreddy · 2023-04-26T08:37:29Z

packages/ti_misp/data_stream/threat_attributes/_dev/test/pipeline/test-misp-sample-ndjson.log

@@ -10,3 +10,4 @@
 {"id":"40687","event_id":"53","object_id":"0","object_relation":null,"category":"Network activity","type":"AS","to_ids":false,"uuid":"54f86825-c80c-47cf-a795-48c1950d210b","timestamp":"1425565733","distribution":"5","sharing_group_id":"0","comment":"","deleted":false,"disable_correlation":false,"first_seen":null,"last_seen":null,"value":"48031","Event":{"org_id":"1","distribution":"3","id":"53","info":"OSINT Analysis of malcious CHM file by OpenDNS","orgc_id":"2","uuid":"54f8662f-c7f0-4f59-a42a-a9a9950d210b"}}
 {"id":"1084","event_id":"2","object_id":"0","object_relation":null,"category":"Network activity","type":"ip-dst","to_ids":true,"uuid":"54324081-3308-4f1f-8674-4953950d210b","timestamp":"1412579457","distribution":"5","sharing_group_id":"0","comment":"","deleted":false,"disable_correlation":false,"first_seen":null,"last_seen":null,"value":"89.160.20.129","Event":{"org_id":"1","distribution":"3","id":"2","info":"OSINT New Indicators of Compromise for APT Group Nitro Uncovered blog post by Palo Alto Networks","orgc_id":"2","uuid":"54323f2c-e50c-4268-896c-4867950d210b"}}
 {"id":"24749","event_id":"10","object_id":"0","object_relation":null,"category":"Network activity","type":"email-dst","to_ids":true,"uuid":"544ff4c2-914c-482f-aa29-4c43950d210b","timestamp":"1414526146","distribution":"5","sharing_group_id":"0","comment":"","deleted":false,"disable_correlation":false,"first_seen":null,"last_seen":null,"value":"lisa.cuddy@wind0ws.kz","Event":{"org_id":"1","distribution":"3","id":"10","info":"OSINT APT28: A Window into Russiaâ€™s Cyber Espionage Operations? blog post by FireEye","orgc_id":"2","uuid":"544fee45-f108-4fa6-ace9-3989950d210b"}}
+{"Event":{"Attribute":[],"CryptographicKey":[],"EventReport":[],"Galaxy":[],"Object":{"Attribute":{"Galaxy":[],"ShadowAttribute":[],"Tag":[{"colour":"#27c1f2","exportable":true,"hide_tag":false,"id":"49","is_custom_galaxy":false,"is_galaxy":false,"local":0,"local_only":false,"name":"Diamond: Capability/TTP","numerical_value":null,"relationship_type":null,"user_id":"0"},{"colour":"#b7bfe8","exportable":true,"hide_tag":false,"id":"16","is_custom_galaxy":false,"is_galaxy":false,"local":0,"local_only":false,"name":"killchain:Delivery","numerical_value":null,"relationship_type":null,"user_id":"0"}],"category":"Network activity","comment":"","deleted":false,"disable_correlation":false,"distribution":"5","event_id":"12874","first_seen":null,"id":"307473","last_seen":null,"object_id":"18647","object_relation":"url","sharing_group_id":"0","timestamp":"1675782548","to_ids":true,"type":"url","uuid":"ebe85f63-1ff2-482b-a1bb-2dcd34f4bebb","value":"/wp\\-includes/js/jquery.min.js"},"ObjectReference":[],"comment":"","deleted":false,"description":"url object describes an url along with its normalized field (like extracted using faup parsing library) and its metadata.","distribution":"5","event_id":"12874","first_seen":null,"id":"18647","last_seen":null,"meta-category":"network","name":"url","sharing_group_id":"0","template_uuid":"60efb77b-40b5-4c46-871b-ed1ed999fce5","template_version":"9","timestamp":"1675782559","uuid":"d950dfa2-9f20-4aa7-b347-6e7d313cdb69"},"Org":{"id":"4","local":true,"name":"Organization SOC","uuid":"9158b048-9b84-4a7e-9aec-57120f3ea268"},"Orgc":{"id":"4","local":true,"name":"Organization SOC","uuid":"9158b048-9b84-4a7e-9aec-57120f3ea268"},"RelatedEvent":[],"ShadowAttribute":[],"Tag":[{"colour":"#33FF00","exportable":true,"hide_tag":false,"id":"13","is_custom_galaxy":false,"is_galaxy":false,"local":0,"local_only":false,"name":"tlp:green","numerical_value":null,"relationship_type":null,"user_id":"0"},{"colour":"#f29035","exportable":true,"hide_tag":false,"id":"65","is_custom_galaxy":false,"is_galaxy":false,"local":0,"local_only":false,"name":"veris:action:social","numerical_value":null,"relationship_type":null,"user_id":"0"},{"colour":"#17a4d1","exportable":true,"hide_tag":false,"id":"62","is_custom_galaxy":false,"is_galaxy":false,"local":0,"local_only":false,"name":"veris:actor:external","numerical_value":null,"relationship_type":null,"user_id":"0"}],"analysis":"0","attribute_count":"3","date":"2023-02-03","disable_correlation":false,"distribution":"1","extends_uuid":"","id":"12874","info":"Phishing - Payment Processed for Inovice75 on Feb 01,2023","locked":false,"org_id":"4","orgc_id":"4","proposal_email_lock":false,"protected":null,"publish_timestamp":"0","published":false,"sharing_group_id":"0","threat_level_id":"3","timestamp":"1675782577","uuid":"0d6aba79-dd30-4ffb-b90e-81a8322517d0"}}


This response doesn't seem to belong to threat attributes API

Was this supposed to be added to threat datastream instead?

Yes, good catch. Thanks

kcreddy · 2023-04-26T08:38:13Z

packages/ti_misp/data_stream/threat_attributes/elasticsearch/ingest_pipeline/default.yml

+      if: 'ctx.misp?.event?.Object != null && ctx.misp.event.Object.size() == 0'
+  - rename:
+      field: misp.event.Attribute
+      target_field: misp.attribute


We are already renaming event.original to misp.attribute earlier in the pipeline.

Sorry, what I actually meant to say was that all these renaming with misp.event.* fields never exists in the Attributes API. It has fields named misp.attribute.* i.e., misp.attribute.Object, misp.attribute.Event, misp.attribute.Tag which were already renamed in the pipeline earlier.

Indeed, you're correct, fixed fixed.

kcreddy

LGTM 👍🏼 Minor suggestion

kcreddy · 2023-04-28T07:41:41Z

packages/ti_misp/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.13.1"
+  changes:
+    - description: Harmonise object and org fields.


org is probably not relevant anymore in this PR

packages/ti_misp/changelog.yml

elasticmachine · 2023-04-28T09:04:49Z

Package ti_misp - 1.13.1 containing this change is available at https://epr.elastic.co/search?package=ti_misp

efd6 added bug Something isn't working Team:Security-External Integrations Integration:MISP labels Apr 19, 2023

efd6 self-assigned this Apr 19, 2023

efd6 force-pushed the 5834-ti_misp-threat_attributes branch from 6840890 to be7a37c Compare April 19, 2023 06:16

efd6 requested a review from P1llus April 19, 2023 06:16

efd6 marked this pull request as ready for review April 19, 2023 12:08

efd6 requested a review from a team as a code owner April 19, 2023 12:08

kcreddy reviewed Apr 26, 2023

View reviewed changes

ti_misp: harmonise object and org fields between datastreams

adb3728

efd6 force-pushed the 5834-ti_misp-threat_attributes branch from be7a37c to 2b762c1 Compare April 26, 2023 21:23

address pr comment

6446f1e

efd6 force-pushed the 5834-ti_misp-threat_attributes branch from 2b762c1 to 6446f1e Compare April 26, 2023 21:24

address pr comment

473dfb6

efd6 force-pushed the 5834-ti_misp-threat_attributes branch from ab82a1c to 473dfb6 Compare April 27, 2023 23:39

kcreddy approved these changes Apr 28, 2023

View reviewed changes

efd6 commented Apr 28, 2023

View reviewed changes

packages/ti_misp/changelog.yml Outdated Show resolved Hide resolved

Update packages/ti_misp/changelog.yml

915895f

efd6 changed the title ~~ti_misp: harmonise object and org fields between datastreams~~ ti_misp: harmonise object fields between datastreams Apr 28, 2023

efd6 merged commit 04aee29 into elastic:main Apr 28, 2023
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ti_misp: harmonise object fields between datastreams #5917

ti_misp: harmonise object fields between datastreams #5917

efd6 commented Apr 19, 2023

elasticmachine commented Apr 19, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Apr 19, 2023 •

edited

Loading

elasticmachine commented Apr 19, 2023

kcreddy left a comment

kcreddy Apr 26, 2023

efd6 Apr 26, 2023

kcreddy Apr 26, 2023

efd6 Apr 26, 2023

kcreddy Apr 27, 2023

efd6 Apr 27, 2023

kcreddy left a comment

kcreddy Apr 28, 2023

elasticmachine commented Apr 28, 2023

ti_misp: harmonise object fields between datastreams #5917

ti_misp: harmonise object fields between datastreams #5917

Conversation

efd6 commented Apr 19, 2023

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Apr 19, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Apr 19, 2023 • edited Loading

🌐 Coverage report

elasticmachine commented Apr 19, 2023

kcreddy left a comment

Choose a reason for hiding this comment

kcreddy Apr 26, 2023

Choose a reason for hiding this comment

efd6 Apr 26, 2023

Choose a reason for hiding this comment

kcreddy Apr 26, 2023

Choose a reason for hiding this comment

efd6 Apr 26, 2023

Choose a reason for hiding this comment

kcreddy Apr 27, 2023

Choose a reason for hiding this comment

efd6 Apr 27, 2023

Choose a reason for hiding this comment

kcreddy left a comment

Choose a reason for hiding this comment

kcreddy Apr 28, 2023

Choose a reason for hiding this comment

elasticmachine commented Apr 28, 2023

elasticmachine commented Apr 19, 2023 •

edited

Loading

elasticmachine commented Apr 19, 2023 •

edited

Loading