[TrendMicro VisionOne] Fix Detection API TMV1-Query header #8083

kcreddy · 2023-10-04T06:38:02Z

Proposed commit message

The Detection API for TrendMicro Vision One takes a required header TMV1-Query which needs to contain an operator to perform the query. Currently there is no operator which is causing the API to return zero results. This PR fixes the header and adds :* to the existing operator to query all events containing uuid.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Related issues

Relates https://github.com/elastic/sdh-beats/issues/3899

elasticmachine · 2023-10-04T06:55:01Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-10-06T09:17:09.801+0000
Duration: 16 min 49 sec

Test stats 🧪

Test	Results
Failed	0
Passed	19
Skipped	0
Total	19

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-10-04T06:55:12Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`3/3`)	💚
Files	100.0% (`3/3`)	💚 3.329
Classes	100.0% (`3/3`)	💚 3.329
Methods	95.652% (`44/46`)	👍 3.219
Lines	95.226% (`1516/1592`)	👍 6.891
Conditionals	100.0% (`0/0`)	💚

elasticmachine · 2023-10-04T07:07:21Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

bhapas

Possible to test this in system test?

kcreddy · 2023-10-06T09:36:44Z

@bhapas Added system test for the same.

bhapas

LGTM

elasticmachine · 2023-10-06T10:31:04Z

Package trend_micro_vision_one - 1.12.1 containing this change is available at https://epr.elastic.co/search?package=trend_micro_vision_one

hussein759 · 2023-10-10T06:49:54Z

@kcreddy and @bhapas,

Regarding the TrendMicro integration for detection with the API path /v3.0/search/detections: When the header {'Authorization': 'Bearer ' + token, 'TMV1-Query': '(uuid:*)'} is added to the query, the result contains a significant amount of data — over 10,000 hits in the last hour. However, after updating to version 1.12.1, the output seems to be limited to roughly 10 events or fewer per hour from vision_one_detection.

Is it accurate that when uuid is set as a wildcard, it should fetch all events? It seems that this expected behavior is not being observed in the recent update. Has this been thoroughly tested?

bhapas · 2023-10-10T10:15:26Z

@kcreddy and @bhapas,

Regarding the TrendMicro integration for detection with the API path /v3.0/search/detections: When the header {'Authorization': 'Bearer ' + token, 'TMV1-Query': '(uuid:*)'} is added to the query, the result contains a significant amount of data — over 10,000 hits in the last hour. However, after updating to version 1.12.1, the output seems to be limited to roughly 10 events or fewer per hour from vision_one_detection.

Is it accurate that when uuid is set as a wildcard, it should fetch all events? It seems that this expected behavior is not being observed in the recent update. Has this been thoroughly tested?

@hussein759 Any notable logs in the agent logs? Like Dropped Event or log.error etc?

hussein759 · 2023-10-10T10:40:42Z

Thanks bapas, i think the request field is again causing issue after this update which was bug of earlier version for request field and it was fixed earlier,uploading the image where error.message shows field [request] not present as part of path which again i m assuming is an array expected but its not mapped correctly.. image attached for your reference

kcreddy · 2023-10-10T11:03:06Z

Hey @hussein759 What Bharat meant was taking Elastic Agent diagnostics logs. For Fleet managed Elastic Agent, you can collect it this way. If not, you can take through cli. If you are seeing any errors or Dropped Events in Agent logs from TrendMicro VisionOne, can you open an issue and link this PR?

Regarding above error on request field, it would be nice if you can enable the Preserve original event from the UI while configuring the integration and share event.original field. You can also share the same in the issue. Thanks!

hussein759 · 2023-10-10T11:32:03Z

Thanks @kcreddy, request field issue is similar to #7871, this was bug and it was fixed, if u refer this issue for request field you will know exactly whats going on, as per trendmicro document it is not mention that request field will contain array however the elastic developer managed to fix it to include the array and @bhapas had raised this issue earlier for this!

hussein759 · 2023-10-10T12:00:34Z

@kcreddy Just to add and be clear that now after this upgrade we started facing this bug again of request field.

bhapas · 2023-10-10T14:00:14Z

@hussein759 What @kcreddy is an example event.original with which you see this issue again in 1.12.1. And this can be retrieved by enabling Preserve Original Event in the integration UI.

hussein759 · 2023-10-11T05:06:15Z

@bhapas, sure here is the event.original field

{"act":["not blocked"],"aggregatedCount":"1","app":"SMB2","appGroup":"CIFS","aptRelated":"0","clientFlag":"src","cnt":"8","compressedFileSize":"0","dceHash1":"0","dceHash2":"0","detectionType":"1","deviceDirection":"outbound","deviceGUID":"73497B359ABF-4189A916-9CD9-23B1-C465","deviceMacAddress":"4c:d9:8f:aa:0a:12","devicePayloadId":"99:10786206:::","deviceRiskConfidenceLevel":3,"dhost":"aeadvsa044.internal.adsic.abudhabi.ae","dmac":"00:00:0c:07:ac:96","dpt":445,"dst":["192.168.15.25"],"dstGroup":"Default","dstZone":"1","dvc":["192.168.5.221"],"dvchost":"EUADVSATM05","eventId":"100119","eventName":"SECURITY_RISK_DETECTION","eventSourceType":3,"eventTime":1696921180000,"eventTimeDT":"2023-10-10T06:59:40+00:00","filePath":"\","filePathName":"\","fileSize":"0","hasdtasres":"No","interestedGroup":"Default","interestedHost":"sam-b-df-09.internal.samuel.eu","interestedIp":["192.168.20.233"],"isHidden":"Yes","logKey":"23497B359ABF-4189A916-9CD9-23B1-C465_DDI-3.8-CAV-JSONv1-en_10786206","malType":"OTHERS","malTypeGroup":"Others","overSsl":"Not over SSL/TLS","pComp":"NCIE","peerGroup":"Default","peerHost":"df-08.internal.samuel.eu","peerIp":["192.168.20.233"],"pname":"Deep Discovery Inspector","potentialRisk":"1","productCode":"pdi","pver":"6.5.1129","remarks":"["IP address: 192.168.20.233"]","rt":"2023-10-10T06:59:40.0000000Z","rtDate":"2023-10-10T00:00:00.0000000Z","rtHour":10,"rtWeekDay":"Tuesday","rt_utc":"2023-10-10T06:59:40.0000000Z","ruleId":3498,"ruleName":"DOWNLOAD FILE - SMB2(REQUEST) - BETA","searchDL":"DDL","senderGUID":"73497B359ABF-4189A916-9CD9-23B1-C465","senderIp":["192.168.5.221"],"severity":2,"shost":"pamm-l-df-09.internal.samuel.eu","smac":"98:72:5d:2d:13:06","spt":52066,"src":["192.168.20.233"],"srcGroup":"Default","srcZone":"1","threatType":"99","uuid":"f4b08e7d-716e-4bc9-9569-2645b7cdb958","vLANId":4095}

kcreddy · 2023-10-11T05:35:53Z

@hussein759 In your sample above the field request doesn't exist unlike the other logs. Hence you have the error.message populated. #8160 should fix it. But first, can you give clarification on why request does not exist on such documents?

Going back to earlier issue with your reduced data, were you seeing higher data ingestion with earlier integration version and when upgraded to 1.12.1 you saw it reduced?

hussein759 · 2023-10-12T05:25:14Z

@kcreddy Observations on Vision One Detection Event Categories:

Categories with request Data in the "trend_micro_vision_one.detection.request" field:

Web_Threat_Detection, Web_Policy_Violation, Message_Suspicious_Detection, Security_Risk_Detection.

For above categories, the "trend_micro_vision_one.detection.request" field displays data.

Categories Lacking Data in the "trend_micro_vision_one.detection.request" field mentioned below
For these categories, there is no data present in the "trend_micro_vision_one.detection.request" field, leading to the appearance of an error message.
field [request] not present as part of path [trend_micro_vision_one.detection.request]

BEHAVIORAL_VIOLATION, INTEGRITY_MONITORING_EVENT, APPLICATION_CONTROL_VIOLATION, FIREWALL_POLICY_VIOLATION.

I can see u have made upgrade request to 1.12.2 to exclude error message from field request if it doesnt contain any data
so one it is pushed to epr this might also solve the request field issue

For the uuid changes you have made with * the logs count are fine and that is no longer any issue

Thanks for all the work!, if u need any more info pls let me know.

kcreddy · 2023-10-12T11:33:29Z

@hussein759 Awesome, thanks for the explanation. That helps great deal!

kcreddy added 2 commits October 4, 2023 12:01

Fix Trendmicro detection API TMV1-Query header

efb0843

update changelog

41df2cb

kcreddy self-assigned this Oct 4, 2023

kcreddy added Integration:TrendMicroVisionOne Trend Micro Vision One bugfix labels Oct 4, 2023

kcreddy marked this pull request as ready for review October 4, 2023 07:07

kcreddy requested a review from a team as a code owner October 4, 2023 07:07

kcreddy added the Team:Security-External Integrations label Oct 4, 2023

Merge remote-tracking branch 'upstream/main' into trendmicro_detection

3c36bfa

bhapas reviewed Oct 4, 2023

View reviewed changes

kcreddy added 2 commits October 6, 2023 12:01

update system tests

b21d7d4

readme and format

2c2d10c

bhapas approved these changes Oct 6, 2023

View reviewed changes

kcreddy merged commit 4ee9696 into elastic:main Oct 6, 2023
4 checks passed

kcreddy mentioned this pull request Oct 11, 2023

[TrendMicro VisionOne] Fix missing request in pipeline #8160

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TrendMicro VisionOne] Fix Detection API TMV1-Query header #8083

[TrendMicro VisionOne] Fix Detection API TMV1-Query header #8083

kcreddy commented Oct 4, 2023 •

edited

elasticmachine commented Oct 4, 2023 •

edited

Build stats

Test stats 🧪

elasticmachine commented Oct 4, 2023 •

edited

elasticmachine commented Oct 4, 2023

bhapas left a comment

kcreddy commented Oct 6, 2023

bhapas left a comment

elasticmachine commented Oct 6, 2023

hussein759 commented Oct 10, 2023

bhapas commented Oct 10, 2023

hussein759 commented Oct 10, 2023

kcreddy commented Oct 10, 2023

hussein759 commented Oct 10, 2023

hussein759 commented Oct 10, 2023

bhapas commented Oct 10, 2023

hussein759 commented Oct 11, 2023

kcreddy commented Oct 11, 2023 •

edited

hussein759 commented Oct 12, 2023

kcreddy commented Oct 12, 2023 •

edited

[TrendMicro VisionOne] Fix Detection API TMV1-Query header #8083

[TrendMicro VisionOne] Fix Detection API TMV1-Query header #8083

Conversation

kcreddy commented Oct 4, 2023 • edited

Proposed commit message

Checklist

Related issues

elasticmachine commented Oct 4, 2023 • edited

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Oct 4, 2023 • edited

🌐 Coverage report

elasticmachine commented Oct 4, 2023

bhapas left a comment

Choose a reason for hiding this comment

kcreddy commented Oct 6, 2023

bhapas left a comment

Choose a reason for hiding this comment

elasticmachine commented Oct 6, 2023

hussein759 commented Oct 10, 2023

bhapas commented Oct 10, 2023

hussein759 commented Oct 10, 2023

kcreddy commented Oct 10, 2023

hussein759 commented Oct 10, 2023

hussein759 commented Oct 10, 2023

bhapas commented Oct 10, 2023

hussein759 commented Oct 11, 2023

kcreddy commented Oct 11, 2023 • edited

hussein759 commented Oct 12, 2023

kcreddy commented Oct 12, 2023 • edited

kcreddy commented Oct 4, 2023 •

edited

elasticmachine commented Oct 4, 2023 •

edited

elasticmachine commented Oct 4, 2023 •

edited

kcreddy commented Oct 11, 2023 •

edited

kcreddy commented Oct 12, 2023 •

edited