[Enhancement] Cisco MERAKI urls user agent support #4873

srilumpa · 2022-12-20T10:26:27Z

What does this PR do?

In some cases, Cisco MERAKI URLS logs contains an "agent" field right after the "mac" field. Since the detection of those fields is realized with a "dissect" processor, the extracted MAC address can then contains the user-agent as well.

This PR aims to extract the User-Agent, leave the MAC address (and only the MAC address) in the cisco_meraki.urls.mac field and parse the User-Agent using the "User Agent" processor.

Example before fix:

{
  "_source": {
    "cisco_meraki": {
      "urls": {
        "mac": "AA-BB-CC-DD-EE-FF agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:108.0) Gecko/20100101 Firefox/108.0'"
      },
      "event_type": "urls",
      "event_subtype": "http_access"
    },
    ...
  }
}

After fix:

{
  "_source": {
    "cisco_meraki": {
      "urls": {
        "mac": "AA-BB-CC-DD-EE-FF agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:108.0) Gecko/20100101 Firefox/108.0'"
      },
      "event_type": "urls",
      "event_subtype": "http_access"
    },
    "user_agent": {
      "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:108.0) Gecko/20100101 Firefox/108.0",
      "version": "108.0",
      "name": "Firefox",
      "device": {
        "name": "Other"
      },
      "os": {
        "name": "Windows",
        "version": "10",
        "full": "Windows 10"
      }
    }
    ...
  }
}

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

I have updated the unitary tests

How to test this PR locally

Execute elastic-package check

Related issues

None

Screenshots

elasticmachine · 2022-12-20T10:29:48Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-01-24T09:46:44.583+0000
Duration: 17 min 2 sec

Test stats 🧪

Test	Results
Failed	0
Passed	19
Skipped	0
Total	19

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

ShourieG · 2022-12-21T12:33:54Z

/test

ShourieG · 2022-12-21T12:50:51Z

@srilumpa The ci error is occurring because there is a mismatch in the readme fields. You need to update the file "sample_event.json" in the directory cisco_meraki/data_stream/log with the new fields introduced/changed by this PR and do run the command elastic-package build . You need to install elastic-package first ofc. This will rebuild the README.md with the new fields and the error will go away.

srilumpa · 2022-12-21T14:48:52Z

@ShourieG thank you for the feedback. I have updated the README.md file as asked.

ShourieG · 2022-12-22T07:44:19Z

@srilumpa Hey there, actually if you manually update the README it's not gonna work. I pulled your branch code and checked a couple of things, and these are the following steps you need to do, so that it passes the ci checks :-

install elastic-package tool from : https://github.com/elastic/elastic-package & make sure you have docker installed.
then after that run the command: elastic-package stack up -v -d
go to the integration folder cisco_meraki/data_stream/log and look for the file "sample_event.json"
add the following block to the to the existing log :

 "user_agent": {
      "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:108.0) Gecko/20100101 Firefox/108.0",
      "version": "108.0",
      "name": "Firefox",
      "device": {
        "name": "Other"
      },
      "os": {
        "name": "Windows",
        "version": "10",
        "full": "Windows 10"
      }
    }

then after this run command elastic-package build from inside the cisco_meraki folder
run command : elastic-package stack up -v -d --services package-registry from inside the cisco_meraki folder
run command eval "$(elastic-package stack shellinit)" or eval $(elastic-package stack shellinit --shell bash)
which ever works on your system
Finally run command elastic-package test pipeline -v --generate from inside the cisco_meraki folder

These steps will regenerate the expected log files, and README files at the same time testing the changes that you have made and mostly make sure it passes our ci tests.

… been processed

srilumpa · 2022-12-22T10:35:19Z

Hi @ShourieG and thank you for you guiding me for this.

I updated the README.md file simply running elastic-package build, not manually, and I am not confident in changing the sample_event.jsonby adding this block since the sample log in it is not an urls log type but one of the security_event log type.

Anyway, here is an updated version after the elastic-package test pipeline -v --generate command has been ran. Here is the output of the command.

2022/12/22 11:22:23 DEBUG Enable verbose logging
Run pipeline tests for the package
--- Test results for package: cisco_meraki - START ---
╭──────────────┬─────────────┬───────────┬────────────────────────────┬────────┬──────────────╮
│ PACKAGE      │ DATA STREAM │ TEST TYPE │ TEST NAME                  │ RESULT │ TIME ELAPSED │
├──────────────┼─────────────┼───────────┼────────────────────────────┼────────┼──────────────┤
│ cisco_meraki │ events      │ pipeline  │ test-mx-events.json        │ PASS   │    42.0565ms │
│ cisco_meraki │ log         │ pipeline  │ test-airmarshal-events.log │ PASS   │   227.7732ms │
│ cisco_meraki │ log         │ pipeline  │ test-events.log            │ PASS   │   102.4388ms │
│ cisco_meraki │ log         │ pipeline  │ test-flows.log             │ PASS   │    35.0057ms │
│ cisco_meraki │ log         │ pipeline  │ test-ip-flow.log           │ PASS   │    27.1247ms │
│ cisco_meraki │ log         │ pipeline  │ test-security-events.log   │ PASS   │    32.5743ms │
│ cisco_meraki │ log         │ pipeline  │ test-urls.log              │ PASS   │    22.0469ms │
╰──────────────┴─────────────┴───────────┴────────────────────────────┴────────┴──────────────╯
--- Test results for package: cisco_meraki - END   ---
Done

ShourieG · 2022-12-22T10:49:11Z

/test

elasticmachine · 2022-12-22T11:07:11Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`2/2`)	💚
Files	100.0% (`9/9`)	💚 2.418
Classes	100.0% (`9/9`)	💚 2.418
Methods	100.0% (`63/63`)	💚 9.347
Lines	98.011% (`1084/1106`)	👍 5.431
Conditionals	100.0% (`0/0`)	💚

ShourieG

LGTM

ShourieG · 2022-12-22T11:08:53Z

@srilumpa have approved the changed

joeafari · 2023-01-20T15:39:46Z

hey @srilumpa @ShourieG
There seems to be a conflict that is preventing the pull request to be finalized. Could you please review and correct it?
Thanks

ShourieG · 2023-01-20T16:04:47Z

@srilumpa, you need to merge latest upstream/main into your branch, it should resolve the merge conflicts

srilumpa · 2023-01-24T10:11:14Z

@joeafari and @ShourieG, I fixed the conflict and updated the tests accordingly

elasticmachine · 2023-01-24T12:34:05Z

Package cisco_meraki - 1.5.1 containing this change is available at https://epr.elastic.co/search?package=cisco_meraki

srilumpa added 5 commits December 20, 2022 10:57

[Cisco MERAKI] Update use case tests

359abee

[Cisco MERAKI] Update pipeline to handle User-Agent in urls log

f67ae31

[Cisco MERAKI] Update fields list

f3f770e

[Cisco MERAKI] Bump integration version

80efb2c

[Cisco MERAKI] Update changelog

8421dd2

srilumpa requested a review from a team as a code owner December 20, 2022 10:26

Update README.md

8d186ac

ShourieG self-requested a review December 22, 2022 07:04

Update test-urls expected logs once elastic-package test pipeline has…

fde01a0

… been processed

ShourieG approved these changes Dec 22, 2022

View reviewed changes

srilumpa added 2 commits January 23, 2023 12:01

Merge branch 'main' into feature/urls_user_agent_support

dbf57bd

Update expected ECS version

3a98925

ShourieG merged commit 2349855 into elastic:main Jan 24, 2023

srilumpa deleted the feature/urls_user_agent_support branch January 27, 2023 17:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] Cisco MERAKI urls user agent support #4873

[Enhancement] Cisco MERAKI urls user agent support #4873

srilumpa commented Dec 20, 2022 •

edited

Loading

elasticmachine commented Dec 20, 2022 •

edited

Loading

Build stats

Test stats 🧪

ShourieG commented Dec 21, 2022

ShourieG commented Dec 21, 2022

srilumpa commented Dec 21, 2022

ShourieG commented Dec 22, 2022 •

edited

Loading

srilumpa commented Dec 22, 2022

ShourieG commented Dec 22, 2022

elasticmachine commented Dec 22, 2022 •

edited

Loading

ShourieG left a comment

ShourieG commented Dec 22, 2022

joeafari commented Jan 20, 2023

ShourieG commented Jan 20, 2023

srilumpa commented Jan 24, 2023

elasticmachine commented Jan 24, 2023

[Enhancement] Cisco MERAKI urls user agent support #4873

[Enhancement] Cisco MERAKI urls user agent support #4873

Conversation

srilumpa commented Dec 20, 2022 • edited Loading

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Dec 20, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

ShourieG commented Dec 21, 2022

ShourieG commented Dec 21, 2022

srilumpa commented Dec 21, 2022

ShourieG commented Dec 22, 2022 • edited Loading

srilumpa commented Dec 22, 2022

ShourieG commented Dec 22, 2022

elasticmachine commented Dec 22, 2022 • edited Loading

🌐 Coverage report

ShourieG left a comment

Choose a reason for hiding this comment

ShourieG commented Dec 22, 2022

joeafari commented Jan 20, 2023

ShourieG commented Jan 20, 2023

srilumpa commented Jan 24, 2023

elasticmachine commented Jan 24, 2023

srilumpa commented Dec 20, 2022 •

edited

Loading

elasticmachine commented Dec 20, 2022 •

edited

Loading

ShourieG commented Dec 22, 2022 •

edited

Loading

elasticmachine commented Dec 22, 2022 •

edited

Loading