Generate geo ip test results with the new db #2339

marc-gr · 2021-12-14T15:25:48Z

What does this PR do?

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

elastic-package version is updated
all integrations use IPs of the allowed set for testing
all test files are generated to use the new geoip database

elasticmachine · 2021-12-14T15:25:50Z

Pinging @elastic/integrations (Team:Integrations)

elasticmachine · 2021-12-14T15:25:50Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2021-12-14T15:41:23Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-12-15T09:04:51.721+0000
Duration: 83 min 29 sec
Commit: 296efbc

Test stats 🧪

Test	Results
Failed	1
Passed	3683
Skipped	5
Total	3689

Test errors

Expand to view the tests failures

`Check integrations / akamai / akamai: check / pipeline test: test-http-json.log – akamai.siem`

no error details

Expand to view the stacktrace

 test case failed: Expected results are different from actual ones:  {
     "expected": [
         {
             "@timestamp": "2017-04-04T10:57:02.000Z",
             "akamai": {
                 "siem": {
                     "bot": {
                         "response_segment": 3,
                         "score": 100
                     },
                     "client_data": {
                         "app_bundle_id": "com.mydomain.myapp",
                         "app_version": "1.23",
                         "sdk_version": "4.7.1",
                         "telemetry_type": 2
                     },
                     "config_id": "14227",
                     "policy_id": "qik1_26545",
                     "request": {
                         "headers": {
                             "Accept": "text/html,application/xhtml xml",
                             "User-Agent": "BOT/0.1 (BOT for JCE)"
                         }
                     },
                     "response": {
                         "headers": {
                             "Content-Length": "150",
                             "Content-Type": "text/html",
                             "Mime-Version": "1.0",
                             "Server": "AkamaiGHost"
                         }
                     },
                     "rules": [
                         {
                             "ruleActions": "alert",
                             "ruleData": "telnet.exe",
                             "ruleMessages": "System Command Access",
                             "ruleSelectors": "ARGS:option",
                             "ruleTags": "OWASP_CRS/WEB_ATTACK/FILE_INJECTION",
                             "ruleVersions": "4",
                             "rules": "950002"
                         },
                         {
                             "ruleActions": "alert",
                             "ruleData": "telnet.exe",
                             "ruleMessages": "System Command Injection",
                             "ruleSelectors": "ARGS:option",
                             "ruleTags": "OWASP_CRS/WEB_ATTACK/COMMAND_INJECT",
                             "ruleVersions": "4",
                             "rules": "950006"
                         },
                         {
                             "ruleActions": "deny",
                             "ruleData": "Vector Score: 10, DENY threshold: 9, Ale",
                             "ruleMessages": "Anomaly Score Exceeded fo",
                             "ruleVersions": "1",
                             "rules": "CMD-INJECTION-ANOMALY"
                         }
                     ],
                     "user_risk": {
                         "allow": 0,
                         "general": {
                             "duc_1d": "30",
                             "duc_1h": "10"
                         },
                         "risk": {
                             "udfp": "1325gdg4g4343g/M",
                             "unp": "74256/H"
                         },
                         "score": 75,
                         "status": 0,
                         "trust": {
                             "ugp": "US"
                         },
                         "uuid": "964d54b7-0821-413a-a4d6-8131770ec8d5"
                     }
                 }
             },
             "client": {
                 "address": "52.91.36.10",
                 "as": {
-                    "number": 14618,
-                    "organization": {
-                        "name": "Amazon.com, Inc."
-                    }
+                    "number": 14618
                 },
                 "geo": {
-                    "city_name": "Ashburn",
-                    "continent_name": "North America",
+                    "city_name": "ASHBURN",
                     "country_iso_code": "US",
-                    "country_name": "United States",
-                    "location": {
-                        "lat": 39.0481,
-                        "lon": -77.4728
-                    },
-                    "region_iso_code": "US-VA",
-                    "region_name": "Virginia"
+                    "region_iso_code": "-VA"
                 },
                 "ip": "52.91.36.10"
             },
             "ecs": {
                 "version": "1.12.0"
             },
             "event": {
                 "category": "network",
                 "id": "1158db1758e37bfe67b7c09",
                 "kind": "event",
                 "original": "{\"format\":\"json\",\"type\":\"akamai_siem\",\"version\":\"1.0\",\"attackData\":{\"clientIP\":\"52.91.36.10\",\"configId\":\"14227\",\"policyId\":\"qik1_26545\",\"ruleActions\":\"YWxlcnQ%3d%3bYWxlcnQ%3d%3bZGVueQ%3d%3d\",\"ruleData\":\"dGVsbmV0LmV4ZQ%3d%3d%3bdGVsbmV0LmV4ZQ%3d%3d%3bVmVjdG9yIFNjb3JlOiAxMCwgREVOWSB0aHJlc2hvbGQ6IDksIEFsZX \",\"ruleMessages\":\"U3lzdGVtIENvbW1hbmQgQWNjZXNz%3bU3lzdGVtIENvbW1hbmQgSW5qZWN0aW9u%3bQW5vbWFseSBTY29yZSBFeGNlZWRlZCBmb3 \",\"ruleSelectors\":\"QVJHUzpvcHRpb24%3d%3bQVJHUzpvcHRpb24%3d%3b\",\"ruleTags\":\"T1dBU1BfQ1JTL1dFQl9BVFRBQ0svRklMRV9JTkpFQ1RJT04%3d%3bT1dBU1BfQ1JTL1dFQl9BVFRBQ0svQ09NTUFORF9JTkpFQ1R \",\"ruleVersions\":\"NA%3d%3d%3bNA%3d%3d%3bMQ%3d%3d\",\"rules\":\"OTUwMDAy%3bOTUwMDA2%3bQ01ELUlOSkVDVElPTi1BTk9NQUxZ\"},\"geo\":{\"asn\":\"14618\",\"city\":\"ASHBURN\",\"continent\":\"288\",\"country\":\"US\",\"regionCode\":\"VA\"},\"httpMessage\":{\"bytes\":\"266\",\"host\":\"www.hmapi.com\",\"method\":\"GET\",\"path\":\"/\",\"port\":\"80\",\"protocol\":\"HTTP/1.1\",\"query\":\"option=com_jce%20telnet.exe\",\"requestHeaders\":\"User-Agent%3a%20BOT%2f0.1%20(BOT%20for%20JCE)%0d%0aAccept%3a%20text%2fhtml,application%2fxhtml+xml\",\"requestId\":\"1158db1758e37bfe67b7c09\",\"responseHeaders\":\"Server%3a%20AkamaiGHost%0d%0aMime-Version%3a%201.0%0d%0aContent-Type%3a%20text%2fhtml%0d%0aContent-Length%3a%20150\",\"start\":\"1491303422\",\"status\":\"200\"},\"userRiskData\":{\"uuid\":\"964d54b7-0821-413a-a4d6-8131770ec8d5\",\"status\":\"0\",\"score\":\"75\",\"risk\":\"udfp:1325gdg4g4343g/M|unp:74256/H\",\"trust\":\"ugp:US\",\"general\":\"duc_1h:10|duc_1d:30\",\"allow\":\"0\"},\"clientData\":{\"appBundleId\":\"com.mydomain.myapp\",\"appVersion\":\"1.23\",\"sdkVersion\":\"4.7.1\",\"telemetryType\":\"2\"},\"botData\":{\"botScore\":\"100\",\"responseSegment\":\"3\"}}",
                 "start": "2017-04-04T10:57:02.000Z"
             },
             "http": {
                 "request": {
                     "id": "1158db1758e37bfe67b7c09",
                     "method": "GET"
                 },
                 "response": {
                     "bytes": 266,
                     "status_code": 200
                 },
                 "version": "1.1"
             },
             "network": {
                 "protocol": "http",
                 "transport": "tcp"
             },
             "observer": {
                 "type": "proxy",
                 "vendor": "akamai"
             },
             "related": {
                 "ip": [
                     "52.91.36.10"
                 ]
             },
             "source": {
                 "address": "52.91.36.10",
                 "as": {
-                    "number": 14618,
-                    "organization": {
-                        "name": "Amazon.com, Inc."
-                    }
+                    "number": 14618
                 },
                 "geo": {
-                    "city_name": "Ashburn",
-                    "continent_name": "North America",
+                    "city_name": "ASHBURN",
                     "country_iso_code": "US",
-                    "country_name": "United States",
-                    "location": {
-                        "lat": 39.0481,
-                        "lon": -77.4728
-                    },
-                    "region_iso_code": "US-VA",
-                    "region_name": "Virginia"
+                    "region_iso_code": "-VA"
                 },
                 "ip": "52.91.36.10"
             },
             "tags": [
                 "preserve_original_event"
             ],
             "url": {
                 "domain": "www.hmapi.com",
                 "full": "www.hmapi.com/?option=com_jce%20telnet.exe",
                 "path": "/",
                 "port": 80,
                 "query": "option=com_jce telnet.exe"
             }
         },
         {
             "@timestamp": "2016-08-11T13:45:33.026Z",
             "akamai": {
                 "siem": {
                     "bot": {
                         "response_segment": 3,
                         "score": 100
                     },
                     "client_data": {
                         "app_bundle_id": "com.mydomain.myapp",
                         "app_version": "1.23",
                         "sdk_version": "4.7.1",
                         "telemetry_type": 2
                     },
                     "config_id": "6724",
                     "policy_id": "scoe_5426",
                     "request": {
                         "headers": {
                             "Accept": "text/html,application/xhtml xml",
                             "User-Agent": "BOT/0.1 (BOT for JCE)"
                         }
                     },
                     "response": {
                         "headers": {
                             "Content-Type": "text/html",
                             "Mime-Version": "1.0",
                             "Server": "AkamaiGHost"
                         }
                     },
                     "rules": [
                         {
                             "ruleActions": "ALERT",
                             "ruleData": "alert(",
                             "ruleMessages": "Cross-site Scripting (XSS) Attack",
                             "ruleSelectors": "ARGS:a",
                             "ruleTags": "WEB_ATTACK/XSS",
                             "rules": "950004"
                         },
                         {
                             "ruleActions": "DENY",
                             "ruleData": "curl",
                             "ruleMessages": "Request Indicates an automated program explored the site",
                             "ruleSelectors": "REQUEST_HEADERS:User-Agent",
                             "ruleTags": "AUTOMATION/MISC",
                             "rules": "990011"
                         }
                     ],
                     "user_risk": {
                         "allow": 0,
                         "general": {
                             "duc_1d": "30",
                             "duc_1h": "10"
                         },
                         "risk": {
                             "udfp": "1325gdg4g4343g/M",
                             "unp": "74256/H"
                         },
                         "score": 75,
                         "status": 0,
                         "trust": {
                             "ugp": "US"
                         },
                         "uuid": "964d54b7-0821-413a-a4d6-8131770ec8d5"
                     }
                 }
             },
             "client": {
                 "address": "52.91.36.10",
                 "as": {
-                    "number": 14618,
-                    "organization": {
-                        "name": "Amazon.com, Inc."
-                    }
+                    "number": 12271
                 },
                 "geo": {
-                    "city_name": "Ashburn",
-                    "continent_name": "North America",
+                    "city_name": "NEWYORK",
                     "country_iso_code": "US",
-                    "country_name": "United States",
-                    "location": {
-                        "lat": 39.0481,
-                        "lon": -77.4728
-                    },
-                    "region_iso_code": "US-VA",
-                    "region_name": "Virginia"
+                    "region_iso_code": "-NY"
                 },
                 "ip": "52.91.36.10"
             },
             "ecs": {
                 "version": "1.12.0"
             },
             "event": {
                 "category": "network",
                 "id": "2ab418ac8515f33",
                 "kind": "event",
                 "original": "{\"format\":\"json\",\"type\":\"akamai_siem\",\"version\":\"1.0\",\"attackData\":{\"clientIP\":\"52.91.36.10\",\"configId\":\"6724\",\"policyId\":\"scoe_5426\",\"ruleActions\":\"QUxFUlQ;REVOWQ==\",\"ruleData\":\"YWxlcnQo;Y3VybA==\",\"ruleMessages\":\"Q3Jvc3Mtc2l0ZSBTY3 JpcHRpbmcgKFhTUykgQXR0YWNr; UmVxdWVzdCBJbmRpY2F0ZXMgYW4 gYXV0b21hdGVkIHByb2 dyYW0gZXhwbG9yZWQgdGhlIHNpdGU=\",\"ruleSelectors\":\"QVJHUzph;UkVRVUVTVF9IRU FERVJTOlVzZXItQWdlbnQ=\",\"ruleTags\":\"V0VCX0FUVEFDSy9YU1M=;QV VUT01BVElPTi9NSVND\",\"ruleVersions\":\";\",\"rules\":\"OTUwMDA0;OTkwMDEx\"},\"geo\":{\"asn\":\"12271\",\"city\":\"NEWYORK\",\"continent\":\"NA\",\"country\":\"US\",\"regionCode\":\"NY\"},\"httpMessage\":{\"bytes\":\"34523\",\"host\":\"www.example.com\",\"method\":\"POST\",\"path\":\"/examples/1/\",\"port\":\"80\",\"protocol\":\"http/2\",\"query\":\"a%3D..%2F..%2F..%2Fetc%2Fpasswd\",\"requestHeaders\":\"User-Agent%3a%20BOT%2f0.1%20(BOT%20for%20JCE)%0d%0aAccept%3a%20text%2fhtml,application%2fxhtml+xml\",\"requestId\":\"2ab418ac8515f33\",\"responseHeaders\":\"Server%3a%20AkamaiGHost%0d%0aMime-Version%3a%201.0%0d%0aContent-Type%3a%20text%2fhtml\",\"start\":\"1470923133.026\",\"status\":\"301\",\"tls\": \"TLSv1.2\"},\"userRiskData\":{\"uuid\":\"964d54b7-0821-413a-a4d6-8131770ec8d5\",\"status\":\"0\",\"score\":\"75\",\"risk\":\"udfp:1325gdg4g4343g/M|unp:74256/H\",\"trust\":\"ugp:US\",\"general\":\"duc_1h:10|duc_1d:30\",\"allow\":\"0\"},\"clientData\":{\"appBundleId\":\"com.mydomain.myapp\",\"appVersion\":\"1.23\",\"sdkVersion\":\"4.7.1\",\"telemetryType\":\"2\"},\"botData\":{\"botScore\":\"100\",\"responseSegment\":\"3\"}}",
                 "start": "2016-08-11T13:45:33.026Z"
             },
             "http": {
                 "request": {
                     "id": "2ab418ac8515f33",
                     "method": "POST"
                 },
                 "response": {
                     "bytes": 34523,
                     "status_code": 301
                 },
                 "version": "2"
             },
             "network": {
                 "protocol": "http",
                 "transport": "tcp"
             },
             "observer": {
                 "type": "proxy",
                 "vendor": "akamai"
             },
             "related": {
                 "ip": [
                     "52.91.36.10"
                 ]
             },
             "source": {
                 "address": "52.91.36.10",
                 "as": {
-                    "number": 14618,
-                    "organization": {
-                        "name": "Amazon.com, Inc."
-                    }
+                    "number": 12271
                 },
                 "geo": {
-                    "city_name": "Ashburn",
-                    "continent_name": "North America",
+                    "city_name": "NEWYORK",
                     "country_iso_code": "US",
-                    "country_name": "United States",
-                    "location": {
-                        "lat": 39.0481,
-                        "lon": -77.4728
-                    },
-                    "region_iso_code": "US-VA",
-                    "region_name": "Virginia"
+                    "region_iso_code": "-NY"
                 },
                 "ip": "52.91.36.10"
             },
             "tags": [
                 "preserve_original_event"
             ],
             "tls": {
                 "version": "1.2",
                 "version_protocol": "tls"
             },
             "url": {
                 "domain": "www.example.com",
                 "full": "www.example.com/examples/1/?a%3D..%2F..%2F..%2Fetc%2Fpasswd",
                 "path": "/examples/1/",
                 "port": 80,
                 "query": "a=../../../etc/passwd"
             }
         },
         null
     ]
 }

Steps errors

Expand to view the steps failures

`Test integration: akamai`

Took 1 min 24 sec . View more details here
Description: eval "$(../../build/elastic-package stack shellinit)" ../../build/elastic-package test -v --report-format xUnit --report-output file --test-coverage

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2021-12-15T11:08:18Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-12-15T09:45:27.784+0000
Duration: 83 min 34 sec
Commit: 3c78cae

Test stats 🧪

Test	Results
Failed	0
Passed	3685
Skipped	5
Total	3690

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

* Generate geo ip test results with the new db * Add changelog entries * Update elastic-package * Cleanup go.sum * Fix test files for mattermost, netflow, atlassian_jira, and tenable_sc * Fix tests for akamai

Generate geo ip test results with the new db

56b113f

marc-gr added enhancement New feature or request Team:Integrations Label for the Integrations team Team:Security-External Integrations labels Dec 14, 2021

marc-gr requested a review from mtojek December 14, 2021 15:25

marc-gr added 2 commits December 14, 2021 16:26

Add changelog entries

e3e42df

Update elastic-package

6b12a54

marc-gr added 5 commits December 14, 2021 17:05

Cleanup go.sum

e45d750

Merge remote-tracking branch 'upstream/master' into update-geoip

b33bcbf

Fix test files for mattermost, netflow, atlassian_jira, and tenable_sc

296efbc

Fix tests for akamai

d77ce4b

Merge remote-tracking branch 'upstream/master' into update-geoip

3c78cae

mtojek approved these changes Dec 15, 2021

View reviewed changes

mtojek merged commit a01caea into elastic:master Dec 15, 2021

marc-gr deleted the update-geoip branch December 15, 2021 10:33

andrewkroh mentioned this pull request May 2, 2022

[production] Promote many packages from snapshot elastic/package-storage#4302

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate geo ip test results with the new db #2339

Generate geo ip test results with the new db #2339

marc-gr commented Dec 14, 2021 •

edited

Loading

elasticmachine commented Dec 14, 2021

elasticmachine commented Dec 14, 2021

elasticmachine commented Dec 14, 2021 •

edited

Loading

Build stats

Test stats 🧪

`Check integrations / akamai / akamai: check / pipeline test: test-http-json.log – akamai.siem`

`Test integration: akamai`

elasticmachine commented Dec 15, 2021

Build stats

Test stats 🧪

Generate geo ip test results with the new db #2339

Generate geo ip test results with the new db #2339

Conversation

marc-gr commented Dec 14, 2021 • edited Loading

What does this PR do?

Checklist

Author's Checklist

elasticmachine commented Dec 14, 2021

elasticmachine commented Dec 14, 2021

elasticmachine commented Dec 14, 2021 • edited Loading

💔 Tests Failed

Build stats

Test stats 🧪

Test errors

Check integrations / akamai / akamai: check / pipeline test: test-http-json.log – akamai.siem

Steps errors

Test integration: akamai

🤖 GitHub comments

elasticmachine commented Dec 15, 2021

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

marc-gr commented Dec 14, 2021 •

edited

Loading

elasticmachine commented Dec 14, 2021 •

edited

Loading

`Check integrations / akamai / akamai: check / pipeline test: test-http-json.log – akamai.siem`

`Test integration: akamai`