[aws.securityhub_findings] Correcting pipeline to utilize ecs network.direction and threat.indicator.type allowed values
#7900
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kgeller
added
bug
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
🤖 GitHub commentsExpand to view the GitHub comments
To re-run your PR in the CI, just comment with:
|
🌐 Coverage report
|
kgeller
commented
Sep 20, 2023
Comment on lines
1549
to
1555
| "DOMAIN": domain-name | ||
| "EMAIL_ADDRESS": email-addr | ||
| "IPV4_ADDRESS": ipv4-addr | ||
| "IPV6_ADDRESS": ipv6-addr | ||
| "MUTEX": mutex | ||
| "PROCESS": process | ||
| "URL": url |
There was a problem hiding this comment.
AWS has some other possible values documented (HASH_MD5 HASH_SHA1 HASH_SHA256 HASH_SHA512), but I wouldn't say they really correspond to any of the ECS allowed values, so I omitted them for now.
There was a problem hiding this comment.
Could those be interpreted as threat.indicator.type: file where we then set threat.enrichments.indicator.file.hash.{md5,sha1,sha256,sha512}: <hash>?
There was a problem hiding this comment.
I can definitely set type:file for those cases. But I can't figure out for sure either way if the actual hash gets sent on the message as well. We don't have any examples of this type in the sample logs, and I am struggling to find anything that helpful on the aws docs.
There was a problem hiding this comment.
I suspect the hash is passed through in the value field of the ThreadIntelIndicator. Like this part of the event:
It would need logic to set the threat.enrichments.indicator.file.hash.{md5,sha1,sha256,sha512} value. I would probably synthesize a document to test the pipeline.
There was a problem hiding this comment.
Updated @andrewkroh
|
Pinging @elastic/security-external-integrations (Team:Security-External Integrations) |
packages/aws/data_stream/securityhub_findings/elasticsearch/ingest_pipeline/default.yml
Outdated
Show resolved
Hide resolved
packages/aws/data_stream/securityhub_findings/elasticsearch/ingest_pipeline/default.yml
Outdated
Show resolved
Hide resolved
chemamartinez
approved these changes
Sep 22, 2023
kgeller
commented
Sep 26, 2023
| @@ -1,3 +1,4 @@ | |||
| {"Action":{"ActionType":"PORT_PROBE","PortProbeAction":{"PortProbeDetails":[{"LocalPortDetails":{"Port":80,"PortName":"HTTP"},"LocalIpDetails":{"IpAddressV4":"1.128.0.0"},"RemoteIpDetails":{"Country":{"CountryName":"Example Country"},"City":{"CityName":"Example City"},"GeoLocation":{"Lon":0,"Lat":0},"Organization":{"AsnOrg":"ExampleASO","Org":"ExampleOrg","Isp":"ExampleISP","Asn":64496}}}],"Blocked":false}},"AwsAccountId":"111111111111","CompanyName":"AWS","Compliance":{"RelatedRequirements":["Req1","Req2"],"Status":"PASSED","StatusReasons":[{"ReasonCode":"CLOUDWATCH_ALARMS_NOT_PRESENT","Description":"CloudWatch alarms do not exist in the account"}]},"Confidence":42,"CreatedAt":"2017-03-22T13:22:13.933Z","Criticality":99,"Description":"The version of openssl found on instance i-abcd1234 is known to contain a vulnerability.","FindingProviderFields":{"Confidence":42,"Criticality":99,"RelatedFindings":[{"ProductArn":"arn:aws:securityhub:us-west-2::product/aws/guardduty","Id":"123e4567-e89b-12d3-a456-426655440000"}],"Severity":{"Label":"MEDIUM","Original":"MEDIUM"},"Types":["Software and Configuration Checks/Vulnerabilities/CVE"]},"FirstObservedAt":"2017-03-22T13:22:13.933Z","GeneratorId":"acme-vuln-9ab348","Id":"us-west-2/111111111111/98aebb2207407c87f51e89943f12b1ef","LastObservedAt":"2017-03-23T13:22:13.933Z","Malware":[{"Name":"Stringler","Type":"COIN_MINER","Path":"/usr/sbin/stringler","State":"OBSERVED"}],"Network":{"Direction":"IN","OpenPortRange":{"Begin":443,"End":443},"Protocol":"TCP","SourceIpV4":"1.128.0.0","SourceIpV6":"2a02:cf40::","SourcePort":"42","SourceDomain":"example1.com","SourceMac":"00:0d:83:b1:c0:8e","DestinationIpV4":"1.128.0.0","DestinationIpV6":"2a02:cf40::","DestinationPort":"80","DestinationDomain":"example2.com"},"NetworkPath":[{"ComponentId":"abc-01a234bc56d8901ee","ComponentType":"AWS::EC2::InternetGateway","Egress":{"Destination":{"Address":["1.128.0.0/24"],"PortRanges":[{"Begin":443,"End":443}]},"Protocol":"TCP","Source":{"Address":["175.16.199.1/24"]}},"Ingress":{"Destination":{"Address":["175.16.199.1/24"],"PortRanges":[{"Begin":443,"End":443}]},"Protocol":"TCP","Source":{"Address":["175.16.199.1/24"]}}}],"Note":{"Text":"Don't forget to check under the mat.","UpdatedBy":"jsmith","UpdatedAt":"2018-08-31T00:15:09Z"},"PatchSummary":{"Id":"pb-123456789098","InstalledCount":"100","MissingCount":"100","FailedCount":"0","InstalledOtherCount":"1023","InstalledRejectedCount":"0","InstalledPendingReboot":"0","OperationStartTime":"2018-09-27T23:37:31Z","OperationEndTime":"2018-09-27T23:39:31Z","RebootOption":"RebootIfNeeded","Operation":"Install"},"Process":{"Name":"syslogd","Path":"/usr/sbin/syslogd","Pid":12345,"ParentPid":56789,"LaunchedAt":"2018-09-27T22:37:31Z","TerminatedAt":"2018-09-27T23:37:31Z"},"ProductArn":"arn:aws:securityhub:us-east-1:111111111111:product/111111111111/default","ProductFields":{"generico/secure-pro/Count":"6","Service_Name":"cloudtrail.amazonaws.com","aws/inspector/AssessmentTemplateName":"My daily CVE assessment","aws/inspector/AssessmentTargetName":"My prod env","aws/inspector/RulesPackageName":"Common Vulnerabilities and Exposures"},"ProductName":"Security Hub","RecordState":"ACTIVE","Region":"us-east-1","RelatedFindings":[{"ProductArn":"arn:aws:securityhub:us-west-2::product/aws/guardduty","Id":"123e4567-e89b-12d3-a456-426655440000"},{"ProductArn":"arn:aws:securityhub:us-west-2::product/aws/guardduty","Id":"AcmeNerfHerder-111111111111-x189dx7824"}],"Remediation":{"Recommendation":{"Text":"Run sudo yum update and cross your fingers and toes.","Url":"http://myfp.com/recommendations/dangerous_things_and_how_to_fix_them.html"}},"Resources":[{"Type":"AwsEc2Instance","Id":"i-cafebabe","Partition":"aws","Region":"us-west-2","Tags":{"billingCode":"Lotus-1-2-3","needsPatching":"true"},"Details":{"IamInstanceProfileArn":"arn:aws:iam::123456789012:role/IamInstanceProfileArn","ImageId":"ami-79fd7eee","IpV4Addresses":["175.16.199.1"],"IpV6Addresses":["2a02:cf40::"],"KeyName":"testkey","LaunchedAt":"2018-09-29T01:25:54Z","MetadataOptions":{"HttpEndpoint":"enabled","HttpProtocolIpv6":"enabled","HttpPutResponseHopLimit":1,"HttpTokens":"optional","InstanceMetadataTags":"disabled"},"NetworkInterfaces":[{"NetworkInterfaceId":"eni-e5aa89a3"}],"SubnetId":"PublicSubnet","Type":"i3.xlarge","VirtualizationType":"hvm","VpcId":"TestVPCIpv6"}}],"Sample":true,"SchemaVersion":"2018-10-08","Severity":{"Label":"CRITICAL","Original":"8.3"},"SourceUrl":"http://threatintelweekly.org/backdoors/8888","ThreatIntelIndicators":[{"Type":"IPV4_ADDRESS","Value":"175.16.199.1","Category":"BACKDOOR","LastObservedAt":"2018-09-27T23:37:31Z","Source":"Threat Intel Weekly","SourceUrl":"http://threatintelweekly.org/backdoors/8888"}],"Threats":[{"FilePaths":[{"FileName":"b.txt","FilePath":"/tmp/b.txt","Hash":"sha256","ResourceId":"arn:aws:ec2:us-west-2:123456789012:volume/vol-032f3bdd89aee112f"}],"ItemCount":3,"Name":"Iot.linux.mirai.vwisi","Severity":"HIGH"}],"Title":"EC2.20 Both VPN tunnels for an AWS Site-to-Site VPN connection should be up","Types":["Software and Configuration Checks/Vulnerabilities/CVE"],"UpdatedAt":"2018-08-31T00:15:09Z","UserDefinedFields":{"reviewedByCio":"true","comeBackToLater":"Check this again on Monday"},"VerificationState":"UNKNOWN","Vulnerabilities":[{"Cvss":[{"BaseScore":4.7,"BaseVector":"AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:N/A:N","Version":"V3"},{"BaseScore":4.7,"BaseVector":"AV:L/AC:M/Au:N/C:C/I:N/A:N","Version":"V2"}],"Id":"CVE-2020-12345","ReferenceUrls":["http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-12418","http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17563"],"RelatedVulnerabilities":["CVE-2020-12345"],"Vendor":{"Name":"Alas","Url":"https://alas.aws.amazon.com/ALAS-2020-1337.html","VendorCreatedAt":"2020-01-16T00:01:43Z","VendorSeverity":"Medium","VendorUpdatedAt":"2020-01-16T00:01:43Z"},"VulnerablePackages":[{"Architecture":"x86_64","Epoch":"1","Name":"openssl","Release":"16.amzn2.0.3","Version":"1.0.2k"}]}],"Workflow":{"Status":"NEW"},"WorkflowState":"NEW"} | |||
| {"Action":{"ActionType":"PORT_PROBE","PortProbeAction":{"PortProbeDetails":[{"LocalPortDetails":{"Port":80,"PortName":"HTTP"},"LocalIpDetails":{"IpAddressV4":"1.128.0.0"},"RemoteIpDetails":{"Country":{"CountryName":"Example Country"},"City":{"CityName":"Example City"},"GeoLocation":{"Lon":0,"Lat":0},"Organization":{"AsnOrg":"ExampleASO","Org":"ExampleOrg","Isp":"ExampleISP","Asn":64496}}}],"Blocked":false}},"AwsAccountId":"111111111111","CompanyName":"AWS","Compliance":{"RelatedRequirements":["Req1","Req2"],"Status":"PASSED","StatusReasons":[{"ReasonCode":"CLOUDWATCH_ALARMS_NOT_PRESENT","Description":"CloudWatch alarms do not exist in the account"}]},"Confidence":42,"CreatedAt":"2017-03-22T13:22:13.933Z","Criticality":99,"Description":"The version of openssl found on instance i-abcd1234 is known to contain a vulnerability.","FindingProviderFields":{"Confidence":42,"Criticality":99,"RelatedFindings":[{"ProductArn":"arn:aws:securityhub:us-west-2::product/aws/guardduty","Id":"123e4567-e89b-12d3-a456-426655440000"}],"Severity":{"Label":"MEDIUM","Original":"MEDIUM"},"Types":["Software and Configuration Checks/Vulnerabilities/CVE"]},"FirstObservedAt":"2017-03-22T13:22:13.933Z","GeneratorId":"acme-vuln-9ab348","Id":"us-west-2/111111111111/98aebb2207407c87f51e89943f12b1ef","LastObservedAt":"2017-03-23T13:22:13.933Z","Malware":[{"Name":"Stringler","Type":"COIN_MINER","Path":"/usr/sbin/stringler","State":"OBSERVED"}],"Network":{"Direction":"IN","OpenPortRange":{"Begin":443,"End":443},"Protocol":"TCP","SourceIpV4":"1.128.0.0","SourceIpV6":"2a02:cf40::","SourcePort":"42","SourceDomain":"example1.com","SourceMac":"00:0d:83:b1:c0:8e","DestinationIpV4":"1.128.0.0","DestinationIpV6":"2a02:cf40::","DestinationPort":"80","DestinationDomain":"example2.com"},"NetworkPath":[{"ComponentId":"abc-01a234bc56d8901ee","ComponentType":"AWS::EC2::InternetGateway","Egress":{"Destination":{"Address":["1.128.0.0/24"],"PortRanges":[{"Begin":443,"End":443}]},"Protocol":"TCP","Source":{"Address":["175.16.199.1/24"]}},"Ingress":{"Destination":{"Address":["175.16.199.1/24"],"PortRanges":[{"Begin":443,"End":443}]},"Protocol":"TCP","Source":{"Address":["175.16.199.1/24"]}}}],"Note":{"Text":"Don't forget to check under the mat.","UpdatedBy":"jsmith","UpdatedAt":"2018-08-31T00:15:09Z"},"PatchSummary":{"Id":"pb-123456789098","InstalledCount":"100","MissingCount":"100","FailedCount":"0","InstalledOtherCount":"1023","InstalledRejectedCount":"0","InstalledPendingReboot":"0","OperationStartTime":"2018-09-27T23:37:31Z","OperationEndTime":"2018-09-27T23:39:31Z","RebootOption":"RebootIfNeeded","Operation":"Install"},"Process":{"Name":"syslogd","Path":"/usr/sbin/syslogd","Pid":12345,"ParentPid":56789,"LaunchedAt":"2018-09-27T22:37:31Z","TerminatedAt":"2018-09-27T23:37:31Z"},"ProductArn":"arn:aws:securityhub:us-east-1:111111111111:product/111111111111/default","ProductFields":{"generico/secure-pro/Count":"6","Service_Name":"cloudtrail.amazonaws.com","aws/inspector/AssessmentTemplateName":"My daily CVE assessment","aws/inspector/AssessmentTargetName":"My prod env","aws/inspector/RulesPackageName":"Common Vulnerabilities and Exposures"},"ProductName":"Security Hub","RecordState":"ACTIVE","Region":"us-east-1","RelatedFindings":[{"ProductArn":"arn:aws:securityhub:us-west-2::product/aws/guardduty","Id":"123e4567-e89b-12d3-a456-426655440000"},{"ProductArn":"arn:aws:securityhub:us-west-2::product/aws/guardduty","Id":"AcmeNerfHerder-111111111111-x189dx7824"}],"Remediation":{"Recommendation":{"Text":"Run sudo yum update and cross your fingers and toes.","Url":"http://myfp.com/recommendations/dangerous_things_and_how_to_fix_them.html"}},"Resources":[{"Type":"AwsEc2Instance","Id":"i-cafebabe","Partition":"aws","Region":"us-west-2","Tags":{"billingCode":"Lotus-1-2-3","needsPatching":"true"},"Details":{"IamInstanceProfileArn":"arn:aws:iam::123456789012:role/IamInstanceProfileArn","ImageId":"ami-79fd7eee","IpV4Addresses":["175.16.199.1"],"IpV6Addresses":["2a02:cf40::"],"KeyName":"testkey","LaunchedAt":"2018-09-29T01:25:54Z","MetadataOptions":{"HttpEndpoint":"enabled","HttpProtocolIpv6":"enabled","HttpPutResponseHopLimit":1,"HttpTokens":"optional","InstanceMetadataTags":"disabled"},"NetworkInterfaces":[{"NetworkInterfaceId":"eni-e5aa89a3"}],"SubnetId":"PublicSubnet","Type":"i3.xlarge","VirtualizationType":"hvm","VpcId":"TestVPCIpv6"}}],"Sample":true,"SchemaVersion":"2018-10-08","Severity":{"Label":"CRITICAL","Original":"8.3"},"SourceUrl":"http://threatintelweekly.org/backdoors/8888","ThreatIntelIndicators":[{"Type":"HASH_MD5","Value":"ae2b1fca515949e5d54fb22b8ed95575","Category":"BACKDOOR","LastObservedAt":"2018-09-27T23:37:31Z","Source":"Threat Intel Weekly","SourceUrl":"http://threatintelweekly.org/backdoors/8888"}],"Threats":[{"FilePaths":[{"FileName":"b.txt","FilePath":"/tmp/b.txt","Hash":"sha256","ResourceId":"arn:aws:ec2:us-west-2:123456789012:volume/vol-032f3bdd89aee112f"}],"ItemCount":3,"Name":"Iot.linux.mirai.vwisi","Severity":"HIGH"}],"Title":"EC2.20 Both VPN tunnels for an AWS Site-to-Site VPN connection should be up","Types":["Software and Configuration Checks/Vulnerabilities/CVE"],"UpdatedAt":"2018-08-31T00:15:09Z","UserDefinedFields":{"reviewedByCio":"true","comeBackToLater":"Check this again on Monday"},"VerificationState":"UNKNOWN","Vulnerabilities":[{"Cvss":[{"BaseScore":4.7,"BaseVector":"AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:N/A:N","Version":"V3"},{"BaseScore":4.7,"BaseVector":"AV:L/AC:M/Au:N/C:C/I:N/A:N","Version":"V2"}],"Id":"CVE-2020-12345","ReferenceUrls":["http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-12418","http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17563"],"RelatedVulnerabilities":["CVE-2020-12345"],"Vendor":{"Name":"Alas","Url":"https://alas.aws.amazon.com/ALAS-2020-1337.html","VendorCreatedAt":"2020-01-16T00:01:43Z","VendorSeverity":"Medium","VendorUpdatedAt":"2020-01-16T00:01:43Z"},"VulnerablePackages":[{"Architecture":"x86_64","Epoch":"1","Name":"openssl","Release":"16.amzn2.0.3","Version":"1.0.2k"}]}],"Workflow":{"Status":"NEW"},"WorkflowState":"NEW"} | |||
There was a problem hiding this comment.
Same exact line as above, but swapped out the ThreatIntelIndicators values for "Type":"HASH_MD5","Value":"ae2b1fca515949e5d54fb22b8ed95575"
andrewkroh
approved these changes
Sep 27, 2023
zmoog
approved these changes
Sep 27, 2023
kaiyan-sheng
approved these changes
Sep 27, 2023
|
Package aws - 2.4.1 containing this change is available at https://epr.elastic.co/search?package=aws |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
ECS fields
network.directionandthreat.indicator.typefields have a prescribed set of allowed values and this updates the data stream to utilize them.Checklist
changelog.ymlfile.I have verified that Kibana version constraints are current according to guidelines.Author's Checklist
How to test this PR locally
Related issues
Screenshots