elastic · andrewkroh · Sep 9, 2022 · Jul 4, 2022 · Jul 4, 2022 · Jul 4, 2022
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
@@ -45,6 +45,7 @@
 /packages/cisco_umbrella @elastic/security-external-integrations
 /packages/cloud_security_posture @elastic/cloud-security-posture
 /packages/cloudflare @elastic/security-external-integrations
+/packages/cloudflare_logpush @elastic/security-external-integrations
 /packages/cockroachdb @elastic/obs-service-integrations
 /packages/containerd @elastic/obs-cloudnative-monitoring
 /packages/crowdstrike @elastic/security-external-integrations

@@ -0,0 +1,3 @@
+dependencies:
+  ecs:
+    reference: git@v8.2.0
@@ -0,0 +1,172 @@
+# Cloudflare Logpush
+
+## Overview
+
+The [Cloudflare Logpush](https://www.cloudflare.com/) integration allows you to monitor Audit, DNS, Firewall Event, HTTP Request, NEL Report, Network Analytics and Spectrum Event Logs. Cloudflare is a content delivery network and DDoS mitigation company. Cloudflare provides a network designed to make everything you connect to the Internet secure, private, fast, and reliable; secure your websites, APIs, and Internet applications; protect corporate networks, employees, and devices; and write and deploy code that runs on the network edge.
+
+The Cloudflare Logpush integration can be used in three different modes to collect data:
+- HTTP Endpoint mode - Cloudflare pushes logs directly to an HTTP endpoint hosted by your Elastic Agent.
+- AWS S3 polling mode - Cloudflare writes data to S3 and Elastic Agent polls the S3 bucket by listing its contents and reading new files.
+- AWS S3 SQS mode - Cloudflare writes data to S3, S3 pushes a new object notification to SQS, Elastic Agent receives the notification from SQS, and then reads the S3 object. Multiple Agents can be used in this mode.
+
+For example, you could use the data from this integration to know which websites have the highest traffic, which areas have the highest network traffic, or observe mitigation statistics.
+
+## Data streams
+
+The Cloudflare Logpush integration collects logs for seven types of events: Audit, DNS, Firewall Event, HTTP Request, NEL Report, Network Analytics, and Spectrum Event.
+
+**Audit**: See Example Schema [here](https://developers.cloudflare.com/logs/reference/log-fields/account/audit_logs/).
+
+**DNS**: See Example Schema [here](https://developers.cloudflare.com/logs/reference/log-fields/zone/dns_logs/).
+
+**Firewall Event**: See Example Schema [here](https://developers.cloudflare.com/logs/reference/log-fields/zone/firewall_events/).
+
+**HTTP Request**: See Example Schema [here](https://developers.cloudflare.com/logs/reference/log-fields/zone/http_requests/).
+
+**NEL Report**: See Example Schema [here](https://developers.cloudflare.com/logs/reference/log-fields/zone/nel_reports/).
+
+**Network Analytics**: See Example Schema [here](https://developers.cloudflare.com/logs/reference/log-fields/account/network_analytics_logs/).
+
+**Spectrum Event**: See Example Schema [here](https://developers.cloudflare.com/logs/reference/log-fields/zone/spectrum_events/).
+
+## Requirements
+
+You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.
+
+This module has been tested against **Cloudflare version v4**.
+
+**Note**: It is recommended to use AWS SQS for Cloudflare Logpush.
+
+## Setup
+
+### To collect data from AWS S3 Bucket, follow the below steps:
+- Configure the [Data Forwarder](https://developers.cloudflare.com/logs/get-started/enable-destinations/aws-s3/) to ingest data into an AWS S3 bucket.
+- The default value of the "Bucket List Prefix" is listed below. However, the user can set the parameter "Bucket List Prefix" according to the requirement.
+
+  | Data Stream Name  | Bucket List Prefix     |
+  | ----------------- | ---------------------- |
+  | Audit Logs        | audit_logs             |
+  | DNS               | dns                    |
+  | Firewall Event    | firewall_event         |
+  | HTTP Request      | http_request           |
+  | NEL Report        | nel_report             |
+  | Network Analytics | network_analytics_logs |
+  | Spectrum Event    | spectrum_event         |
+
+### To collect data from AWS SQS, follow the below steps:
+1. If data forwarding to an AWS S3 Bucket hasn't been configured, then first setup an AWS S3 Bucket as mentioned in the above documentation.
+2. To setup an SQS queue, follow "Step 1: Create an Amazon SQS queue" mentioned in the [Documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ways-to-add-notification-config-to-bucket.html).
+  - While creating an SQS Queue, please provide the same bucket ARN that has been generated after creating an AWS S3 Bucket.
+3. Setup event notification for an S3 bucket. Follow this [Link](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-event-notifications.html).
+  - The user has to perform Step 3 for all the data-streams individually, and each time prefix parameter should be set the same as the S3 Bucket List Prefix as created earlier. (for example, `audit_logs/` for audit data stream.)
+  - For all the event notifications that have been created, select the event type as s3:ObjectCreated:*, select the destination type SQS Queue, and select the queue that has been created in Step 2.
+
+**Note**:
+  - Credentials for the above AWS S3 and SQS input types should be configured using the [link](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-aws-s3.html#aws-credentials-config).
+  - Data collection via AWS S3 Bucket and AWS SQS are mutually exclusive in this case.
+
+### To collect data from the Cloudflare HTTP Endpoint, follow the below steps:
+- Reference link to [Enable HTTP destination](https://developers.cloudflare.com/logs/get-started/enable-destinations/http/) for Cloudflare Logpush.
+- Add same custom header along with its value on both the side for additional security.
+- For example, while creating a job along with a header and value for a particular dataset:
+```
+curl --location --request POST 'https://api.cloudflare.com/client/v4/zones/<ZONE ID>/logpush/jobs' \
+--header 'X-Auth-Key: <X-AUTH-KEY>' \
+--header 'X-Auth-Email: <X-AUTH-EMAIL>' \
+--header 'Authorization: <BASIC AUTHORIZATION>' \
+--header 'Content-Type: application/json' \
+--data-raw '{
+    "name":"<public domain>",
+    "destination_conf": "https://<public domain>:<public port>?header_<secret_header>=<secret_value>",
+    "dataset": "http_requests",
+    "logpull_options": "fields=RayID,EdgeStartTimestamp&timestamps=rfc3339"
+}'
+```
+
+### Enabling the integration in Elastic
+1. In Kibana, go to Management > Integrations
+2. In the integrations search bar type **Cloudflare Logpush**.
+3. Click the **Cloudflare Logpush** integration from the search results.
+4. Click the **Add Cloudflare Logpush** button to add Cloudflare Logpush integration.
+5. Enable the Integration with the HTTP Endpoint or AWS S3 input.
+6. Under the AWS S3 input, there are two types of inputs: using AWS S3 Bucket or using SQS.
+7. Configure Cloudflare to send logs to the Elastic Agent.
+
+## Logs reference
+
+### audit
+
+This is the `audit` dataset.
+Default port for HTTP Endpoint: _9560_
+
+#### Example
+
+{{event "audit"}}
+
+{{fields "audit"}}
+
+### dns
+
+This is the `dns` dataset.
+Default port for HTTP Endpoint: _9561_
+
+#### Example
+
+{{event "dns"}}
+
+{{fields "dns"}}
+
+### firewall_event
+
+This is the `firewall_event` dataset.
+Default port for HTTP Endpoint: _9562_
+
+#### Example
+
+{{event "firewall_event"}}
+
+{{fields "firewall_event"}}
+
+### http_request
+
+This is the `http_request` dataset.
+Default port for HTTP Endpoint: _9563_
+
+#### Example
+
+{{event "http_request"}}
+
+{{fields "http_request"}}
+
+### nel_report
+
+This is the `nel_report` dataset.
+Default port for HTTP Endpoint: _9564_
+
+#### Example
+
+{{event "nel_report"}}
+
+{{fields "nel_report"}}
+
+### network_analytics
+
+This is the `network_analytics` dataset.
+Default port for HTTP Endpoint: _9565_
+
+#### Example
+
+{{event "network_analytics"}}
+
+{{fields "network_analytics"}}
+
+### spectrum_event
+
+This is the `spectrum_event` dataset.
+Default port for HTTP Endpoint: _9566_
+
+#### Example
+
+{{event "spectrum_event"}}
+
+{{fields "spectrum_event"}}
@@ -0,0 +1,58 @@
+version: '2.3'
+services:
+  cloudflare-logpush-audit-http-endpoint:
+    image: docker.elastic.co/observability/stream:v0.7.0
+    volumes:
+      - ./sample_logs:/sample_logs:ro
+    environment:
+      - STREAM_PROTOCOL=webhook
+      - STREAM_ADDR=http://elastic-agent:9560/
+    command: log --start-signal=SIGHUP --delay=5s /sample_logs/audit.log
+  cloudflare-logpush-dns-http-endpoint:
+    image: docker.elastic.co/observability/stream:v0.7.0
+    volumes:
+      - ./sample_logs:/sample_logs:ro
+    environment:
+      - STREAM_PROTOCOL=webhook
+      - STREAM_ADDR=http://elastic-agent:9561/
+    command: log --start-signal=SIGHUP --delay=5s /sample_logs/dns.log
+  cloudflare-logpush-firewall-event-http-endpoint:
+    image: docker.elastic.co/observability/stream:v0.7.0
+    volumes:
+      - ./sample_logs:/sample_logs:ro
+    environment:
+      - STREAM_PROTOCOL=webhook
+      - STREAM_ADDR=http://elastic-agent:9562/
+    command: log --start-signal=SIGHUP --delay=5s /sample_logs/firewall_event.log
+  cloudflare-logpush-http-request-http-endpoint:
+    image: docker.elastic.co/observability/stream:v0.7.0
+    volumes:
+      - ./sample_logs:/sample_logs:ro
+    environment:
+      - STREAM_PROTOCOL=webhook
+      - STREAM_ADDR=http://elastic-agent:9563/
+    command: log --start-signal=SIGHUP --delay=5s /sample_logs/http_request.log
+  cloudflare-logpush-nel-report-http-endpoint:
+    image: docker.elastic.co/observability/stream:v0.7.0
+    volumes:
+      - ./sample_logs:/sample_logs:ro
+    environment:
+      - STREAM_PROTOCOL=webhook
+      - STREAM_ADDR=http://elastic-agent:9564/
+    command: log --start-signal=SIGHUP --delay=5s /sample_logs/nel_report.log
+  cloudflare-logpush-network-analytics-http-endpoint:
+    image: docker.elastic.co/observability/stream:v0.7.0
+    volumes:
+      - ./sample_logs:/sample_logs:ro
+    environment:
+      - STREAM_PROTOCOL=webhook
+      - STREAM_ADDR=http://elastic-agent:9565/
+    command: log --start-signal=SIGHUP --delay=5s /sample_logs/network_analytics.log
+  cloudflare-logpush-spectrum-event-http-endpoint:
+    image: docker.elastic.co/observability/stream:v0.7.0
+    volumes:
+      - ./sample_logs:/sample_logs:ro
+    environment:
+      - STREAM_PROTOCOL=webhook
+      - STREAM_ADDR=http://elastic-agent:9566/
+    command: log --start-signal=SIGHUP --delay=5s /sample_logs/spectrum_event.log
@@ -0,0 +1 @@
+{"ActionResult":true,"ActionType":"token_create","ActorEmail":"user@example.com","ActorID":"enl3j9du8rnx2swwd9l32qots7l54t9s","ActorIP":"81.2.69.142","ActorType":"user","ID":"73fd39ed-5aab-4a2a-b93c-c9a4abf0c425","Interface":"UI","Metadata":{"token_name":"test","token_tag":"b7261c49a793a82678d12285f0bc1401"},"NewValue":{"key1":"value1","key2":"value2"},"OldValue":{"key3":"value4","key4":"value4"},"OwnerID":"enl3j9du8rnx2swwd9l32qots7l54t9s","ResourceID":"enl3j9du8rnx2swwd9l32qots7l54t9s","ResourceType":"account","When":"2021-11-30T20:19:48Z"}
@@ -0,0 +1 @@
+{"ColoCode":"MRS","EDNSSubnet":"1.128.0.0","EDNSSubnetLength":0,"QueryName":"example.com","QueryType":65535,"ResponseCached":false,"ResponseCode":0,"SourceIP":"175.16.199.0","Timestamp":"2022-05-26T09:23:54Z"}
@@ -0,0 +1 @@
+{"ClientRequestScheme":"https","MatchIndex":1,"ClientRefererHost":"abc.example.com","Source":"firewallrules","ClientRequestUserAgent":"Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","ClientRefererPath":"/abc/checkout","Metadata":{"filter":"1ced07e066a34abf8b14f2a99593bc8d","type":"customer"},"EdgeResponseStatus":403,"ClientRequestProtocol":"HTTP/1.1","OriginatorRayID":"00","RayID":"713d477539b55c29","ClientRequestMethod":"GET","ClientIP":"175.16.199.0","ClientRequestPath":"/abc/checkout","Action":"block","Kind":"firewall","RuleID":"7dc666e026974dab84884c73b3e2afe1","ClientIPClass":"searchEngine","ClientASNDescription":"CLOUDFLARENET","ClientCountry":"us","ClientRefererQuery":"?sourcerer=(default%3A(id%3A!n%2CselectedPatterns%3A!(eqldemo%2C%27logs-endpoint.*-eqldemo%27%2C%27logs-system.*-eqldemo%27%2C%27logs-windows.*-eqldemo%27%2Cmetricseqldemo)))&timerange=(global%3A(linkTo%3A!()%2Ctimerange%3A(from%3A%272022-04-05T00%3A00%3A01.199Z%27%2CfromStr%3Anow-24h%2Ckind%3Arelative%2Cto%3A%272022-04-06T00%3A00%3A01.200Z%27%2CtoStr%3Anow))%2Ctimeline%3A(linkTo%3A!()%2Ctimerange%3A(from%3A%272022-04-05T00%3A00%3A01.201Z%27%2CfromStr%3Anow-24h%2Ckind%3Arelative%2Cto%3A%272022-04-06T00%3A00%3A01.202Z%27%2CtoStr%3Anow)))","ClientRequestQuery":"?sourcerer=(default%3A(id%3A!n%2CselectedPatterns%3A!(eqldemo%2C%27logs-endpoint.*-eqldemo%27%2C%27logs-system.*-eqldemo%27%2C%27logs-windows.*-eqldemo%27%2Cmetricseqldemo)))&timerange=(global%3A(linkTo%3A!()%2Ctimerange%3A(from%3A%272022-04-05T00%3A00%3A01.199Z%27%2CfromStr%3Anow-24h%2Ckind%3Arelative%2Cto%3A%272022-04-06T00%3A00%3A01.200Z%27%2CtoStr%3Anow))%2Ctimeline%3A(linkTo%3A!()%2Ctimerange%3A(from%3A%272022-04-05T00%3A00%3A01.201Z%27%2CfromStr%3Anow-24h%2Ckind%3Arelative%2Cto%3A%272022-04-06T00%3A00%3A01.202Z%27%2CtoStr%3Anow)))","OriginResponseStatus":0,"EdgeColoCode":"IAD","ClientRefererScheme":"referer URL scheme","Datetime":"2022-05-31T05:23:43Z","ClientRequestHost":"xyz.example.com","ClientASN":15169}
@@ -0,0 +1 @@
+{"BotScore":"20","BotScoreSrc":"Verified Bot","BotTags":"bing","CacheCacheStatus":"dynamic","CacheResponseBytes":983828,"CacheResponseStatus":200,"CacheTieredFill":false,"ClientASN":43766,"ClientCountry":"sa","ClientDeviceType":"desktop","ClientIP":"175.16.199.0","ClientIPClass":"noRecord","ClientMTLSAuthCertFingerprint":"Fingerprint","ClientMTLSAuthStatus":"unknown","ClientRequestBytes":5800,"ClientRequestHost":"xyz.example.com","ClientRequestMethod":"POST","ClientRequestPath":"/xyz/checkout","ClientRequestProtocol":"HTTP/1.1","ClientRequestReferer":"https://example.com/s/example/default?sourcerer=(default:(id:!n,selectedPatterns:!(example,%27logs-endpoint.*-example%27,%27logs-system.*-example%27,%27logs-windows.*-example%27)))&timerange=(global:(linkTo:!(),timerange:(from:%272022-05-16T06:26:36.340Z%27,fromStr:now-24h,kind:relative,to:%272022-05-17T06:26:36.340Z%27,toStr:now)),timeline:(linkTo:!(),timerange:(from:%272022-04-17T22:00:00.000Z%27,kind:absolute,to:%272022-04-18T21:59:59.999Z%27)))&timeline=(activeTab:notes,graphEventId:%27%27,id:%279844bdd4-4dd6-5b22-ab40-3cd46fce8d6b%27,isOpen:!t)","ClientRequestScheme":"https","ClientRequestSource":"edgeWorkerFetch","ClientRequestURI":"/s/example/api/telemetry/v2/clusters/_stats","ClientRequestUserAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36","ClientSrcPort":0,"ClientSSLCipher":"NONE","ClientSSLProtocol":"TLSv1.2","ClientTCPRTTMs":0,"ClientXRequestedWith":"Request With","Cookies":{"key":"value"},"EdgeCFConnectingO2O":false,"EdgeColoCode":"RUH","EdgeColoID":339,"EdgeEndTimestamp":"2022-05-25T13:25:32Z","EdgePathingOp":"wl","EdgePathingSrc":"macro","EdgePathingStatus":"nr","EdgeRateLimitAction":"unknown","EdgeRateLimitID":0,"EdgeRequestHost":"abc.example.com","EdgeResponseBodyBytes":980397,"EdgeResponseBytes":981308,"EdgeResponseCompressionRatio":0,"EdgeResponseContentType":"application/json","EdgeResponseStatus":200,"EdgeServerIP":"1.128.0.0","EdgeStartTimestamp":"2022-05-25T13:25:26Z","EdgeTimeToFirstByteMs":5333,"OriginDNSResponseTimeMs":3,"OriginIP":"67.43.156.0","OriginRequestHeaderSendDurationMs":0,"OriginResponseBytes":0,"OriginResponseDurationMs":5319,"OriginResponseHeaderReceiveDurationMs":5155,"OriginResponseHTTPExpires":"2022-05-27T13:25:26Z","OriginResponseHTTPLastModified":"2022-05-26T13:25:26Z","OriginResponseStatus":200,"OriginResponseTime":5232000000,"OriginSSLProtocol":"TLSv1.2","OriginTCPHandshakeDurationMs":24,"OriginTLSHandshakeDurationMs":53,"ParentRayID":"710e98d93d50357d","RayID":"710e98d9367f357d","SecurityLevel":"off","SmartRouteColoID":20,"UpperTierColoID":0,"WAFAction":"unknown","WAFFlags":"0","WAFMatchedVar":"example","WAFProfile":"unknown","WAFRuleID":"98d93d5","WAFRuleMessage":"matchad variable message","WorkerCPUTime":0,"WorkerStatus":"unknown","WorkerSubrequest":true,"WorkerSubrequestCount":0,"ZoneID":393347122,"ZoneName":"example.com"}
@@ -0,0 +1 @@
+{"ClientIPASN":"13335","ClientIPASNDescription":"CLOUDFLARENET","ClientIPCountry":"US","LastKnownGoodColoCode":"SJC","Phase":"connection","Timestamp":"2021-07-27T00:01:07Z","Type":"network-error"}
@@ -0,0 +1 @@
+{"AttackCampaignID":"xyz987","AttackID":"abc777","ColoCountry":"AD","ColoGeoHash":"gbuun","ColoID":46,"ColoName":"SJC","Datetime":"2021-07-27T00:01:07Z","DestinationASN":1900,"DestinationASNDescription":"asn description","DestinationCountry":"AD","DestinationGeoHash":"gbuun","DestinationPort":0,"Direction":"ingress","GREChecksum":10,"GREEthertype":10,"GREHeaderLength":1024,"GREKey":10,"GRESequenceNumber":10,"GREVersion":10,"ICMPChecksum":10,"ICMPCode":10,"ICMPType":10,"IPDestinationAddress":"175.16.199.0","IPDestinationSubnet":"/24","IPFragmentOffset":1480,"IPHeaderLength":20,"IPMoreFragments":1480,"IPProtocol":6,"IPProtocolName":"tcp","IPSourceAddress":"67.43.156.0","IPSourceSubnet":"/24","IPTotalLength":1024,"IPTotalLengthBuckets":10,"IPTtl":240,"IPTtlBuckets":2,"IPv4Checksum":0,"IPv4DontFragment":0,"IPv4Dscp":46,"IPv4Ecn":1,"IPv4Identification":1,"IPv4Options":1,"IPv6Dscp":46,"IPv6Ecn":1,"IPv6ExtensionHeaders":"header","IPv6FlowLabel":1,"IPv6Identification":1,"MitigationReason":"BLOCKED","MitigationScope":"local","MitigationSystem":"flowtrackd","Outcome":"pass","ProtocolState":"OPEN","RuleID":"rule1","RulesetID":"3b64149bfa6e4220bbbc2bd6db589552","RulesetOverrideID":"id1","SampleInterval":1,"SourceASN":1500,"SourceASNDescription":"Source ASN Description","SourceCountry":"AD","SourceGeoHash":"gbuun","SourcePort":0,"TCPAcknowledgementNumber":1000,"TCPChecksum":10,"TCPDataOffset":0,"TCPFlags":1,"TCPFlagsString":"Human-readable flags string","TCPMss":512,"TCPOptions":"mss","TCPSackBlocks":1,"TCPSacksPermitted":1,"TCPSequenceNumber":100,"TCPTimestampEcr":100,"TCPTimestampValue":100,"TCPUrgentPointer":10,"TCPWindowScale":10,"TCPWindowSize":10,"UDPChecksum":10,"UDPPayloadLength":10,"Verdict":"pass"}
@@ -0,0 +1 @@
+{"Application":"7ef659a2f8ef4810a9bade96fdad7c75","ClientAsn":200391,"ClientBytes":0,"ClientCountry":"bg","ClientIP":"67.43.156.0","ClientMatchedIpFirewall":"UNKNOWN","ClientPort":40456,"ClientProto":"tcp","ClientTcpRtt":0,"ClientTlsCipher":"UNK","ClientTlsClientHelloServerName":"server name","ClientTlsProtocol":"unknown","ClientTlsStatus":"UNKNOWN","ColoCode":"SOF","ConnectTimestamp":"2022-05-26T09:24:00Z","DisconnectTimestamp":"1970-01-01T00:00:00Z","Event":"connect","IpFirewall":false,"OriginBytes":0,"OriginIP":"175.16.199.0","OriginPort":3389,"OriginProto":"tcp","OriginTcpRtt":0,"OriginTlsCipher":"UNK","OriginTlsFingerprint":"0000000000000000000000000000000000000000000000000000000000000000.","OriginTlsMode":"off","OriginTlsProtocol":"unknown","OriginTlsStatus":"UNKNOWN","ProxyProtocol":"off","Status":0,"Timestamp":"2022-05-26T09:24:00Z"}
@@ -0,0 +1,6 @@
+# newer versions go on top
+- version: 0.1.0
+  changes:
+    - description: Initial Release.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/3643
@@ -0,0 +1,4 @@
+fields:
+  tags:
+    - preserve_original_event
+    - preserve_duplicate_custom_fields