[cloudflare_logpush] Initial Release for the Cloudflare Logpush #3643

vinit-chauhan · 2022-07-04T08:02:56Z

What does this PR do?

Generated the skeleton of the Cloudflare Log Push integration package.
Added a data stream.
Added data collection logic to the data stream.
Added the ingest pipeline for the data stream.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files
Added dashboards and visualizations.
Added test for pipeline for the data stream.
Added system test cases for the data stream.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^7.17.0 || ^8.0.0).

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/cloudflare_logpush directory.
Run the following command to run tests.

elastic-package test

Related issues

Closes Cloudflare Log Push #1727

Screenshots

elasticmachine · 2022-07-04T08:08:01Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-09-06T08:09:52.171+0000
Duration: 20 min 18 sec

Test stats 🧪

Test	Results
Failed	0
Passed	44
Skipped	0
Total	44

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-07-04T08:20:30Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2022-07-04T08:38:00Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`7/7`)	💚
Files	100.0% (`7/7`)	💚 2.89
Classes	100.0% (`7/7`)	💚 2.89
Methods	100.0% (`89/89`)	💚 10.389
Lines	94.37% (`2380/2522`)	👍 2.619
Conditionals	100.0% (`0/0`)	💚

legoguy1000 · 2022-07-15T16:02:29Z

Why a separate integration instead of just adding the new datastreams/inputs to the original one?

jamiehynds · 2022-07-19T11:42:02Z

Why a separate integration instead of just adding the new datastreams/inputs to the original one?

Hey @legoguy1000 - we explored the possibility of adding Logpush to your Logpull integration, but there are several event types that are only supported by Log Push and it would be tricky to align content such as dashboards within a single integration. We're going to update the Log Pull integration to make it clearer that the integration is based on Logpull.

legoguy1000 · 2022-07-19T11:52:06Z

I get the issue with the additional datasets though perhaps since the only 2 datasets in the original cloudflare integration are http and audit logs. My thought is add the httpjson inputs to the new audit and http event data streams and then deprecate the original integration. That way u may not have a conflict and u only have to maintain a single integration/pipelines/dashboards.... going forward.

CyberTaoFlow · 2022-07-19T21:40:11Z

bump!

packages/cloudflare_logpush/_dev/build/docs/README.md

r00tu53r · 2022-07-21T15:03:43Z

@vinit-elastic it would be useful to know a benchmark on the number of events that can be pumped to ES per second (EPS) with this input (say on a single node cluster). Would you be able to run a test check the performance ?

packages/cloudflare_logpush/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

vinit-chauhan · 2022-07-22T15:34:08Z

Hey @r00tu53r

Elastic-Agent is running on one machine, Elasticsearch and Kibana on another machine.

Configuration of the machine in which Elastic-agent is running:
CPU - 2
No of Cores - 1
RAM - 8GB

When no of the workers are set to 5 (Default), then:
Ingestion rate = 46 events/sec
The size of each message is approx = 0.8 KB
Therefore, the volume of data ingested in one day would be around 3.17 GB

When no of the workers are set to 100, then:
Ingestion rate = 440 events/sec
The size of each message is approx = 0.8 KB
Therefore, the volume of data ingested in one day would be around 30 GB

jamiehynds · 2022-07-25T10:04:45Z

@vinit-elastic Logpush delivers logs in less than one minute with batches of no more than 100,000 records per file. Based on the ingestion rate of 440 EPS with 100 quarters, that only gets us to ~25% of what Cloudflare could send in a minute. Do we know if there's additional workers we can add or other variables configuration to ensure we can support that rate of ingestion?

vinit-chauhan · 2022-07-25T13:22:19Z

Hey @jamiehynds - The low ingestion rate may be due to the hardware configuration. Moreover, we have kept the number of workers as a configurable parameter so that if the data rate is huge for some users, they can tweak the number per the requirement.

RichiCoder1 · 2022-07-29T01:37:34Z

This looks awesome! Is there any chance support for the queue form of aws-s3 could be supported too? That's primarily how we have these setup today, and is much more efficient.

packages/cloudflare_logpush/data_stream/audit/agent/stream/http_endpoint.yml.hbs

andrewkroh · 2022-08-02T12:42:40Z

packages/cloudflare_logpush/_dev/build/docs/README.md

+
+### Audit Logs
+
+- Default port for HTTP Endpoint: _9560_


I don't like that we require more than one port to be exposed to the Internet. I've opened a proposal issue to discuss a possible solution in elastic/beats#32578.

andrewkroh · 2022-08-02T12:54:54Z

packages/cloudflare_logpush/data_stream/dns/agent/stream/aws-s3.yml.hbs

@@ -0,0 +1,26 @@
+bucket_arn: {{bucket_arn}}


It seems like the integration should be capable of using SQS notification and S3 polling. Could we create two separate input configurations like aws-s3-polling.yml.hbs and aws-s3-sqs.yml.hbs? If they are separate I think that would be easier to maintain and to configure.

I see that you combined everything into one file, did my recommendation to use aws-s3-polling.yml.hbs and aws-s3-sqs.yml.hbs not work? What problems did you encounter?

We started with keeping a separate file for S3 Polling and SQS ( aws-s3-polling.yml.hbs and aws-s3-sqs.yml.hbs ). However, while doing so, we observed the following behavior.

Though the hbs files were different, the input type was the same input: aws-s3. So, the toggle for both S3 Pooling and S3 SQS was not working properly. So, whenever a user changes the state of one toggle to collect data for only one of the input types, either S3 Pooling or S3 SQS, other one starts automatically. That's the reason why we kept the combined everything into one file.

Moreover, to avoid confusion we've added an indicator in the description of both (S3-Pooling ->[S3] and S3-SQS ->[SQS]) to make it more user-friendly. Now, the users only need to enable the toggle button (Collect logs via S3 Bucket) in order to collect the logs through S3-Polling input and disable it for SQS.

packages/cloudflare_logpush/_dev/build/docs/README.md

packages/cloudflare_logpush/changelog.yml

vinit-chauhan · 2022-08-16T07:13:45Z

/test

packages/cloudflare_logpush/_dev/build/docs/README.md

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

efd6 · 2022-08-30T23:32:15Z

packages/cloudflare_logpush/_dev/build/docs/README.md

+
+## Overview
+
+The [Cloudflare Logpush](https://www.cloudflare.com/) integration allows you to monitor Audit, DNS, Firewall Event, HTTP Request, NEL Report, Network Analytics and Spectrum Event Logs. Cloudflare is content delivery network and DDoS mitigation company. Cloudflare is a global network designed to make everything you connect to the Internet secure, private, fast, and reliable. Secure your websites, APIs, and Internet applications. Protect corporate networks, employees, and devices. Write and deploy code that runs on the network edge.


Suggested change

The [Cloudflare Logpush](https://www.cloudflare.com/) integration allows you to monitor Audit, DNS, Firewall Event, HTTP Request, NEL Report, Network Analytics and Spectrum Event Logs. Cloudflare is content delivery network and DDoS mitigation company. Cloudflare is a global network designed to make everything you connect to the Internet secure, private, fast, and reliable. Secure your websites, APIs, and Internet applications. Protect corporate networks, employees, and devices. Write and deploy code that runs on the network edge.

The [Cloudflare Logpush](https://www.cloudflare.com/) integration allows you to monitor Audit, DNS, Firewall Event, HTTP Request, NEL Report, Network Analytics and Spectrum Event Logs. Cloudflare is a content delivery network and DDoS mitigation company. Cloudflare provides a network designed to make everything you connect to the Internet secure, private, fast, and reliable; secure your websites, APIs, and Internet applications; protect corporate networks, employees, and devices; and write and deploy code that runs on the network edge.

efd6 · 2022-08-30T23:33:50Z

packages/cloudflare_logpush/_dev/build/docs/README.md

+The Cloudflare Logpush integration can be used in three different modes to collect data:
+- HTTP Endpoint mode - Cloudflare pushes logs directly to an HTTP endpoint hosted by your Elastic Agent
+- AWS S3 polling mode - Cloudflare writes data to S3 and Elastic Agent polls the S3 bucket by listing its contents and reading new files
+- AWS S3 SQS mode - Cloudflare writes data to S3, S3 pushes a new object notification to SQS, Elastic Agent receives the notification from SQS, and then reads the S3 object. Multiple Agents can be used in this mode.
+


Suggested change

The Cloudflare Logpush integration can be used in three different modes to collect data:

- HTTP Endpoint mode - Cloudflare pushes logs directly to an HTTP endpoint hosted by your Elastic Agent

- AWS S3 polling mode - Cloudflare writes data to S3 and Elastic Agent polls the S3 bucket by listing its contents and reading new files

- AWS S3 SQS mode - Cloudflare writes data to S3, S3 pushes a new object notification to SQS, Elastic Agent receives the notification from SQS, and then reads the S3 object. Multiple Agents can be used in this mode.

The Cloudflare Logpush integration can be used in three different modes to collect data:

- HTTP Endpoint mode - Cloudflare pushes logs directly to an HTTP endpoint hosted by your Elastic Agent.

- AWS S3 polling mode - Cloudflare writes data to S3 and Elastic Agent polls the S3 bucket by listing its contents and reading new files.

- AWS S3 SQS mode - Cloudflare writes data to S3, S3 pushes a new object notification to SQS, Elastic Agent receives the notification from SQS, and then reads the S3 object. Multiple Agents can be used in this mode.

efd6 · 2022-08-30T23:35:30Z

packages/cloudflare_logpush/_dev/build/docs/README.md

+- AWS S3 SQS mode - Cloudflare writes data to S3, S3 pushes a new object notification to SQS, Elastic Agent receives the notification from SQS, and then reads the S3 object. Multiple Agents can be used in this mode.
+
+
+For example, you could use the data from this integration to know about which websites have the highest traffic, which areas have the highest network traffic, or mitigation statistics.


Suggested change

For example, you could use the data from this integration to know about which websites have the highest traffic, which areas have the highest network traffic, or mitigation statistics.

For example, you could use the data from this integration to know which websites have the highest traffic, which areas have the highest network traffic, or observe mitigation statistics.

efd6 · 2022-08-30T23:36:42Z

packages/cloudflare_logpush/_dev/build/docs/README.md

+
+## Data streams
+
+The Cloudflare Logpush integration collects logs for seven types of events: Audit, DNS, Firewall Event, HTTP Request, NEL Report, Network Analytics and Spectrum Event.


Suggested change

The Cloudflare Logpush integration collects logs for seven types of events: Audit, DNS, Firewall Event, HTTP Request, NEL Report, Network Analytics and Spectrum Event.

The Cloudflare Logpush integration collects logs for seven types of events: Audit, DNS, Firewall Event, HTTP Request, NEL Report, Network Analytics, and Spectrum Event.

efd6 · 2022-08-30T23:52:06Z

packages/cloudflare_logpush/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

+      value: [authentication]
+  - date:
+      field: json.When
+      if: ctx.json?.When != null && ctx.json?.When != ''


Suggested change

if: ctx.json?.When != null && ctx.json?.When != ''

if: ctx.json?.When != null && ctx.json.When != ''

efd6 · 2022-08-31T01:10:53Z

...ages/cloudflare_logpush/data_stream/spectrum_event/elasticsearch/ingest_pipeline/default.yml

+      ignore_failure: true
+  - date:
+      field: json.ConnectTimestamp
+      if: ctx.json?.ConnectTimestamp != null && ctx.json?.ConnectTimestamp != ''


Suggested change

if: ctx.json?.ConnectTimestamp != null && ctx.json?.ConnectTimestamp != ''

if: ctx.json?.ConnectTimestamp != null && ctx.json.ConnectTimestamp != ''

efd6 · 2022-08-31T01:11:05Z

...ages/cloudflare_logpush/data_stream/spectrum_event/elasticsearch/ingest_pipeline/default.yml

+            value: '{{{_ingest.on_failure_message}}}'
+  - date:
+      field: json.DisconnectTimestamp
+      if: ctx.json?.DisconnectTimestamp != null && ctx.json?.DisconnectTimestamp != ''


Suggested change

if: ctx.json?.DisconnectTimestamp != null && ctx.json?.DisconnectTimestamp != ''

if: ctx.json?.DisconnectTimestamp != null && ctx.json.DisconnectTimestamp != ''

efd6 · 2022-08-31T01:12:58Z

packages/cloudflare_logpush/data_stream/spectrum_event/sample_event.json

+                "time": "2022-05-26T09:24:00.000Z"
+            },
+            "disconnect": {
+                "time": "1970-01-01T00:00:00.000Z"


This looks like an empty value has been used. I can see that it's like that in the event.original, but is that real?

Yes, We got this value in the actual response from Cloudflare itself.

efd6 · 2022-08-31T01:14:29Z

...ages/cloudflare_logpush/data_stream/spectrum_event/elasticsearch/ingest_pipeline/default.yml

+        - yyyy-MM-dd'T'HH:mm:ss.SSSZ
+        - UNIX_MS
+      timezone: UTC
+      target_field: cloudflare_logpush.spectrum_event.connect.time


Maybe also set event.start from this.

efd6 · 2022-08-31T01:14:44Z

...ages/cloudflare_logpush/data_stream/spectrum_event/elasticsearch/ingest_pipeline/default.yml

+        - yyyy-MM-dd'T'HH:mm:ss.SSSZ
+        - UNIX_MS
+      timezone: UTC
+      target_field: cloudflare_logpush.spectrum_event.disconnect.time


Set event.end from this?

vinit-chauhan added 2 commits July 4, 2022 12:04

Initial Release for the Cloudflare Logpush

6849bc8

Updated the changelog file.

ca0443f

Add integration package in CODEOWNERS file.

027ee14

vinit-chauhan added enhancement New feature or request Team:Security-External Integrations New Integration labels Jul 4, 2022

vinit-chauhan changed the title ~~Initial Release for the Cloudflare Logpush~~ [cloudflare_logpush] Initial Release for the Cloudflare Logpush Jul 18, 2022

jamiehynds reviewed Jul 21, 2022

View reviewed changes

packages/cloudflare_logpush/_dev/build/docs/README.md Outdated Show resolved Hide resolved

jamiehynds reviewed Jul 21, 2022

View reviewed changes

packages/cloudflare_logpush/_dev/build/docs/README.md Outdated Show resolved Hide resolved

r00tu53r reviewed Jul 21, 2022

View reviewed changes

packages/cloudflare_logpush/data_stream/audit/elasticsearch/ingest_pipeline/default.yml Show resolved Hide resolved

Updated the README file.

ec557da

Added on_failure in date and convert processor

295433b

andrewkroh mentioned this pull request Aug 2, 2022

[Filebeat] http_endpoint input instances should be able to "share" port elastic/beats#32578

Closed

andrewkroh reviewed Aug 2, 2022

View reviewed changes

andrewkroh added the Integration:Cloudflare label Aug 2, 2022

Added Support of SQS input type

fe97668

Update a user guide for SQS input type

a0e704e

Added support of Authentication Header in HTTP Endpoint input

f6b1798

andrewkroh reviewed Aug 26, 2022

View reviewed changes

packages/cloudflare_logpush/_dev/build/docs/README.md Outdated Show resolved Hide resolved

Update packages/cloudflare_logpush/_dev/build/docs/README.md

d61ab4d

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

jamiehynds mentioned this pull request Aug 29, 2022

[Cloudflare] Growing Delay in Logpull after network/other issues #4088

Closed

Rebuilt the README file

2714950

vinit-chauhan force-pushed the package_cloudflare_logpush branch from 1918aa4 to 2714950 Compare August 29, 2022 10:45

efd6 reviewed Aug 31, 2022

View reviewed changes

vinit-chauhan added 3 commits August 31, 2022 19:04

Changes as per review comments

a8db26c

Add support for geo_point data type

b64eb56

Update the descriptions in manifest

ba2ec43

andrewkroh approved these changes Sep 7, 2022

View reviewed changes

efd6 approved these changes Sep 9, 2022

View reviewed changes

andrewkroh merged commit 7deb897 into elastic:main Sep 9, 2022

andrewkroh mentioned this pull request Sep 13, 2022

[production] Promote many packages from snapshot elastic/package-storage#5506

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[cloudflare_logpush] Initial Release for the Cloudflare Logpush #3643

[cloudflare_logpush] Initial Release for the Cloudflare Logpush #3643

vinit-chauhan commented Jul 4, 2022 •

edited

Loading

elasticmachine commented Jul 4, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Jul 4, 2022

elasticmachine commented Jul 4, 2022 •

edited

Loading

legoguy1000 commented Jul 15, 2022

jamiehynds commented Jul 19, 2022

legoguy1000 commented Jul 19, 2022

CyberTaoFlow commented Jul 19, 2022

r00tu53r commented Jul 21, 2022

vinit-chauhan commented Jul 22, 2022 •

edited

Loading

jamiehynds commented Jul 25, 2022

vinit-chauhan commented Jul 25, 2022

RichiCoder1 commented Jul 29, 2022 •

edited

Loading

andrewkroh Aug 2, 2022

andrewkroh Aug 2, 2022

andrewkroh Aug 26, 2022

vinit-chauhan Aug 29, 2022

vinit-chauhan commented Aug 16, 2022

efd6 Aug 30, 2022

efd6 Aug 30, 2022

efd6 Aug 30, 2022

efd6 Aug 30, 2022

efd6 Aug 30, 2022

efd6 Aug 31, 2022

efd6 Aug 31, 2022

efd6 Aug 31, 2022

vinit-chauhan Aug 31, 2022

efd6 Aug 31, 2022

efd6 Aug 31, 2022


		## Overview

		The [Cloudflare Logpush](https://www.cloudflare.com/) integration allows you to monitor Audit, DNS, Firewall Event, HTTP Request, NEL Report, Network Analytics and Spectrum Event Logs. Cloudflare is content delivery network and DDoS mitigation company. Cloudflare is a global network designed to make everything you connect to the Internet secure, private, fast, and reliable. Secure your websites, APIs, and Internet applications. Protect corporate networks, employees, and devices. Write and deploy code that runs on the network edge.

		- AWS S3 SQS mode - Cloudflare writes data to S3, S3 pushes a new object notification to SQS, Elastic Agent receives the notification from SQS, and then reads the S3 object. Multiple Agents can be used in this mode.


		For example, you could use the data from this integration to know about which websites have the highest traffic, which areas have the highest network traffic, or mitigation statistics.


		## Data streams

		The Cloudflare Logpush integration collects logs for seven types of events: Audit, DNS, Firewall Event, HTTP Request, NEL Report, Network Analytics and Spectrum Event.

	if: ctx.json?.When != null && ctx.json?.When != ''
	if: ctx.json?.When != null && ctx.json.When != ''

	if: ctx.json?.ConnectTimestamp != null && ctx.json?.ConnectTimestamp != ''
	if: ctx.json?.ConnectTimestamp != null && ctx.json.ConnectTimestamp != ''

	if: ctx.json?.DisconnectTimestamp != null && ctx.json?.DisconnectTimestamp != ''
	if: ctx.json?.DisconnectTimestamp != null && ctx.json.DisconnectTimestamp != ''

[cloudflare_logpush] Initial Release for the Cloudflare Logpush #3643

[cloudflare_logpush] Initial Release for the Cloudflare Logpush #3643

Conversation

vinit-chauhan commented Jul 4, 2022 • edited Loading

What does this PR do?

Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Jul 4, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Jul 4, 2022

elasticmachine commented Jul 4, 2022 • edited Loading

🌐 Coverage report

legoguy1000 commented Jul 15, 2022

jamiehynds commented Jul 19, 2022

legoguy1000 commented Jul 19, 2022

CyberTaoFlow commented Jul 19, 2022

r00tu53r commented Jul 21, 2022

vinit-chauhan commented Jul 22, 2022 • edited Loading

jamiehynds commented Jul 25, 2022

vinit-chauhan commented Jul 25, 2022

RichiCoder1 commented Jul 29, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vinit-chauhan commented Aug 16, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vinit-chauhan commented Jul 4, 2022 •

edited

Loading

elasticmachine commented Jul 4, 2022 •

edited

Loading

elasticmachine commented Jul 4, 2022 •

edited

Loading

vinit-chauhan commented Jul 22, 2022 •

edited

Loading

RichiCoder1 commented Jul 29, 2022 •

edited

Loading