[Cloudflare Logpush] Add non_aws_bucket_name option to AWS S3 input #8278

chemamartinez · 2023-10-24T16:36:29Z

Proposed commit message

It exposes the non_aws_bucket_name of the AWS S3 input for the Cloudflare Logpush integration so users are able to collect logs from a 3rd party S3 bucket. Apart from the global setting, a local one for each data stream has been added so a different bucket can be set for each data stream.

See more about how to use the non_aws_bucket_name here.

The new parameter is called Cloudflare R2 bucket name, as Cloudflare R2 is the S3-compatible storage service that Cloudflare provides by default, so it is assumed to be the one used by its users. Nevertheless, it is also noted that the parameter can be used to specify any other S3-compliant service.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Screenshots

elasticmachine · 2023-10-24T16:37:33Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2023-10-24T16:49:22Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-10-25T11:47:32.058+0000
Duration: 28 min 48 sec

Test stats 🧪

Test	Results
Failed	0
Passed	115
Skipped	0
Total	115

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-10-24T17:06:00Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`18/18`)	💚
Files	100.0% (`18/18`)	💚 2.941
Classes	100.0% (`18/18`)	💚 2.941
Methods	100.0% (`215/215`)	💚 8.877
Lines	91.959% (`4883/5310`)	👎 -0.761
Conditionals	100.0% (`0/0`)	💚

efd6 · 2023-10-24T21:25:24Z

packages/cloudflare_logpush/data_stream/access_request/manifest.yml

+        multi: false
+        required: false
+        show_user: true
+        description: "This can replace Bucket ARN with a Bucket Name for collecting logs from a 3rd party S3 compatible service. It will override the global Non AWS Bucket Name if provided."


I think it would be helpful to have a brief section in the global docs about this. Probably something similar to the text in the docs that you link to in the PR description. Though maybe what you have in the package manifest is enough.

efd6 · 2023-10-24T21:25:43Z

packages/cloudflare_logpush/data_stream/audit/agent/stream/aws-s3.yml.hbs

 {{/each}}
 {{#contains "forwarded" tags}}
 publisher_pipeline.disable_host: true
 {{/contains}}
 {{#if processors}}
 processors:
 {{processors}}
-{{/if}}
+{{/if}}


Final new line (and below).

efd6 · 2023-10-24T21:27:41Z

packages/cloudflare_logpush/manifest.yml

+            multi: false
+            required: false
+            show_user: true
+            description: "This can replace Bucket ARN with a Bucket Name for collecting logs from a 3rd party S3 compatible service. This is a global setting which can be overriden by specific local bucket names for each data stream if required. \nUsing non-AWS S3 compatible buckets requires the use of Access Key ID and Secret Access Key for authentication. To specify the S3 bucket name, use the non_aws_bucket_name config and the endpoint must be set to replace the default API endpoint."


Suggested change

description: "This can replace Bucket ARN with a Bucket Name for collecting logs from a 3rd party S3 compatible service. This is a global setting which can be overriden by specific local bucket names for each data stream if required. \nUsing non-AWS S3 compatible buckets requires the use of Access Key ID and Secret Access Key for authentication. To specify the S3 bucket name, use the non_aws_bucket_name config and the endpoint must be set to replace the default API endpoint."

description: |-

This can replace Bucket ARN with a Bucket Name for collecting logs from a 3rd party S3 compatible service. This is a global setting which can be overriden by specific local bucket names for each data stream if required.

Using non-AWS S3 compatible buckets requires the use of Access Key ID and Secret Access Key for authentication. To specify the S3 bucket name, use the non_aws_bucket_name config and the endpoint must be set to replace the default API endpoint.

packages/cloudflare_logpush/manifest.yml

kcreddy · 2023-10-25T05:33:49Z

packages/cloudflare_logpush/manifest.yml

+            multi: false
+            required: false
+            show_user: true
+            description: "This can replace Bucket ARN with a Bucket Name for collecting logs from a 3rd party S3 compatible service. This is a global setting which can be overriden by specific local bucket names for each data stream if required. \nUsing non-AWS S3 compatible buckets requires the use of Access Key ID and Secret Access Key for authentication. To specify the S3 bucket name, use the non_aws_bucket_name config and the endpoint must be set to replace the default API endpoint."


Probably the wording should be To specify the non-AWS S3 bucket name, use the Non AWS Bucket Name config

kcreddy

LGTM 👍🏼

bhapas

Can you add system test to run on CI. Probably, Localstack as a Non-AWS service?

jamiehynds · 2023-10-25T08:17:19Z

@chemamartinez should we considering renaming thenon_aws_bucket_name setting to Cloudflare R2. Non AWS Bucket Name probably isn't going to resonate with Cloudflare users, if they are looking to ingest via R2, so maybe worth making the name and description more relevant to the Cloudflare integration?

bhapas · 2023-10-25T08:54:59Z

Can you add system test to run on CI. Probably, Localstack as a Non-AWS service?

I think the system tests can be added in #7034

bhapas

LGTM.. You can proceed to merge after @jamiehynds approval

chemamartinez · 2023-10-25T09:42:20Z

@chemamartinez should we considering renaming thenon_aws_bucket_name setting to Cloudflare R2. Non AWS Bucket Name probably isn't going to resonate with Cloudflare users, if they are looking to ingest via R2, so maybe worth making the name and description more relevant to the Cloudflare integration?

@jamiehynds sure! Looks like more user-friendly this way. Here you have the requested changes: da179ed.

elasticmachine · 2023-10-25T14:29:03Z

Package cloudflare_logpush - 1.16.0 containing this change is available at https://epr.elastic.co/search?package=cloudflare_logpush

Add non_aws_bucket_name to AWS S3 input

b3de7ca

chemamartinez added enhancement New feature or request Team:Security-External Integrations Integration:Cloudflare Logpush Cloudflare Logpush labels Oct 24, 2023

chemamartinez self-assigned this Oct 24, 2023

Update changelog

8bb528d

chemamartinez marked this pull request as ready for review October 24, 2023 16:37

chemamartinez requested a review from a team as a code owner October 24, 2023 16:37

efd6 reviewed Oct 24, 2023

View reviewed changes

kcreddy reviewed Oct 25, 2023

View reviewed changes

packages/cloudflare_logpush/manifest.yml Outdated Show resolved Hide resolved

kcreddy reviewed Oct 25, 2023

View reviewed changes

Add requested changes

9eaa1b2

chemamartinez requested review from kcreddy and efd6 October 25, 2023 07:38

kcreddy approved these changes Oct 25, 2023

View reviewed changes

bhapas reviewed Oct 25, 2023

View reviewed changes

efd6 approved these changes Oct 25, 2023

View reviewed changes

bhapas approved these changes Oct 25, 2023

View reviewed changes

Rename non-AWS Bucket Name to Cloudflare R2 Bucket Name

da179ed

chemamartinez force-pushed the cloudflare_logpush_non_aws_bucket branch from 655cf3f to da179ed Compare October 25, 2023 09:40

Update Cloudflare R2 URI in docs

d23d34e

chemamartinez merged commit 19ba187 into elastic:main Oct 25, 2023
4 checks passed

jamiehynds mentioned this pull request Nov 7, 2023

Getting Cloudflare logs via Logpush using Cloudflare R2 service #8412

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Cloudflare Logpush] Add non_aws_bucket_name option to AWS S3 input #8278

[Cloudflare Logpush] Add non_aws_bucket_name option to AWS S3 input #8278

chemamartinez commented Oct 24, 2023 •

edited

Loading

elasticmachine commented Oct 24, 2023

elasticmachine commented Oct 24, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Oct 24, 2023 •

edited

Loading

efd6 Oct 24, 2023 •

edited

Loading

efd6 Oct 24, 2023

efd6 Oct 24, 2023

kcreddy Oct 25, 2023

kcreddy left a comment

bhapas left a comment

jamiehynds commented Oct 25, 2023

bhapas commented Oct 25, 2023

bhapas left a comment

chemamartinez commented Oct 25, 2023

elasticmachine commented Oct 25, 2023

[Cloudflare Logpush] Add non_aws_bucket_name option to AWS S3 input #8278

[Cloudflare Logpush] Add non_aws_bucket_name option to AWS S3 input #8278

Conversation

chemamartinez commented Oct 24, 2023 • edited Loading

Proposed commit message

Checklist

Screenshots

elasticmachine commented Oct 24, 2023

elasticmachine commented Oct 24, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Oct 24, 2023 • edited Loading

🌐 Coverage report

efd6 Oct 24, 2023 • edited Loading

Choose a reason for hiding this comment

efd6 Oct 24, 2023

Choose a reason for hiding this comment

efd6 Oct 24, 2023

Choose a reason for hiding this comment

kcreddy Oct 25, 2023

Choose a reason for hiding this comment

kcreddy left a comment

Choose a reason for hiding this comment

bhapas left a comment

Choose a reason for hiding this comment

jamiehynds commented Oct 25, 2023

bhapas commented Oct 25, 2023

bhapas left a comment

Choose a reason for hiding this comment

chemamartinez commented Oct 25, 2023

elasticmachine commented Oct 25, 2023

chemamartinez commented Oct 24, 2023 •

edited

Loading

elasticmachine commented Oct 24, 2023 •

edited

Loading

elasticmachine commented Oct 24, 2023 •

edited

Loading

efd6 Oct 24, 2023 •

edited

Loading