[Filebeat] - S3 Input - Add support for only iterating/accessing only… #28252

aspacca · 2021-10-05T13:17:44Z

… specific folders or datapaths

Enhancement

What does this PR do?

We added a config param that defines the prefix datapath in which we run the list-objects API on.

Why is it important?

Certain hosted S3 buckets, managed by third parties, will give access to only specific datapaths, meaning that the S3 input will always get access denied, because it tries to iterate over the whole S3 bucket.

For example Cisco managed S3 buckets for Umbrella:
https://docs.umbrella.com/deployment-umbrella/docs/cisco-managed-s3-bucket

This only allows you to run the list-objects API inside the specified data path, , else it will just return 403 Access Denied.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes #27965

Use cases

Screenshots

Logs

… specific folders or datapaths

mergify · 2021-10-05T13:19:14Z

This pull request does not have a backport label. Could you fix it @aspacca? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2021-10-05T13:19:42Z

Pinging @elastic/integrations (Team:Integrations)

botelastic · 2021-10-05T13:19:53Z

This pull request doesn't have a Team:<team> label.

elasticmachine · 2021-10-05T13:35:45Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Duration: 94 min 23 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.

aspacca · 2021-10-05T14:59:09Z

x-pack/filebeat/input/awss3/input.go

@@ -269,5 +271,5 @@ func getRegionForBucketARN(ctx context.Context, s3Client *s3.Client, bucketARN s
 		return "", err
 	}

-	return string(resp.LocationConstraint), nil
+	return string(s3.NormalizeBucketLocation(resp.LocationConstraint)), nil


only this change should be backported to 7.15

kaiyan-sheng · 2021-10-05T16:28:02Z

filebeat/docs/modules/aws.asciidoc

@@ -178,6 +184,10 @@ Use to vertically scale the input.

 Wait interval between completion of a list request to the S3 bucket and beginning of the next one. Default to be 120 seconds.

+*`var.bucket_bucket_list_prefix`*::


This should be bucket_list_prefix?

good catch! 👍
fixed

mergify · 2021-10-06T14:31:52Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b Filebeat-S3-Input-Add-support-for-only-iterating-accessing-only-specific-folders-or-datapaths upstream/Filebeat-S3-Input-Add-support-for-only-iterating-accessing-only-specific-folders-or-datapaths
git merge upstream/master
git push upstream Filebeat-S3-Input-Add-support-for-only-iterating-accessing-only-specific-folders-or-datapaths

…rating-accessing-only-specific-folders-or-datapaths

mergify · 2021-10-07T02:20:41Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b Filebeat-S3-Input-Add-support-for-only-iterating-accessing-only-specific-folders-or-datapaths upstream/Filebeat-S3-Input-Add-support-for-only-iterating-accessing-only-specific-folders-or-datapaths
git merge upstream/master
git push upstream Filebeat-S3-Input-Add-support-for-only-iterating-accessing-only-specific-folders-or-datapaths

…rating-accessing-only-specific-folders-or-datapaths

kaiyan-sheng

LGTM! This is a great feature to add!!

#28252) * [Filebeat] - S3 Input - Add support for only iterating/accessing only specific folders or datapaths (cherry picked from commit 764045e)

elastic#28252) * [Filebeat] - S3 Input - Add support for only iterating/accessing only specific folders or datapaths

…terating/accessing only… (#28369) * [Filebeat] - S3 Input - Add support for only iterating/accessing only specific folders or datapaths (cherry picked from commit 764045e) Co-authored-by: Andrea Spacca <andrea.spacca@elastic.co>

* singleton sysinfo host to avoid frequently collecting host info * add Host object to Stats object * update changelog * set procStats.host to nil if any error calling sysinfo.Host() * Update aws-lambda-go library version to 1.13.3 (#28236) * [cloud][docker] use the private docker namespace (#28286) * [7.x] [DOCS] Update api_key example on elasticsearch output (#28288) * packetbeat/protos/dns: don't render missing A and AAAA addresses from truncated records (#28297) * seccomp: allow clone3 syscall for x86 (#28117) clone3 is a linux syscall that is now used by glibc starting version 2.34. It is used when pthread_create() gets called. Current seccomp filters do not allow this syscall leading to crashes like runtime/cgo: pthread_create failed: Operation not permitted See elastic/apm-server#6238 for more details * Osquerybeat: Improve handling of osquery.autoload file, allow customizations (#28289) Previously the osquery.autoload file was overwritten every time on osquerybeat start and stamped with our extension. After the change we check the content of the file and do not overwrite it on each osquerybeat start. This allows the user to deploy their own extensions if their want and start osquery with that. * Osquerybeat: Runner and Fetcher unit tests (#28290) * Runner and Fetcher unit tests * Fix header formatting * Tweak test * Update go release version 1.17.1 (#27543) * format of conditional build tags has changed * matching of * in regexes was fixed, thus breaking some of our code: golang/go#46123 * iproute package was missing from the new Golang Docker image, thus, we had to add it for our tests * go.mod file contains separate require directive for transitive dependencies * Move labels and annotations under kubernetes.namespace. (#27917) * Move labels and annotations under kubernetes.namespace. * Remove GCP support from Functionbeat (#28253) * Fix build tags for Go 1.17 (#28338) * [Elastic Agent] Add ability to communicate with Kibana through service token (#28096) * Add ability to communicate with Kibana through service token. Add ability to pass service token to container subcommand. * Add changelog entry. * Fix go fmt. * Add username to ASA Security negotiation log (#26975) * Add username to ASA Security negotiation log I added the username user.name field to ASA Security negotiation log line. * adding support for both formats * adding changelog entry * updating geo fields in expected output files * reverse formatting * reverting to older version of file * reverting formatting again * regenrate golden files again * remove formatting, ready for review * fixing missing message due to no newline * fix dissect pattern to fit correctly Co-authored-by: Marius Iversen <marius.iversen@elastic.co> * x-pack/filebeat/module/cisco: loosen time parsing and add group and session type capture (#28325) * Redis: remove deprecated fields (#28246) * Redis: remove deprecated fields * Disable generator tests temporarily (#28362) * Windows/perfmon metricset - remove deprecated perfmon.counters configuration (#28282) * remove deprecated config * changelog * [Filebeat] - S3 Input - Add support for only iterating/accessing only… (#28252) * [Filebeat] - S3 Input - Add support for only iterating/accessing only specific folders or datapaths * Breaking change for 8.0, namespace_annotations replaced by namespace.annotations (#28230) * Breaking change for 8.0, namespace_annotations replaced by namespace.annotations * Take care of namespace being nil * [Heartbeat] Setuid to regular user / lower capabilities when possible (#27878) partial fix for #27648 , this PR: Detects if the user is running as root then: Checks to see if an environment variable BEAT_SETUID_AS (set in our Docker.tmpl) is present Attempts to Setuid , Setgid and Setgroups to that user / groups Invokes setcap to drop all privileges except NET_RAW+ep This PR also fixes the broken syscall filtering in heartbeat, some non-syscall strings were breaking that. With the changes here elastic-agent can still run as root, but the subprocesses can lower their privileges ASAP. This should also make it possible for heartbeat to safely run ICMP pings and synthetics. Synthetics must run as non-root, but ICMP requires NET_RAW. This lets us be consistent in our docs with the recommendation that elastic-agent run as root. * mage fmt Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com> Co-authored-by: Ugo Sangiorgi <ugo.sangiorgi@elastic.co> Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com> Co-authored-by: Arnaud Lefebvre <a.lefebvre@outlook.fr> Co-authored-by: Aleksandr Maus <aleksandr.maus@elastic.co> Co-authored-by: apmmachine <58790750+apmmachine@users.noreply.github.com> Co-authored-by: Michael Katsoulis <michaelkatsoulis88@gmail.com> Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> Co-authored-by: LaZyDK <dennisperto@gmail.com> Co-authored-by: Marius Iversen <marius.iversen@elastic.co> Co-authored-by: Andrea Spacca <andrea.spacca@elastic.co> Co-authored-by: Mariana Dima <mariana@elastic.co> Co-authored-by: Andrew Cholakian <andrew@andrewvc.com>

elastic#28252) * [Filebeat] - S3 Input - Add support for only iterating/accessing only specific folders or datapaths

* singleton sysinfo host to avoid frequently collecting host info * add Host object to Stats object * update changelog * set procStats.host to nil if any error calling sysinfo.Host() * Update aws-lambda-go library version to 1.13.3 (elastic#28236) * [cloud][docker] use the private docker namespace (elastic#28286) * [7.x] [DOCS] Update api_key example on elasticsearch output (elastic#28288) * packetbeat/protos/dns: don't render missing A and AAAA addresses from truncated records (elastic#28297) * seccomp: allow clone3 syscall for x86 (elastic#28117) clone3 is a linux syscall that is now used by glibc starting version 2.34. It is used when pthread_create() gets called. Current seccomp filters do not allow this syscall leading to crashes like runtime/cgo: pthread_create failed: Operation not permitted See elastic/apm-server#6238 for more details * Osquerybeat: Improve handling of osquery.autoload file, allow customizations (elastic#28289) Previously the osquery.autoload file was overwritten every time on osquerybeat start and stamped with our extension. After the change we check the content of the file and do not overwrite it on each osquerybeat start. This allows the user to deploy their own extensions if their want and start osquery with that. * Osquerybeat: Runner and Fetcher unit tests (elastic#28290) * Runner and Fetcher unit tests * Fix header formatting * Tweak test * Update go release version 1.17.1 (elastic#27543) * format of conditional build tags has changed * matching of * in regexes was fixed, thus breaking some of our code: golang/go#46123 * iproute package was missing from the new Golang Docker image, thus, we had to add it for our tests * go.mod file contains separate require directive for transitive dependencies * Move labels and annotations under kubernetes.namespace. (elastic#27917) * Move labels and annotations under kubernetes.namespace. * Remove GCP support from Functionbeat (elastic#28253) * Fix build tags for Go 1.17 (elastic#28338) * [Elastic Agent] Add ability to communicate with Kibana through service token (elastic#28096) * Add ability to communicate with Kibana through service token. Add ability to pass service token to container subcommand. * Add changelog entry. * Fix go fmt. * Add username to ASA Security negotiation log (elastic#26975) * Add username to ASA Security negotiation log I added the username user.name field to ASA Security negotiation log line. * adding support for both formats * adding changelog entry * updating geo fields in expected output files * reverse formatting * reverting to older version of file * reverting formatting again * regenrate golden files again * remove formatting, ready for review * fixing missing message due to no newline * fix dissect pattern to fit correctly Co-authored-by: Marius Iversen <marius.iversen@elastic.co> * x-pack/filebeat/module/cisco: loosen time parsing and add group and session type capture (elastic#28325) * Redis: remove deprecated fields (elastic#28246) * Redis: remove deprecated fields * Disable generator tests temporarily (elastic#28362) * Windows/perfmon metricset - remove deprecated perfmon.counters configuration (elastic#28282) * remove deprecated config * changelog * [Filebeat] - S3 Input - Add support for only iterating/accessing only… (elastic#28252) * [Filebeat] - S3 Input - Add support for only iterating/accessing only specific folders or datapaths * Breaking change for 8.0, namespace_annotations replaced by namespace.annotations (elastic#28230) * Breaking change for 8.0, namespace_annotations replaced by namespace.annotations * Take care of namespace being nil * [Heartbeat] Setuid to regular user / lower capabilities when possible (elastic#27878) partial fix for elastic#27648 , this PR: Detects if the user is running as root then: Checks to see if an environment variable BEAT_SETUID_AS (set in our Docker.tmpl) is present Attempts to Setuid , Setgid and Setgroups to that user / groups Invokes setcap to drop all privileges except NET_RAW+ep This PR also fixes the broken syscall filtering in heartbeat, some non-syscall strings were breaking that. With the changes here elastic-agent can still run as root, but the subprocesses can lower their privileges ASAP. This should also make it possible for heartbeat to safely run ICMP pings and synthetics. Synthetics must run as non-root, but ICMP requires NET_RAW. This lets us be consistent in our docs with the recommendation that elastic-agent run as root. * mage fmt Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com> Co-authored-by: Ugo Sangiorgi <ugo.sangiorgi@elastic.co> Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com> Co-authored-by: Arnaud Lefebvre <a.lefebvre@outlook.fr> Co-authored-by: Aleksandr Maus <aleksandr.maus@elastic.co> Co-authored-by: apmmachine <58790750+apmmachine@users.noreply.github.com> Co-authored-by: Michael Katsoulis <michaelkatsoulis88@gmail.com> Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> Co-authored-by: LaZyDK <dennisperto@gmail.com> Co-authored-by: Marius Iversen <marius.iversen@elastic.co> Co-authored-by: Andrea Spacca <andrea.spacca@elastic.co> Co-authored-by: Mariana Dima <mariana@elastic.co> Co-authored-by: Andrew Cholakian <andrew@andrewvc.com>

[Filebeat] - S3 Input - Add support for only iterating/accessing only…

111eb25

… specific folders or datapaths

aspacca requested a review from kaiyan-sheng October 5, 2021 13:17

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 5, 2021

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Oct 5, 2021

changelog

01ff21f

aspacca added the Team:Integrations Label for the Integrations team label Oct 5, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 5, 2021

aspacca added backport-v7.16.0 Automated backport with mergify needs_team Indicates that the issue/PR needs a Team:* label labels Oct 5, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 5, 2021

mergify bot removed the backport-skip Skip notification from the automated backport with mergify label Oct 5, 2021

make update

36db120

aspacca commented Oct 5, 2021

View reviewed changes

kaiyan-sheng reviewed Oct 5, 2021

View reviewed changes

cr fixes

b2fc016

Merge branch 'master' into Filebeat-S3-Input-Add-support-for-only-ite…

9dfa35c

…rating-accessing-only-specific-folders-or-datapaths

Andrea Spacca added 2 commits October 7, 2021 09:40

Merge branch 'master' into Filebeat-S3-Input-Add-support-for-only-ite…

22bd6ba

…rating-accessing-only-specific-folders-or-datapaths

Merge branch 'master' into Filebeat-S3-Input-Add-support-for-only-ite…

b9f0f00

…rating-accessing-only-specific-folders-or-datapaths

kaiyan-sheng approved these changes Oct 12, 2021

View reviewed changes

aspacca merged commit 764045e into elastic:master Oct 12, 2021

mergify bot mentioned this pull request Oct 12, 2021

[7.x](backport #28252) [Filebeat] - S3 Input - Add support for only iterating/accessing only… #28369

Merged

newly12 pushed a commit to newly12/beats that referenced this pull request Oct 13, 2021

[Filebeat] - S3 Input - Add support for only iterating/accessing only… (

f23aff7

elastic#28252) * [Filebeat] - S3 Input - Add support for only iterating/accessing only specific folders or datapaths

aspacca mentioned this pull request Oct 19, 2021

[Filebeat] AWS Input - Document bucket_list_prefix #28527

Merged

1 task

Icedroid pushed a commit to Icedroid/beats that referenced this pull request Nov 1, 2021

[Filebeat] - S3 Input - Add support for only iterating/accessing only… (

2698bbc

elastic#28252) * [Filebeat] - S3 Input - Add support for only iterating/accessing only specific folders or datapaths

kaiyan-sheng mentioned this pull request Aug 9, 2023

[AWS] Remove duplicated number_of_workers settings from the custom logs integration elastic/integrations#7319

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] - S3 Input - Add support for only iterating/accessing only… #28252

[Filebeat] - S3 Input - Add support for only iterating/accessing only… #28252

aspacca commented Oct 5, 2021 •

edited

Loading

mergify bot commented Oct 5, 2021

elasticmachine commented Oct 5, 2021

botelastic bot commented Oct 5, 2021

elasticmachine commented Oct 5, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

aspacca Oct 5, 2021

kaiyan-sheng Oct 5, 2021

aspacca Oct 5, 2021

mergify bot commented Oct 6, 2021

mergify bot commented Oct 7, 2021

kaiyan-sheng left a comment

		@@ -178,6 +184,10 @@ Use to vertically scale the input.

		Wait interval between completion of a list request to the S3 bucket and beginning of the next one. Default to be 120 seconds.

		`var.bucket_bucket_list_prefix`::

[Filebeat] - S3 Input - Add support for only iterating/accessing only… #28252

[Filebeat] - S3 Input - Add support for only iterating/accessing only… #28252

Conversation

aspacca commented Oct 5, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

mergify bot commented Oct 5, 2021

elasticmachine commented Oct 5, 2021

botelastic bot commented Oct 5, 2021

elasticmachine commented Oct 5, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

aspacca Oct 5, 2021

Choose a reason for hiding this comment

kaiyan-sheng Oct 5, 2021

Choose a reason for hiding this comment

aspacca Oct 5, 2021

Choose a reason for hiding this comment

mergify bot commented Oct 6, 2021

mergify bot commented Oct 7, 2021

kaiyan-sheng left a comment

Choose a reason for hiding this comment

aspacca commented Oct 5, 2021 •

edited

Loading

elasticmachine commented Oct 5, 2021 •

edited by jenkins-beats-ci bot

Loading