Add use_regex param in S3KeySensor #1172
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## main #1172 +/- ##
=======================================
Coverage 98.58% 98.59%
=======================================
Files 90 90
Lines 5387 5395 +8
=======================================
+ Hits 5311 5319 +8
Misses 76 76
☔ View full report in Codecov by Sentry. |
pankajastro
requested review from
phanikumv,
pankajkoti,
Lee-W,
sunank200 and
utkarsharma2
as code owners
Lee-W
approved these changes
Jun 7, 2023
infohash
reviewed
Jun 9, 2023
infohash
reviewed
Jun 9, 2023
pankajastro
force-pushed
the
add_use_regex
branch
from
cc1860b
to
1d704a1
Compare
Lee-W
approved these changes
Jun 12, 2023
pankajastro
force-pushed
the
add_use_regex
branch
from
1d704a1
to
b352432
Compare
for more information, see https://pre-commit.ci
infohash
reviewed
Jun 18, 2023
| """ | ||
| bucket_name, key = S3Hook.get_s3_bucket_key(bucket_val, key, "bucket_name", "bucket_key") | ||
| if wildcard_match: | ||
| keys = await self.get_file_metadata(client, bucket_name, key) | ||
| key_matches = [k for k in keys if fnmatch.fnmatch(k["Key"], key)] | ||
| if len(key_matches) == 0: | ||
| return False | ||
| elif use_regex: | ||
| keys = await self.get_file_metadata(client, bucket_name) |
There was a problem hiding this comment.
Third parameter key (optional) is missing. Was this intentional?
There was a problem hiding this comment.
yes, listing objects from s3 does not work well with regex search. We are using the key get_file_metadata for prefix search.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes: #1154