[FEATURE] Feature request: I want to filter dynamoDB tables into Athena as configurable list #606 #607
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
7. dynamo_connector_filter (Optional) Can be used to fetch selected tables from DynamoDB. By default the connector will attempt to get all DynamoDB tables while connect via Athena. . (e.g. * or tbl1,tbl2,tbl3) Note : “dynamo_connector_filter” is not available with existing connectors code , I have updated code to meet requirements, in order to use dynaodb table filter.
Updated document added dynamo db filter information in .7
vaquarkhan
changed the title
Added missing import
henrymai
reviewed
May 24, 2022
|
|
||
| private Set<TableName> filterTables(Set<TableName> combinedTables) | ||
| { | ||
| String dynamoDBFilter = "*"; |
There was a problem hiding this comment.
It looks like dynamoDBFilter is immediately overwritten after being initialized to "*" (on line 242).
This line should just be replaced with:
String dynamoDBFilter = System.getenv("dynamo_connector_filter");
And then remove line 242
| } | ||
|
|
||
| if (null != dynamoDBFilter) { | ||
| flag = dynamoDBFilter.toUpperCase().contains("*"); |
There was a problem hiding this comment.
What if the filter is set to: sometable1,sometable2*,sometable3 ?
I think either we should throw an exception here or implement actual wildcard or regex matching if you want to support the case where a * is contained in the filter rather than equals the filter.
|
closing this as it has been stale for long time. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available: [FEATURE] Feature request: I want to filter tables to list into DynamoDB #606
Description of changes:
User has multiple teams and tables also contain KMS key few are configured via Lake formation , now I can see following issues.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.