-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Amazon Athena data source #634
Conversation
gruuya
commented
Feb 17, 2022
- Add another S3-focused data source, this time backed by Amazon Athena, internally employing our SQLAlchemy FDW
- Install missing libs in the engine Dockerfile (PyAthena, pandas)
- Fix CSV mount example, and make Snowflake plugin default to batch size of 10000 if none specified
build_commandline_help(credentials_schema) + "\n" + build_commandline_help(params_schema) | ||
) | ||
|
||
_icon_file = "snowflake.svg" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use something like https://symbols.getvecta.com/stencil_5/0_aws-athena.bb0d0ced14.svg?
splitgraph/ingestion/csv/__init__.py
Outdated
"s3_secret_key": "abcd", | ||
"connection": | ||
{ | ||
"connection_type": "s3", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
indentation
"aws_access_key_id": {"type": "string", "title": "AWS Access Key Id"}, | ||
"aws_secret_access_key": {"type": "string", "title": "AWS Secret Access Key"}, | ||
}, | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These should be required, e.g.:
} | |
"required": ["aws_access_key_id", "aws_secret_access_key"], | |
} |
aws_access_key_id = self.params["aws_access_key_id"] | ||
aws_secret_access_key = self.params["aws_secret_access_key"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this comes from credentials (not merged into params)
aws_access_key_id = self.params["aws_access_key_id"] | |
aws_secret_access_key = self.params["aws_secret_access_key"] | |
aws_access_key_id = self.credentials["aws_access_key_id"] | |
aws_secret_access_key = self.credentials["aws_secret_access_key"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Curiously, it actually works properly either way.