Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign updbWriteTable s3.location #73
Comments
|
PR #72 fixes issue |
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign up|
PR #72 fixes issue |
Appending
When appending to an already exiting table
dbWriteTablewill just utilize parameter s3.location for S3 location. However an issue will occur if backend data in s3 is situated in a different location i.e.Backend s3 location:
s3://path/to/bucket1/tbl_data/s3.location parameter:
s3://path/to/bucket2/tbl_data/To over come this the existing s3 location should overwrite
s3.locationparameter when the parameterappendis set toTRUEBackend S3 structure
Currently data is sent to the s3 path:
s3:/path/to/bucket/{tbl}/This is ok when user only has 1 database/schema for Athena. When a use has 2 databases i.e:
The back end will return:
s3:/path/to/bucket/{tbl}/for both cases. Ideally it should align to the following schema:s3:/path/to/bucket/{schema}/{tbl}/and return: