You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When appending to an already exiting table dbWriteTable will just utilize parameter s3.location for S3 location. However an issue will occur if backend data in s3 is situated in a different location i.e.
Backend s3 location: s3://path/to/bucket1/tbl_data/
s3.location parameter: s3://path/to/bucket2/tbl_data/
To over come this the existing s3 location should overwrite s3.location parameter when the parameter append is set to TRUE
Backend S3 structure
Currently data is sent to the s3 path:
s3:/path/to/bucket/{tbl}/
This is ok when user only has 1 database/schema for Athena. When a use has 2 databases i.e:
The back end will return: s3:/path/to/bucket/{tbl}/ for both cases. Ideally it should align to the following schema: s3:/path/to/bucket/{schema}/{tbl}/ and return:
Appending
When appending to an already exiting table
dbWriteTable
will just utilize parameter s3.location for S3 location. However an issue will occur if backend data in s3 is situated in a different location i.e.Backend s3 location:
s3://path/to/bucket1/tbl_data/
s3.location parameter:
s3://path/to/bucket2/tbl_data/
To over come this the existing s3 location should overwrite
s3.location
parameter when the parameterappend
is set toTRUE
Backend S3 structure
Currently data is sent to the s3 path:
s3:/path/to/bucket/{tbl}/
This is ok when user only has 1 database/schema for Athena. When a use has 2 databases i.e:
The back end will return:
s3:/path/to/bucket/{tbl}/
for both cases. Ideally it should align to the following schema:s3:/path/to/bucket/{schema}/{tbl}/
and return:The text was updated successfully, but these errors were encountered: