Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dbWriteTable s3.location #73

Closed
DyfanJones opened this issue Jan 30, 2020 · 1 comment
Closed

dbWriteTable s3.location #73

DyfanJones opened this issue Jan 30, 2020 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@DyfanJones
Copy link
Owner

Appending

When appending to an already exiting table dbWriteTable will just utilize parameter s3.location for S3 location. However an issue will occur if backend data in s3 is situated in a different location i.e.
Backend s3 location: s3://path/to/bucket1/tbl_data/
s3.location parameter: s3://path/to/bucket2/tbl_data/

To over come this the existing s3 location should overwrite s3.location parameter when the parameter append is set to TRUE

Backend S3 structure

Currently data is sent to the s3 path:

s3:/path/to/bucket/{tbl}/

This is ok when user only has 1 database/schema for Athena. When a use has 2 databases i.e:

dbWriteTable(con, "schema1.tbl", tbl)
dbWriteTable(con, "schema12.tbl", tbl)

The back end will return: s3:/path/to/bucket/{tbl}/ for both cases. Ideally it should align to the following schema: s3:/path/to/bucket/{schema}/{tbl}/ and return:

s3:/path/to/bucket/schema1/tbl/
s3:/path/to/bucket/schema2/tbl/
@DyfanJones DyfanJones added the bug Something isn't working label Jan 30, 2020
@DyfanJones DyfanJones self-assigned this Jan 30, 2020
DyfanJones pushed a commit that referenced this issue Jan 30, 2020
@DyfanJones
Copy link
Owner Author

PR #72 fixes issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant