Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dbWriteTable s3.location #73

Closed
DyfanJones opened this issue Jan 30, 2020 · 1 comment
Closed

dbWriteTable s3.location #73

DyfanJones opened this issue Jan 30, 2020 · 1 comment
Assignees
Labels
bug

Comments

@DyfanJones
Copy link
Owner

@DyfanJones DyfanJones commented Jan 30, 2020

Appending

When appending to an already exiting table dbWriteTable will just utilize parameter s3.location for S3 location. However an issue will occur if backend data in s3 is situated in a different location i.e.
Backend s3 location: s3://path/to/bucket1/tbl_data/
s3.location parameter: s3://path/to/bucket2/tbl_data/

To over come this the existing s3 location should overwrite s3.location parameter when the parameter append is set to TRUE

Backend S3 structure

Currently data is sent to the s3 path:

s3:/path/to/bucket/{tbl}/

This is ok when user only has 1 database/schema for Athena. When a use has 2 databases i.e:

dbWriteTable(con, "schema1.tbl", tbl)
dbWriteTable(con, "schema12.tbl", tbl)

The back end will return: s3:/path/to/bucket/{tbl}/ for both cases. Ideally it should align to the following schema: s3:/path/to/bucket/{schema}/{tbl}/ and return:

s3:/path/to/bucket/schema1/tbl/
s3:/path/to/bucket/schema2/tbl/
@DyfanJones
Copy link
Owner Author

@DyfanJones DyfanJones commented Jan 31, 2020

PR #72 fixes issue

@DyfanJones DyfanJones closed this Jan 31, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.