Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

replace rusoto_s3: part 1 #244

Merged
merged 1 commit into from
Jan 18, 2023
Merged

Conversation

jychen7
Copy link
Collaborator

@jychen7 jychen7 commented Jan 18, 2023

Why

step 2 of #240: replace rusoto_s3 (in use_memory_table: true path) with S3 client from object_store (s3://)

rusoto_s3 is used in columnq::io::s3 and columnq::io::s3 is used in two places:

  • columnq::partitions_from_table_source
  • columnq::table::delta

we would like to remove rusoto_s3 and switch to object_store, so can unify S3/GCS/Azure in future

How

this PR (part 1) creates columnq::io::object_store and migrate columnq::partitions_from_table_source.
next PR (part 2) will focus on migrating columnq::table::delta and delete columnq::io::s3 completely

Test

Integration tests are improved to include single file and directory.

@houqp houqp merged commit 9db6b41 into roapi:main Jan 18, 2023
@jychen7 jychen7 deleted the 240-replace-rusoto-part-1 branch January 18, 2023 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants