Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add s3 download by range #1083

Merged
merged 17 commits into from Sep 16, 2021
Merged

Conversation

proggga
Copy link
Contributor

@proggga proggga commented Sep 3, 2021

Debug s3 storage download by ranges

@proggga proggga requested a review from a team as a code owner September 3, 2021 18:53
@proggga proggga force-pushed the make_s3_download_by_range branch 13 times, most recently from a21a01c to a3ec48d Compare September 8, 2021 14:08
Copy link
Member

@usernamedt usernamedt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, can you please add an integration test with S3_RANGE_BATCH_ENABLED turned on?

internal/databases/postgres/delta_map_downloader.go Outdated Show resolved Hide resolved
pkg/storages/s3/folder.go Show resolved Hide resolved
pkg/storages/s3/folder.go Outdated Show resolved Hide resolved
pkg/storages/s3/folder.go Outdated Show resolved Hide resolved
pkg/storages/s3/folder.go Outdated Show resolved Hide resolved
@proggga proggga changed the title WIP Add s3 download by range Add s3 download by range Sep 13, 2021
pkg/storages/s3/s3reader.go Outdated Show resolved Hide resolved
pkg/storages/s3/s3reader.go Show resolved Hide resolved
@usernamedt usernamedt merged commit fd4fd2c into wal-g:master Sep 16, 2021
@proggga proggga deleted the make_s3_download_by_range branch September 20, 2021 14:56
perekalov pushed a commit to perekalov/wal-g that referenced this pull request Jan 12, 2022
* Add s3 download by range

* Add config add devel

* Full remake

* Small fixes

* Retrigger

* Fix exp backoff for s3reader

* Simplified debug log for s3Reader

* Extract s3reader

* go fmt changes

* Removed debug line

* Reuse last Body

* Fix config, add test for mysql with ranges

* Small rename

* Add comment, fix test, fix error msg

* Rename bucket

* Add devel, fix prefix for mysql test

* Fix docs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants