Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make S3 partSize as config option #273

Closed
vanyasvl opened this issue Sep 23, 2021 · 1 comment · Fixed by #288
Closed

Make S3 partSize as config option #273

vanyasvl opened this issue Sep 23, 2021 · 1 comment · Fixed by #288
Assignees
@Slach

Comments

@vanyasvl
Copy link

vanyasvl commented Sep 23, 2021 •
edited
Loading

Default partSize calculation partSize := cfg.General.MaxFileSize / 10000 isn't work good with some S3 compatible storages like Swift. Swift has a low performance with a small chunks and doesn't accept chunks more than 5Gb.

So it will be good to have S3 partSize in config.
@Slach
Copy link
Collaborator

Slach commented Sep 25, 2021

thanks a lot for reporting, it make sense, and could be implements

@Slach Slach self-assigned this Sep 25, 2021
@alschastny alschastny mentioned this issue Sep 30, 2021
Merged
@Slach Slach mentioned this issue Oct 24, 2021
Merged
Slach added a commit that referenced this issue Oct 24, 2021
@Slach
Merge pull request #288 from Altinity/master
8567fc6 
# v1.2.0

INCOMPATIBLE CHANGES
- REST API `/backup/status` now return only latest executed command with status and error message

IMPROVEMENTS
- Added REST API `/backup/list/local` and `/backup/list/remote` to allow list backup types separately
- Decreased background backup creation time via REST API `/backup/create`, during avoid list remote backups for update metrics value 
- Decreased backup creation time, during avoid scan whole `system.tables` when set `table` query string parameter or `--tables` cli parameter     
- Added `last` and `filter` query string parameters to REST API `/backup/actions`, to avoid pass to client long JSON documents
- Improved `FTP` remote storage parallel upload / download
- Added `FTP_CONCURRENCY` to allow, by default MAX_CPU / 2 
- Added `FTP_DEBUG` setting, to allow debug FTP commands
- Added `FTP` to CI/CD on any commit
- Added race condition check to CI/CD

BUG FIXES
- environment variable `LOG_LEVEL` now apply to `clickhouse-backup server` properly
- fix #280, incorrect prometheus metrics measurement for `/backup/create`, `/backup/upload`, `/backup/download`
- fix #273, return `S3_PART_SIZE` back, but calculates it smartly
- fix #252, now you can pass `last` and `filter` query string parameters
- fix #246, incorrect error messages when use `REMOTE_STORAGE=none`
- fix #283, properly handle error message from `FTP` server 
- fix #268, properly restore legacy backup for schema without database name
- fix #287
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Slach Slach

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Release 1.2.0
2 participants
@Slach @vanyasvl