list-remote is slow when the number of backups is high #318
Assignees
Milestone
Comments
filimonov
changed the title
|
We need to get metadata.json to show backup size "backup number is high" mean thousand backups in the same paths? |
Let's add some flag to get it without sizes. Or get the size only for the last backup. |
yep. Backups every 2 hours. 90 days retention. |
|
Need local disk cache (can't remote cache cause will broke file layout) for slow only on first time |
This was referenced Feb 2, 2022
Merged
sushraju
added a commit
to muxinc/clickhouse-backup
that referenced
this issue
Jun 27, 2022
* fix Altinity#311 * fix Altinity#312 * fix https://github.com/Altinity/clickhouse-backup/runs/4385266807 * fix wrong amd64 `libc` dependency * change default skip_tables pattern to exclude INFORMATION_SCHEMA database for clickhouse 21.11+ * actualize GET /backup/actions, and fix config.go `CLICKHOUSE_SKIP_TABLES` definition * add COS_DEBUG separate setting, wait details in Altinity#316 * try to resolve Altinity#317 * Allow using OIDC token for AWS credentials * update ReadMe.md add notes about INFORMATION_SCHEMA.* * fix Altinity#220, allow total_bytes as uint64 fetching fix allocations for `tableMetadataForDownload` fix getTableSizeFromParts behavior only for required tables fix Error handling on some suggested cases * fix Altinity#331, corner case when `Table` and `Database` have the same name. update clickhouse-go to 1.5.1 * fix Altinity#331 * add SFTP_DEBUG to try debug Altinity#335 * fix bug, recursuve=>recursive * BackUPList use 'recursive=true', and other codes do not change, hope this can pass CI * Force recursive equals true locally * Reset recursive flag to false * fix Altinity#111 * add inner Interface for COS * properly fix for recursive delimiter, fix Altinity#338 * Fix bug about metadata.json, we should check the file name first, instead of appending metadata.json arbitrary * add ability to restore schema ON CLUSTER, fix Altinity#145 * fix bug about clickhouse-backup list remote which shows no backups info, clickhouse-backup create_remote which will not delete the right backups * fix `Address: NULL pointer` when DROP TABLE ... ON CLUSTER, fix Altinity#145 * try to fix `TestServerAPI` https://github.com/Altinity/clickhouse-backup/runs/4727526265 * try to fix `TestServerAPI` https://github.com/Altinity/clickhouse-backup/runs/4727754542 * Give up using metaDataFilePath variable * fix bug * Add support encrypted disk (include s3 encrypted disks), fix [Altinity#260](Altinity#260) add 21.12 to test matrix fix FTP MkDirAll behavior fix `restore --rm` behavior for 20.12+ for tables which have dependent objects (like dictionary) * try to fix failed build https://github.com/Altinity/clickhouse-backup/runs/4749276032 * add S3 only disks check for 21.8+ * fix Altinity#304 * fix Altinity#309 * try return GCP_TESTS back * fix run GCP_TESTS * fix run GCP_TESTS, again * split build-artifacts and build-test-artifacts * try to fix https://github.com/Altinity/clickhouse-backup/runs/4757549891 * debug workflows/build.yaml * debug workflows/build.yaml * debug workflows/build.yaml * final download atrifacts for workflows/build.yaml * fix build docker https://github.com/Altinity/clickhouse-backup/runs/4758167628 * fix integration_tests https://github.com/AlexAkulov/clickhouse-backup/runs/4758357087 * Improve list remote speed via local metadata cache, fix Altinity#318 * try to fix https://github.com/Altinity/clickhouse-backup/runs/4763790332 * fix test after fail https://github.com/Altinity/clickhouse-backup/runs/4764141333 * fix concurrency MkDirAll for FTP remote storage, improve `invalid compression_format` error message * fix TestLongListRemote * Clean code, do not name variables so sloppily, names should be meaningful * Update clickhouse.go Change partitions => part * Not change Files filed in json file * Code should be placed in proper position * Update server.go * fix bug * Invoke SoftSelect should begin with ch. * fix error, clickhouse.common.TablePathEncode => common.TablePathEncode * refine code * try to commit * fix bug * Remove unused codes * Use NewReplacer * Add `CLICKHOUSE_IGNORE_NOT_EXISTS_ERROR_DURING_FREEZE`, fix Altinity#319 * fix test fail https://github.com/Altinity/clickhouse-backup/runs/4825973411?check_suite_focus=true * run only TestSkipNotExistsTable on Github actions * try to fix TestSkipNotExistsTable * try to fix TestSkipNotExistsTable * try to fix TestSkipNotExistsTable, for ClickHouse version v1.x * try to fix TestSkipNotExistsTable, for ClickHouse version v1.x * add microseconds to log, try to fix TestSkipNotExistsTable, for ClickHouse version v20.8 * add microseconds to log, try to fix TestSkipNotExistsTable, for ClickHouse version v20.8 * fix connectWithWait, some versions of clickhouse accept connections during process /entrypoint-initdb.d, need wait to continue * add TestProjections * rename dropAllDatabases to more mental and clear name * skip TestSkipNotExistsTable * Support specified partition backup (Altinity#356) * Support specify partition during backup create Authored-by: wangzhen <wangzhen@growingio.com> * fix PROJECTION restore Altinity#320 * fix TestProjection fail after https://github.com/Altinity/clickhouse-backup/actions/runs/1712868840 * switch to `altinity-qa-test` bucket in GCS test * update github.com/mholt/archiver/v3 and github.com/ClickHouse/clickhouse-go to latest version, remove old github.com/mholt/archiver usage * fix `How to convert MergeTree to ReplicatedMergeTree` instruction * fix `FTP` connection usage in MkDirAll * optimize ftp.go connection pool * Add `UPLOAD_BY_PART` config settings for improve upload/download concurrency fix Altinity#324 * try debug https://github.com/AlexAkulov/clickhouse-backup/runs/4920777422 * try debug https://github.com/AlexAkulov/clickhouse-backup/runs/4920777422 * fix vsFTPd 500 OOPS: vsf_sysutil_bind, maximum number of attempts to find a listening port exceeded, fix https://github.com/AlexAkulov/clickhouse-backup/runs/4921182982 * try to fix race condition in GCP https://github.com/AlexAkulov/clickhouse-backup/runs/4924432841 * update clickhouse-go to 1.5.3, properly handle `--schema` parameter for show local backup size after `download` * add `Database not exists` corner case for `IgnoreNotExistsErrorDuringFreeze` option * prepare release 1.3.0 - Add implementation `--diff-from-remote` for `upload` command and properly handle `required` on download command, fix Altinity#289 - properly `REMOTE_STORAGE=none` error handle, fix Altinity#375 - Add support for `--partitions` on create, upload, download, restore CLI commands and API endpoint fix Altinity#378, properly implementation of Altinity#356 - Add `print-config` cli command fix Altinity#366 - API Server optimization for speed of `last_backup_size_remote` metric calculation to make it async during REST API startup and after download/upload, fix Altinity#309 - Improve `list remote` speed via local metadata cache in `$TEMP/.clickhouse-backup.$REMOTE_STORAGE`, fix Altinity#318 - fix Altinity#375, properly `REMOTE_STORAGE=none` error handle - fix Altinity#379, will try to clean `shadow` if `create` fail during `moveShadow` - more precise calculation backup size during `upload`, for backups created with `--partitions`, fix bug after Altinity#356 - fix `restore --rm` behavior for 20.12+ for tables which have dependent objects (like dictionary) - fix concurrency by `FTP` creation directories during upload, reduce connection pool usage - properly handle `--schema` parameter for show local backup size after `download` - add ClickHouse 22.1 instead of 21.12 to test matrix * fix build https://github.com/Altinity/clickhouse-backup/runs/5033550335 * Add `API_ALLOW_PARALLEL` to support multiple parallel execution calls for, WARNING, control command names don't try to execute multiple same commands and be careful, it could allocate much memory during upload / download, fix Altinity#332 * apt-get update too slow today on github ;( * fix TestLongListRemote * fix Altinity#340, properly handle errors on S3 during Walk() and delete old backup * Add TestFlows tests to GH workflow (Altinity#5) * add actions tests * Update test.yaml * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * updated * added config.yml * added config.yml * update * updated files * added tests for views * updated tests * updated * fixed snapshots * updated tests in response to @Slach * upload new stuff * rerun * fix * fix * remove file * added requirements * fix fails * ReRun actions * Moved credentials * added secrets * ReRun actions * Edited test.yaml * Edited test.yaml * ReRun actions * removed TE flag * update * update * update * fix type * update * try to reanimate ugly github actions and ugly python tests * try to reanimate ugly config_rbac.py * fix Altinity#300 fix WINDOW VIEW restore fix restore for different compression_format than backup created fix most of xfail in regression.py merge test.yaml and build.yaml in github actions Try to add experimental support for backup `MaterializedMySQL` and `MaterializedPostgeSQL` tables, restore MySQL tables not impossible now without replace `table_name.json` to `Engine=MergeTree`, PostgreSQL not supported now, see ClickHouse/ClickHouse#32902 * return format back * fix build.yaml after https://github.com/Altinity/clickhouse-backup/actions/runs/1800312966 * fix build.yaml after https://github.com/Altinity/clickhouse-backup/actions/runs/1800312966 * build fixes after https://github.com/Altinity/clickhouse-backup/runs/5079597138 * build fixes after https://github.com/Altinity/clickhouse-backup/runs/5079630559 * build fixes after https://github.com/Altinity/clickhouse-backup/runs/5079669062 * fix tfs report * fix upload artifact for tfs report * fix upload artifact for clickhouse logs, remove unused BackupOptions * suuka * fix upload `clickhouse-logs` artifacts and tfs `report.html` * fix upload `clickhouse-logs` artifacts * fix upload `clickhouse-logs` artifacts, fix tfs reports * fix tfs reports * change retention to allow upload-artifacts work * fix ChangeLog.md * skip gcs and aws remote storage tests if secrets not set * remove short output * increase timeout to allow download images during pull * remove upload `tesflows-clickhouse-logs` artifacts to avoid 500 error * fix upload_release_assets action for properly support arm64 * switch to mantainable `softprops/action-gh-release` * fix Unexpected input(s) 'release_name' * move internal, config, util into `pkg` refactoring * updated test requirements * refactoring `filesystemhelper.Chown` remove unnecessary getter/setter, try to reproduce access denied for Altinity#388 (comment) * resolve Altinity#390, for 1.2.3 hotfix branch * backport 1.3.x Dockerfile and Makefile to allow 1.2.3 docker ARM support * fix Altinity#387 (comment), improve documentation related to memory and CPU usage * fix Altinity#388, improve restore ON CLUSTER for VIEW with TO clause * fix Altinity#388, improve restore ATTACH ... VIEW ... ON CLUSTER, GCS golang sdk updated to latest * fix Altinity#385, properly handle multiple incremental backup sequences + `BACKUPS_TO_KEEP_REMOTE` * fix Altinity#392, correct download for recursive sequence of diff backups when `DOWNLOAD_BY_PART` true fix integration_test.go, add RUN_ADVANCED_TESTS environment, fix minio_nodelete.sh * try to reduce upload artifact jobs, look actions/upload-artifact#171 and https://github.com/Altinity/clickhouse-backup/runs/5229552384?check_suite_focus=true * try to docker-compose up from first time https://github.com/AlexAkulov/clickhouse-backup/runs/5231510719?check_suite_focus=true * disable telemetry for GCS related to googleapis/google-cloud-go#5664 * update aws-sdk-go and GCS storage SDK * DROP DATABASE didn't clean S3 files, DROP TABLE clean! * - fix Altinity#406, properly handle `path` for S3, GCS for case when it begin from "/" * fix getTablesWithSkip * fix Altinity#409 * cherry pick release.yaml from 1.3.x to 1.2.x * fix Altinity#409, for 1.3.x avoid delete partially uploaded backups via `backups_keep_remote` option * Updated requirements file * fix Altinity#409, for 1.3.x avoid delete partially uploaded backups via `backups_keep_remote` option * fix testflows test * fix testflows test * restore tests after update minio * Fix incorrect in progress check on the example of Kubernetes CronJob * removeOldBackup error log from fatal to warning, to avoid race-condition deletion during multi-shard backup * switch to golang 1.18 Signed-off-by: Slach <bloodjazman@gmail.com> * add 22.3 to test matrix, fix Altinity#422, avoid cache broken (partially uploaded) remote backup metadata. * add 22.3 to test matrix * fix Altinity#404, switch to 22.3 by default Signed-off-by: Slach <bloodjazman@gmail.com> * fix Altinity#404, update to archiver/v4, properly support context during upload / download and correct error handler, reduce `SELECT * system.disks` calls Signed-off-by: Slach <bloodjazman@gmail.com> * cleanup ChangeLog.md, finally before 1.3.2 release Signed-off-by: Slach <bloodjazman@gmail.com> * continue fix Altinity#404 Signed-off-by: Slach <bloodjazman@gmail.com> * continue fix Altinity#404, properly calculate max_parts_count Signed-off-by: Slach <bloodjazman@gmail.com> * continue fix Altinity#404, properly calculate max_parts_count Signed-off-by: Slach <bloodjazman@gmail.com> * add multithreading GZIP implementation Signed-off-by: Slach <bloodjazman@gmail.com> * add multithreading GZIP implementation Signed-off-by: Slach <bloodjazman@gmail.com> * add multithreading GZIP implementation Signed-off-by: Slach <bloodjazman@gmail.com> * Updated Testflows README.md * add `S3_ALLOW_MULTIPART_DOWNLOAD` to config, to improve download speed, fix Altinity#431 Signed-off-by: Slach <bloodjazman@gmail.com> * fix snapshot after change default config Signed-off-by: Slach <bloodjazman@gmail.com> * fix testflows healthcheck for slow internet connection during `clickhouse_backup` start Signed-off-by: Slach <bloodjazman@gmail.com> * fix snapshot after change defaultConfig Signed-off-by: Slach <bloodjazman@gmail.com> * - add support backup/restore user defined functions https://clickhouse.com/docs/en/sql-reference/statements/create/function, fix Altinity#420 Signed-off-by: Slach <bloodjazman@gmail.com> * Updated README.md in testflows tests * remove unnecessary SQL query for calculateMaxSize, refactoring test to allow restoreRBAC with restart on 21.8 (strange bug, clickhouse stuck after try to run too much distributed DDL queries from ZK), update LastBackupSize metric during API call /list/remote, add healthcheck to docker-compose in integration tests Signed-off-by: Slach <bloodjazman@gmail.com> * try to fix GitHub actions Signed-off-by: Slach <bloodjazman@gmail.com> * try to fix GitHub actions, WTF, why testflows failed? Signed-off-by: Slach <bloodjazman@gmail.com> * add `clickhouse_backup_number_backups_remote`, `clickhouse_backup_number_backups_local`, `clickhouse_backup_number_backups_remote_expected`,`clickhouse_backup_number_backups_local_expected` prometheus metric, fix Altinity#437 Signed-off-by: Slach <bloodjazman@gmail.com> * add ability to apply `system.macros` values to `path` field in all types of `remote_storage`, fix Altinity#438 Signed-off-by: Slach <bloodjazman@gmail.com> * use all disks for upload and download for mutli-disk volumes in parallel when `upload_by_part: true` fix Altinity#400 Signed-off-by: Slach <bloodjazman@gmail.com> * fix wrong warning for .gz, .bz2, .br archive extensions during download, fix Altinity#441 Signed-off-by: Slach <bloodjazman@gmail.com> * fix Altinity#441, again ;( Signed-off-by: Slach <bloodjazman@gmail.com> * try to improve strange parts long tail during test Signed-off-by: Slach <bloodjazman@gmail.com> * update actions/download-artifact@v3 and actions/upload-artifact@v2, after actions fail Signed-off-by: Slach <bloodjazman@gmail.com> * downgrade actions/upload-artifact@v2.2.4, actions/upload-artifact#270, after actions fail https://github.com/AlexAkulov/clickhouse-backup/runs/6481819375 Signed-off-by: Slach <bloodjazman@gmail.com> * fix upload data go routines wait, expect improve upload speed the same as 1.3.2 Signed-off-by: Slach <bloodjazman@gmail.com> * prepare 1.4.1 Signed-off-by: Slach <bloodjazman@gmail.com> * Fix typo in Example.md * Set default value for max_parts_count in Azure config * fix `--partitions` parameter parsing, fix Altinity#425 Signed-off-by: Slach <bloodjazman@gmail.com> * remove unnecessary logs, fix release.yaml to mark properly tag in GitHub release Signed-off-by: Slach <bloodjazman@gmail.com> * add `API_INTEGRATION_TABLES_HOST` option to allow use DNS name in integration tables system.backup_list, system.backup_actions Signed-off-by: Slach <bloodjazman@gmail.com> * add `API_INTEGRATION_TABLES_HOST` fix for tesflows fails Signed-off-by: Slach <bloodjazman@gmail.com> * fix `upload_by_part: false` max file size calculation, fix Altinity#454 * upgrade actions/upload-artifact@v3, actions/upload-artifact#270, after actions fail https://github.com/Altinity/clickhouse-backup/runs/6962550621 * [clickhouse-backup] fixes on top of upstream * upstream versions Co-authored-by: Slach <bloodjazman@gmail.com> Co-authored-by: Vilmos Nebehaj <vilmos@sprig.com> Co-authored-by: Eugene Klimov <eklimov@altinity.com> Co-authored-by: root <root@SLACH-MINI.localdomain> Co-authored-by: wangzhen <wangzhen@growingio.com> Co-authored-by: W <wangzhenaaa7@gmail.com> Co-authored-by: Andrey Zvonov <32552679+zvonand@users.noreply.github.com> Co-authored-by: zvonand <azvonov@altinity.com> Co-authored-by: benbiti <wangshouben@hotmail.com> Co-authored-by: Vitaliis <vsviderskyi@altinity.com> Co-authored-by: Toan Nguyen <hgiasac@gmail.com> Co-authored-by: Guido Iaquinti <guido@posthog.com> Co-authored-by: ricoberger <mail@ricoberger.de>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
On GCP it list all objects recursively (?) (probably, need to be confirmed)
Even on aws it do HEAD & GET metadata object for every backup (confirmed by logs).
The text was updated successfully, but these errors were encountered: