Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: allow setting chunk_size #874

Merged
merged 1 commit into from
Mar 27, 2024
Merged

feat: allow setting chunk_size #874

merged 1 commit into from
Mar 27, 2024

Conversation

dermasmid
Copy link
Contributor

while running a backup to gcs it seemed to be very slow to upload, setting the chunk_size might make it faster

@Slach
Copy link
Collaborator

Slach commented Mar 25, 2024

run following command to fix tests

rm ./test/testflows/clickhouse_backup/tests/snapshots/cli.py.cli.snapshot
touch ./test/testflows/.env
RUN_TESTS="/clickhouse backup/cli/default config" ./test/testflows/run.sh
ls -la ./test/testflows/clickhouse_backup/tests/snapshots/cli.py.cli.snapshot
git add ./test/testflows/

@Slach
Copy link
Collaborator

Slach commented Mar 25, 2024

thanks for contibuting

do you have statistict about speedup improvement for your change?
and how required memory increased during execution?

@dermasmid
Copy link
Contributor Author

i just added the option to configure the chunk_size, so i can test if it will speed up in my case. i didnt change the default which is 16Mb

@dermasmid
Copy link
Contributor Author

https://cloud.google.com/blog/topics/developers-practitioners/optimize-cloud-storage-upload-performance-client-libraries#:~:text=For%20resumable%20uploads,the%20whole%20file.

@dermasmid
Copy link
Contributor Author

the command you gave me resulted in:

+ cp -fl ./clickhouse-backup/clickhouse-backup-race ./clickhouse-backup/clickhouse-backup-race-docker
Traceback (most recent call last):
  File "/Users/cheskeltwersky/Documents/clickhouse-backup/test/testflows/clickhouse_backup/regression.py", line 10, in <module>
    from testflows.core import *
ModuleNotFoundError: No module named 'testflows'
warning: no applicable files found in input directories
ls: ./test/testflows/clickhouse_backup/tests/snapshots/cli.py.cli.snapshot: No such file or directory

@Slach
Copy link
Collaborator

Slach commented Mar 25, 2024

@dermasmid
my fault
try

apt-get install python-venv
python -m venv ./venv
./venv/bin/pip3 install -r ./test/testflows/requirements.txt
./venv/bin/activate
touch ./test/testflows/.env
RUN_TESTS="/clickhouse backup/cli/default config" ./test/testflows/run.sh

@dermasmid
Copy link
Contributor Author

got this now:

           9m 1s              [clickhouse1] 
               9m 1s              [clickhouse1] 
               9m 1s              [clickhouse1] Error response from daemon: No such container: WARN[0000]
               9m 1s              [clickhouse1] /bin/bash: line 1: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 2: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 3: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 4: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 5: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 6: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 7: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 8: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 9: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 10: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 11: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 12: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 13: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 14: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 15: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 16: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 17: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 18: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 19: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 20: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 21: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 22: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 23: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 24: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 25: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 26: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 27: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 28: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 29: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 30: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 31: WARN[0000]: command not found
               9m 1s              [clickhouse1] /bin/bash: line 32: dc84edf827959bdc1b304ebdfff54d969b9df93ff0572d991e3d872994903cf0: command not found
              9m 31s              [clickhouse1] 
              9m 31s              [clickhouse1] 
              9m 31s              [clickhouse1] Error response from daemon: No such container: WARN[0000]
              9m 31s              [clickhouse1] /bin/bash: line 1: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 2: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 3: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 4: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 5: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 6: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 7: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 8: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 9: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 10: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 11: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 12: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 13: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 14: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 15: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 16: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 17: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 18: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 19: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 20: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 21: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 22: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 23: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 24: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 25: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 26: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 27: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 28: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 29: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 30: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 31: WARN[0000]: command not found
              9m 31s              [clickhouse1] /bin/bash: line 32: dc84edf827959bdc1b304ebdfff54d969b9df93ff0572d991e3d872994903cf0: command not found
              10m 1s              [clickhouse1] 
              10m 1s              [clickhouse1] 
              10m 1s         ⟥    Exception: Traceback (most recent call last):
                                      File "/opt/homebrew/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 223, in read
                                        d = self.queue.get(timeout=timeleft)
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                      File "/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/queue.py", line 179, in get
                                        raise Empty
                                    _queue.Empty

                                    During handling of the above exception, another exception occurred:

                                    Traceback (most recent call last):
                                      File "/opt/homebrew/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 187, in expect
                                        data = self.read(timeout=min(timeleft, 0.1), raise_exception=True)
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                      File "/opt/homebrew/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 235, in read
                                        raise TimeoutError(timeout)
                                    testflows.uexpect.uexpect.TimeoutError: Timeout 0.071s

                                    During handling of the above exception, another exception occurred:

                                    Traceback (most recent call last):
                                      File "/Users/cheskeltwersky/Documents/clickhouse-backup/test/testflows/clickhouse_backup/../helpers/cluster.py", line 578, in bash
                                        self.shells[shell_id]("echo 1")
                                      File "/opt/homebrew/lib/python3.12/site-packages/testflows/connect/shell.py", line 411, in __call__
                                        self.open(timeout)
                                      File "/opt/homebrew/lib/python3.12/site-packages/testflows/connect/shell.py", line 305, in open
                                        self.child.expect(self.prompt)
                                      File "/opt/homebrew/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 205, in expect
                                        raise exception
                                    testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 30.000s for '[#\\$] ' buffer 'Error response from daemon: No such container: \x1b[33mWARN\x1b[0m[0000]\r\n/bin/bash: line 1: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 2: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 3: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 4: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 5: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 6: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 7: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 8: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 9: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 10: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 11: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 12: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 13: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 14: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 15: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 16: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 17: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 18: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 19: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 20: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 21: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 22: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 23: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 24: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 25: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 26: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 27: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 28: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 29: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 30: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 31: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 32: dc84edf827959bdc1b304ebdfff54d969b9df93ff0572d991e3d872994903cf0: command not found\r\n' or '45,72,72,6f,72,20,72,65,73,70,6f,6e,73,65,20,66,72,6f,6d,20,64,61,65,6d,6f,6e,3a,20,4e,6f,20,73,75,63,68,20,63,6f,6e,74,61,69,6e,65,72,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,33,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,34,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,35,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,36,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,37,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,38,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,39,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,30,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,31,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,32,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,33,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,34,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,35,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,36,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,37,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,38,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,39,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,30,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,31,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,32,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,33,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,34,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,35,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,36,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,37,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,38,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,39,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,33,30,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,33,31,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,33,32,3a,20,64,63,38,34,65,64,66,38,32,37,39,35,39,62,64,63,31,62,33,30,34,65,62,64,66,66,66,35,34,64,39,36,39,62,39,64,66,39,33,66,66,30,35,37,32,64,39,39,31,65,33,64,38,37,32,39,39,34,39,30,33,63,66,30,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a'

                                    During handling of the above exception, another exception occurred:

                                    Traceback (most recent call last):
                                      File "/Users/cheskeltwersky/Documents/clickhouse-backup/test/testflows/clickhouse_backup/regression.py", line 79, in <module>
                                        regression()
                                      File "/opt/homebrew/lib/python3.12/site-packages/testflows/_core/test.py", line 3273, in __call__
                                        return self.__run__(**args)
                                               ^^^^^^^^^^^^^^^^^^^^
                                      File "/opt/homebrew/lib/python3.12/site-packages/testflows/_core/test.py", line 3371, in __run__
                                        return self.type(**kwargs, test=self)(**args)
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                      File "/opt/homebrew/lib/python3.12/site-packages/testflows/_core/test.py", line 2061, in __call__
                                        return callable()
                                               ^^^^^^^^^^
                                      File "/opt/homebrew/lib/python3.12/site-packages/testflows/_core/test.py", line 1959, in callable
                                        return _test_wrapper()
                                               ^^^^^^^^^^^^^^^
                                      File "/opt/homebrew/lib/python3.12/site-packages/testflows/_core/test.py", line 1943, in _test_wrapper
                                        r = test(**self.kwargs["args"])
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                      File "/opt/homebrew/lib/python3.12/site-packages/testflows/_core/test.py", line 3273, in __call__
                                        return self.__run__(**args)
                                               ^^^^^^^^^^^^^^^^^^^^
                                      File "/opt/homebrew/lib/python3.12/site-packages/testflows/_core/test.py", line 3373, in __run__
                                        return run(test)
                                               ^^^^^^^^^
                                      File "/opt/homebrew/lib/python3.12/site-packages/testflows/_core/test.py", line 3295, in run
                                        return process_func_result(self.func(test, **args))
                                                                   ^^^^^^^^^^^^^^^^^^^^^^^
                                      File "/Users/cheskeltwersky/Documents/clickhouse-backup/test/testflows/clickhouse_backup/regression.py", line 48, in regression
                                        with Cluster(local, nodes=nodes) as cluster:
                                      File "/Users/cheskeltwersky/Documents/clickhouse-backup/test/testflows/clickhouse_backup/../helpers/cluster.py", line 616, in __enter__
                                        self.up()
                                      File "/Users/cheskeltwersky/Documents/clickhouse-backup/test/testflows/clickhouse_backup/../helpers/cluster.py", line 736, in up
                                        self.node(node_name).wait_healthy()
                                      File "/Users/cheskeltwersky/Documents/clickhouse-backup/test/testflows/clickhouse_backup/../helpers/cluster.py", line 145, in wait_healthy
                                        if self.query("select version()", no_checks=True, timeout=300, steps=False).exitcode == 0:
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                      File "/Users/cheskeltwersky/Documents/clickhouse-backup/test/testflows/clickhouse_backup/../helpers/cluster.py", line 375, in query
                                        r = self.cluster.bash(self.name)(command, *nargs, **kwargs)
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                      File "/Users/cheskeltwersky/Documents/clickhouse-backup/test/testflows/clickhouse_backup/../helpers/cluster.py", line 585, in bash
                                        raise RuntimeError(f"failed to open bash to node {node}")
                                    RuntimeError: failed to open bash to node clickhouse1
              10m 1s         ⟥⟤ Error waiting until container clickhouse1 is healthy, /clickhouse backup/docker-compose cluster/wait all nodes report healthy/waiting until container clickhouse1 is healthy, RuntimeError
              10m 1s       ⟥⟤ Error wait all nodes report healthy, /clickhouse backup/docker-compose cluster/wait all nodes report healthy, RuntimeError
              14m 0s     ⟥⟤ Error docker-compose cluster, /clickhouse backup/docker-compose cluster, RuntimeError
              14m 0s   ⟥⟤ Error clickhouse backup, /clickhouse backup, RuntimeError
                         Traceback (most recent call last):
                           File "/opt/homebrew/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 223, in read
                             d = self.queue.get(timeout=timeleft)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                           File "/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/queue.py", line 179, in get
                             raise Empty
                         _queue.Empty

                         During handling of the above exception, another exception occurred:

                         Traceback (most recent call last):
                           File "/opt/homebrew/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 187, in expect
                             data = self.read(timeout=min(timeleft, 0.1), raise_exception=True)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                           File "/opt/homebrew/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 235, in read
                             raise TimeoutError(timeout)
                         testflows.uexpect.uexpect.TimeoutError: Timeout 0.071s

                         During handling of the above exception, another exception occurred:

                         Traceback (most recent call last):
                           File "/Users/cheskeltwersky/Documents/clickhouse-backup/test/testflows/clickhouse_backup/../helpers/cluster.py", line 578, in bash
                             self.shells[shell_id]("echo 1")
                           File "/opt/homebrew/lib/python3.12/site-packages/testflows/connect/shell.py", line 411, in __call__
                             self.open(timeout)
                           File "/opt/homebrew/lib/python3.12/site-packages/testflows/connect/shell.py", line 305, in open
                             self.child.expect(self.prompt)
                           File "/opt/homebrew/lib/python3.12/site-packages/testflows/uexpect/uexpect.py", line 205, in expect
                             raise exception
                         testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 30.000s for '[#\\$] ' buffer 'Error response from daemon: No such container: \x1b[33mWARN\x1b[0m[0000]\r\n/bin/bash: line 1: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 2: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 3: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 4: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 5: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 6: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 7: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 8: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 9: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 10: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 11: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 12: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 13: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 14: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 15: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 16: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 17: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 18: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 19: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 20: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 21: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 22: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 23: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 24: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 25: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 26: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 27: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 28: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 29: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 30: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 31: \x1b[33mWARN\x1b[0m[0000]: command not found\r\n/bin/bash: line 32: dc84edf827959bdc1b304ebdfff54d969b9df93ff0572d991e3d872994903cf0: command not found\r\n' or '45,72,72,6f,72,20,72,65,73,70,6f,6e,73,65,20,66,72,6f,6d,20,64,61,65,6d,6f,6e,3a,20,4e,6f,20,73,75,63,68,20,63,6f,6e,74,61,69,6e,65,72,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,33,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,34,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,35,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,36,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,37,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,38,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,39,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,30,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,31,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,32,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,33,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,34,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,35,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,36,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,37,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,38,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,31,39,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,30,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,31,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,32,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,33,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,34,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,35,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,36,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,37,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,38,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,32,39,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,33,30,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,33,31,3a,20,1b,5b,33,33,6d,57,41,52,4e,1b,5b,30,6d,5b,30,30,30,30,5d,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a,2f,62,69,6e,2f,62,61,73,68,3a,20,6c,69,6e,65,20,33,32,3a,20,64,63,38,34,65,64,66,38,32,37,39,35,39,62,64,63,31,62,33,30,34,65,62,64,66,66,66,35,34,64,39,36,39,62,39,64,66,39,33,66,66,30,35,37,32,64,39,39,31,65,33,64,38,37,32,39,39,34,39,30,33,63,66,30,3a,20,63,6f,6d,6d,61,6e,64,20,6e,6f,74,20,66,6f,75,6e,64,d,a'

                         During handling of the above exception, another exception occurred:

                         Traceback (most recent call last):
                           File "/Users/cheskeltwersky/Documents/clickhouse-backup/test/testflows/clickhouse_backup/regression.py", line 79, in <module>
                             regression()
                           File "/opt/homebrew/lib/python3.12/site-packages/testflows/_core/test.py", line 3273, in __call__
                             return self.__run__(**args)
                                    ^^^^^^^^^^^^^^^^^^^^
                           File "/opt/homebrew/lib/python3.12/site-packages/testflows/_core/test.py", line 3371, in __run__
                             return self.type(**kwargs, test=self)(**args)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                           File "/opt/homebrew/lib/python3.12/site-packages/testflows/_core/test.py", line 2061, in __call__
                             return callable()
                                    ^^^^^^^^^^
                           File "/opt/homebrew/lib/python3.12/site-packages/testflows/_core/test.py", line 1959, in callable
                             return _test_wrapper()
                                    ^^^^^^^^^^^^^^^
                           File "/opt/homebrew/lib/python3.12/site-packages/testflows/_core/test.py", line 1943, in _test_wrapper
                             r = test(**self.kwargs["args"])
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
                           File "/opt/homebrew/lib/python3.12/site-packages/testflows/_core/test.py", line 3273, in __call__
                             return self.__run__(**args)
                                    ^^^^^^^^^^^^^^^^^^^^
                           File "/opt/homebrew/lib/python3.12/site-packages/testflows/_core/test.py", line 3373, in __run__
                             return run(test)
                                    ^^^^^^^^^
                           File "/opt/homebrew/lib/python3.12/site-packages/testflows/_core/test.py", line 3295, in run
                             return process_func_result(self.func(test, **args))
                                                        ^^^^^^^^^^^^^^^^^^^^^^^
                           File "/Users/cheskeltwersky/Documents/clickhouse-backup/test/testflows/clickhouse_backup/regression.py", line 48, in regression
                             with Cluster(local, nodes=nodes) as cluster:
                           File "/Users/cheskeltwersky/Documents/clickhouse-backup/test/testflows/clickhouse_backup/../helpers/cluster.py", line 616, in __enter__
                             self.up()
                           File "/Users/cheskeltwersky/Documents/clickhouse-backup/test/testflows/clickhouse_backup/../helpers/cluster.py", line 736, in up
                             self.node(node_name).wait_healthy()
                           File "/Users/cheskeltwersky/Documents/clickhouse-backup/test/testflows/clickhouse_backup/../helpers/cluster.py", line 145, in wait_healthy
                             if self.query("select version()", no_checks=True, timeout=300, steps=False).exitcode == 0:
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                           File "/Users/cheskeltwersky/Documents/clickhouse-backup/test/testflows/clickhouse_backup/../helpers/cluster.py", line 375, in query
                             r = self.cluster.bash(self.name)(command, *nargs, **kwargs)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                           File "/Users/cheskeltwersky/Documents/clickhouse-backup/test/testflows/clickhouse_backup/../helpers/cluster.py", line 585, in bash
                             raise RuntimeError(f"failed to open bash to node {node}")
                         RuntimeError: failed to open bash to node clickhouse1

Failing

✘ [ Error ] /clickhouse backup (14m 0s)

Coverage

QA-SRS013 ClickHouse Backup Utility
  72 requirements (72 untested 100.0%)

1 suite (1 errored)
21 steps (18 ok, 3 errored)

Total time 14m 0s

Executed on Mar 25,2024 21:16
TestFlows.com Open-Source Software Testing Framework v1.9.230705.1122619
ls: ./test/testflows/clickhouse_backup/tests/snapshots/cli.py.cli.snapshot: No such file or directory

@Slach
Copy link
Collaborator

Slach commented Mar 26, 2024

sorry
edit
./test/testflows/.env
set content

export CLICKHOUSE_TESTS_DIR=${PWD}/test/testflows/clickhouse_backup

and run

./venv/bin/activate
RUN_TESTS="/clickhouse backup/cli/default config" ./test/testflows/run.sh

@dermasmid
Copy link
Contributor Author

i ran it but didnt get a new file

Coverage

QA-SRS013 ClickHouse Backup Utility
  72 requirements (2 satisfied 2.8%, 70 untested 97.2%)

1 suite (1 ok)
2 scenarios (2 ok)
43 steps (43 ok)

Total time 38s 312ms

Executed on Mar 27,2024 11:37
TestFlows.com Open-Source Software Testing Framework v1.9.230705.1122619
(venv) cheskeltwersky@Cheskels-Mac clickhouse-backup % git status
On branch chunk-size
Your branch is up to date with 'origin/chunk-size'.

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        deleted:    test/testflows/clickhouse_backup/tests/snapshots/cli.py.cli.snapshot

@Slach
Copy link
Collaborator

Slach commented Mar 27, 2024

great then let's run whole testsuite

./test/testflows/run.sh

@Slach
Copy link
Collaborator

Slach commented Mar 27, 2024

ok. to avoid conflict with master i will merge your PR (all integration test passed)

please be ready fix something, if memory usage will significant change

@Slach Slach merged commit 36b42e9 into Altinity:master Mar 27, 2024
13 of 18 checks passed
@dermasmid
Copy link
Contributor Author

Thanks.

Once there's an updated release I'll be able to test it

Slach added a commit that referenced this pull request Mar 27, 2024
@Slach
added GCS_CHUNK_SIZE config parameter, try to speedup GCS upload fix …
f4e5fe6 
…#874, thanks @dermasmid

Merge branch 'master' into v2.5
# Conflicts:
#	test/testflows/clickhouse_backup/tests/snapshots/cli.py.cli.snapshot
@dermasmid
Copy link
Contributor Author

Is there an image with these changes in them, so I can test it?

@Slach
Copy link
Collaborator

Slach commented Mar 29, 2024

try altinity/clickhouse-backup:master this is not persistent tag, this is temporary tag

@Slach Slach mentioned this pull request Mar 31, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@dermasmid @Slach