CHINA-280: Set 1 hour request timeout for S3 during restore #154
Conversation
ianton-ru
force-pushed
the
CHINA-280
branch
8 times, most recently
from
94cb7cb
to
b67ffc0
Compare
|
Failed tests look like docker/docker-py#3256 |
Fix in #153 |
ch_backup/clickhouse/disks.py
Outdated
|
|
||
| request_timeout_ms = int(disk_config.get("request_timeout_ms", 0)) | ||
| if request_timeout_ms < CH_OBJECT_STORAGE_REQUEST_TIMEOUT_MS: | ||
| disks_config[tmp_disk_name]["request_timeout_ms"] = str( | ||
| CH_OBJECT_STORAGE_REQUEST_TIMEOUT_MS | ||
| ) | ||
| disks_config[disk_name] = { | ||
| "request_timeout_ms": { | ||
| "@replace": "replace", | ||
| "#text": str(CH_OBJECT_STORAGE_REQUEST_TIMEOUT_MS), | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
have tested manualy and:
root@clickhouse01:/# cat /tmp/clickhouse-disks-config.xml
<?xml version="1.0" encoding="utf-8"?>
<yandex>
<storage_configuration>
<disks>
<hdd1>
<path>/hdd1/</path>
</hdd1>
<hdd2>
<path>/hdd2/</path>
</hdd2>
<s3>
<type>s3</type>
<endpoint>http://minio01:9000/cloud-storage-01/data/</endpoint>
<access_key_id>***</access_key_id>
<secret_access_key>**</secret_access_key>
</s3>
<s3_source>
<type>s3</type>
<endpoint>http://minio01:9000/cloud-storage-01/data/</endpoint>
<access_key_id>**</access_key_id>
<secret_access_key>**</secret_access_key>
</s3_source>
</disks>
</storage_configuration>
Is that exepected?
Ps. Im not sure if the test runned correctly due to broken CI, will recheck after green CI.
There was a problem hiding this comment.
Can't understand. ch-backup doesn't modify original config files, it creates new one with name cloud_storage_tmp_disk_<disk_name>_source.xml. So clickhouse-disks-config.xml - it's an original file, not created by ch-backup.
There was a problem hiding this comment.
/tmp/clickhouse-disks-config.xml not /var/lib/clcikhouse/... We are generating additional config for clickhouse-disks usage. But I think we don't have to worry that as long as CI is broken.
Co-authored-by: MikhailBurdukov <102754618+MikhailBurdukov@users.noreply.github.com>
| disks_config[disk_name] = { | ||
| "request_timeout_ms": { | ||
| "@replace": "replace", | ||
| "#text": str(CH_OBJECT_STORAGE_REQUEST_TIMEOUT_MS), | ||
| } |
There was a problem hiding this comment.
What is the motivation of replacing request_timeout_ms for clickhouse-server config? AFAIK there is only one long-running operation(CopyObject) and is it done using with clickhouse-disk. Maybe make sense to keep request_timeout_ms only for clickhouse-disks config?
There was a problem hiding this comment.
For consistency.
If source disk is not used in clickhouse-server, better remove additional config for server at all.
No description provided.