Upload a backup to GCS is stuck, when upload_concurrency: 0

[mglotov]

environment: k8s 1.26
clickhouse config:

...
    podTemplates:
      - name: clickhouse-22.8-lts
...
        spec:
          serviceAccount: clickhouse-pod
          containers:
            - name: clickhouse
              image: clickhouse/clickhouse-server:22.8
              volumeMounts:
                - name: server-storage
                  mountPath: /var/lib/clickhouse
              resources:
                requests:
                  memory: "1024Mi"
                  cpu: 100m
                limits:
                  memory: "1280Mi"
            - name: clickhouse-backup
              image: altinity/clickhouse-backup:master
              imagePullPolicy: Always
              command:
                - bash
                - -xc
                - "/bin/clickhouse-backup server"
              env:
                - name: LOG_LEVEL
                  value: "debug"
                - name: ALLOW_EMPTY_BACKUPS
                  value: "true"
                - name: API_LISTEN
                  value: "0.0.0.0:7171"
                - name: BACKUPS_TO_KEEP_REMOTE
                  value: "14"
                - name: REMOTE_STORAGE
                  value: "gcs"
                - name: GCS_BUCKET
                  value: "BUCKET_NAME"
                - name: GCS_PATH
                  value: backup/shard-{shard}
                - name: CLICKHOUSE_SKIP_TABLES
                  value: "system.*,INFORMATION_SCHEMA.*,default.*" # All tables except these will be backed up 

GCP node type: n1-standard-1

Upload operation is stuck on:

2023/07/10 05:45:46.015224 debug /tmp/.clickhouse-backup-metadata.cache.GCS load 14 elements logger=gcs
2023/07/10 05:45:46.097387 debug /tmp/.clickhouse-backup-metadata.cache.GCS save 14 elements logger=gcs
2023/07/10 05:45:46.100377 debug prepare table concurrent semaphore with concurrency=0 len(tablesForUpload)=7 backup=BACKUP_NAME operation=upload

Looks like a bug with a concurrency calculation for an upload operation, because for download it's 1, but for upload it's 0.
clickhouse-backup config:

clickhouse-backup print-config|grep concurrency
    download_concurrency: 1
    upload_concurrency: 0
    concurrency: 2
    concurrency: 2
    concurrency: 2

[Slach]

could, you remove upload_concurrency from your /etc/clickhouse-backup/config.yml in this case it will CPU_CORES/2 for GCS

[mglotov]

could, you remove upload_concurrency from your /etc/clickhouse-backup/config.yml in this case it will CPU_CORES/2 for GCS

It isn't set manually anywhere

[mglotov]

As a workaround, I've set the next env var for the clickhouse-backup container:

                - name: UPLOAD_CONCURRENCY
                  value: "1"

[Slach]

Do you have only 1 vCPU for your clickhouse installation?
could you share
cat /proc/cpuinfo
inside clickhouse-backup container?

[mglotov]

Do you have only 1 vCPU for your clickhouse installation? could you share cat /proc/cpuinfo inside clickhouse-backup container?

Yes, the node has only 1 vCPU

chi-dc1-cluster1-0-0-0:/# cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 79
model name	: Intel(R) Xeon(R) CPU @ 2.20GHz
stepping	: 0
microcode	: 0xffffffff
cpu MHz		: 2200.218
cache size	: 56320 KB
physical id	: 0
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat md_clear arch_capabilities
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa mmio_stale_data retbleed
bogomips	: 4400.43
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management: