SSH sessions not closed when running in systemd.service for list,upload,download command

[No0key]

Hello!
I created an issue that SSH sessions were not closed after the list command, you closed it with status fixed. I updated version of clickhouse-backup tool.

Command clickhouse-backup -c /path/to/config list now close SSH sessions, but when running in systemd service SSH session still remain open.

Logs from remote sftp storage

NOT CORRECT
Command: curl localhost:7171/backup/list/remote

Dec  6 09:16:53 sftp_server sshd[10231]: Accepted publickey for backup_user from backup_host_ip port 43472 ssh2: RSA...
Dec  6 09:16:53 sftp_server sshd[10231]: pam_unix(sshd:session): session opened for user backup_user by (uid=0)
Dec  6 09:16:53 sftp_server systemd-logind[1078]: New session 2079 of user backup_user.
....

CORRECT
Command: clickhouse-backup -c /path/to/config list

Dec  6 09:17:07 sftp_server sshd[10268]: Accepted publickey for backup_user from backup_host_ip port 43522 ssh2: RSA...
Dec  6 09:17:07 sftp_server sshd[10268]: pam_unix(sshd:session): session opened for user backup_user by (uid=0)
Dec  6 09:17:07 sftp_server systemd-logind[1078]: New session 2082 of user backup_user.
Dec  6 09:17:07 sftp_server sshd[10268]: pam_unix(sshd:session): session closed for user backup_user
Dec  6 09:17:07 sftp_server systemd-logind[1078]: Removed session 2082.

Systemd service file

[Unit]
Description=clickhouse-backup rest server
Documentation=https://github.com/AlexAkulov/clickhouse-backup
After=network-online.target systemd-resolved.service clickhouse-server.service
Wants=network-online.target systemd-resolved.service clickhouse-server.service

[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/bin/clickhouse-backup -c /opt/conf/clickhouse-backup/config.yml server
Restart=always
RestartSec=3

SyslogIdentifier=clickhouse-backup
StartLimitInterval=0

OOMScoreAdjust=0
MemoryHigh=20%

CPUSchedulingPolicy=other
CPUSchedulingPriority=0
CPUQuota=40%
Nice=0

LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

[Slach]

which clickhouse-backup version do you use?

[No0key]

[ubuntu-20] root : ~ clickhouse-backup --version                                     
Version:	 2.1.1
Git Commit:	 233bed562ab2ebe6def3d55ade81007633aefeab
Build Date:	 2022-10-20

[Slach]

Do you use remote_storage: sftp?

[No0key]

Yes, my config is below

general:
  remote_storage: sftp
  max_file_size: 1073741824
  disable_progress_bar: true
  backups_to_keep_local: 12 # tables amount
  backups_to_keep_remote: 192 # tables amount
  log_level: info
  allow_empty_backups: false
  download_concurrency: 40
  upload_concurrency: 8
  restore_schema_on_cluster: ""
  upload_by_part: true
  download_by_part: true

sftp:
  address: my_remote_storage
  username: backup_user
  password: ""
  key: "/root/.ssh/id_rsa"
  path: /remote_path
  concurrency: 8
  compression_format: tar
  compression_level: 3
  debug: true

[No0key]

Problem is not only with list command, it is appear with download, upload operations too.

[Slach]

what happens if you change
concurrency: 8 to concurrency: 1 in sftp section? is connection closed in this case?

[No0key]

No, still open

[Slach]

could you check altinity/clickhiuse-backup:2.1.3
i tried multiple times with stantard openssh server
and got

sshd                  | debug1: session_close_by_channel: channel 0 child 0
sshd                  | Close session: user root from 172.26.0.9 port 43200 id 0

for every command which run from API server

[Slach]

Which sftp server do you use?

[No0key]

Using standard sftp subsystem in openssh server, here is config for ssh server which enables sftp

# override default of no subsystems
Subsystem sftp /usr/lib/sftp-server -f local5 -l DEBUG3

Its not closing connection only when running API server.

[No0key]

Tried version 2.1.3, still connections remain open

Jan  5 07:27:57 backup-host sshd[7977]: Accepted publickey for backup_user from BACKUP_HOST_IP port 12470 ssh2: RSA SHA256.....REDACTED
Jan  5 07:27:57 backup-host sshd[7977]: pam_unix(sshd:session): session opened for user backup_user by (uid=0)
Jan  5 07:27:57 backup-host systemd-logind[1078]: New session 63458 of user backup_user

[No0key]

Any update on this issue?

[Slach]

reproduced on my side
will released on 2.2.0 in couple of days