Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Backup fails to upload through FTP protocol #283

Closed
mmolinac opened this issue Oct 15, 2021 · 3 comments · Fixed by #288
Closed

Backup fails to upload through FTP protocol #283

mmolinac opened this issue Oct 15, 2021 · 3 comments · Fixed by #288
Assignees
@Slach

Comments

@mmolinac
Copy link

Hi!
We're trying to upload backups created with clickhouse-backup to our remote location, with a similar configuration for both SFTP and FTP.
Whilst the first one works like a charm, we want to use FTP as it's faster.
However, we get an error doing it with FTP:

error: 550 clickhouse01/clickhouse01-dev-20211015-1549/shadow/mycompany/daily_local: No such file or directory

Our config.ftp.yml config file below (sensitive information redacted for security reasons):

---
general:
    remote_storage: ftp
    disable_progress_bar: false
    backups_to_keep_local: 0
    backups_to_keep_remote: 0
clickhouse:
    username: "backup_user"
    password: "12341234"
    host: "localhost"
    port: 9000
    data_path: ""
    skip_tables:
        - system.*
    timeout: 5m
    freeze_by_part: false
    secure: false
    skip_verify: false
ftp:
    address: "u111111.your-storagebox.de:21"
    timeout: 2m
    username: "u111111"
    password: "222222"
    path: "clickhouse01"  # This path has been created beforehand
    compression_format: gzip  # also fails if it's "tar"
    compression_level: 1
    debug: true

Now follows the trace of the command, with additional debug information:

# LOG_LEVEL=debug clickhouse-backup --config /etc/clickhouse-backup/config.ftp.yml upload clickhouse01-dev-20211015-1549
2021/10/15 15:49:31 debug SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER'
2021/10/15 15:49:31 debug SELECT * FROM system.disks;
2021/10/15 15:49:31 debug SELECT * FROM system.disks;
220 ProFTPD Server (Hetzner Backup) [::ffff:77.44.66.22]
USER u111111
331 Password required for u111111
PASS 222222
230 User u111111 logged in
FEAT
211-Features:
 AUTH TLS
 CCC
 CLNT
 EPRT
 EPSV
 HOST
 LANG ko-KR.UTF-8;ko-KR;it-IT.UTF-8;it-IT;es-ES.UTF-8;es-ES;bg-BG.UTF-8;bg-BG;ru-RU.UTF-8;ru-RU;zh-TW.UTF-8;zh-TW;zh-CN.UTF-8;zh-CN;ja-JP.UTF-8;ja-JP;fr-FR.UTF-8;fr-FR;en-US.UTF-8;en-US*
 MDTM
 MFF modify;UNIX.group;UNIX.mode;
 MFMT
 MLST modify*;perm*;size*;type*;unique*;UNIX.group*;UNIX.groupname*;UNIX.mode*;UNIX.owner*;UNIX.ownername*;
 PBSZ
 PROT
 RANG STREAM
 REST STREAM
 SIZE
 SSCN
 TVFS
 UTF8
211 End
TYPE I
200 Type set to I
OPTS UTF8 ON
200 UTF8 set to on
2021/10/15 15:49:32 debug SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER'
2021/10/15 15:49:32 debug SELECT * FROM system.disks;
PASV
227 Entering Passive Mode (78,47,6,211,192,0).
MLSD clickhouse01
150 Opening BINARY mode data connection for MLSD
226 Transfer complete
PASV
227 Entering Passive Mode (78,47,6,211,208,213).
MLSD clickhouse01
150 Opening BINARY mode data connection for MLSD
226 Transfer complete
PASV
227 Entering Passive Mode (78,47,6,211,224,182).
MLSD .
150 Opening BINARY mode data connection for MLSD
226 Transfer complete
PASV
227 Entering Passive Mode (78,47,6,211,250,179).
MLSD clickhouse01/clickhouse01-dev-20211015-1546
150 Opening BINARY mode data connection for MLSD
226 Transfer complete
PASV
227 Entering Passive Mode (78,47,6,211,196,49).
MLSD clickhouse01/clickhouse01-dev-20211014-1930
150 Opening BINARY mode data connection for MLSD
226 Transfer complete
PASV
227 Entering Passive Mode (78,47,6,211,220,5).
RETR clickhouse01/clickhouse01-dev-20211014-1930/metadata.json
150 Opening BINARY mode data connection for clickhouse01/clickhouse01-dev-20211014-1930/metadata.json (983 bytes)
226 Transfer complete
PASV
227 Entering Passive Mode (78,47,6,211,198,5).
MLSD clickhouse01/clickhouse01-dev-20211015-1325
150 Opening BINARY mode data connection for MLSD
226 Transfer complete
PASV
227 Entering Passive Mode (78,47,6,211,214,228).
RETR clickhouse01/clickhouse01-dev-20211015-1325/metadata.json
150 Opening BINARY mode data connection for clickhouse01/clickhouse01-dev-20211015-1325/metadata.json (982 bytes)
226 Transfer complete
PASV
227 Entering Passive Mode (78,47,6,211,214,242).
MLSD clickhouse01/clickhouse01-dev-20211015-1544
150 Opening BINARY mode data connection for MLSD
226 Transfer complete
PASV
227 Entering Passive Mode (78,47,6,211,247,162).
MLSD clickhouse01/clickhouse01-dev-20211015-1345
150 Opening BINARY mode data connection for MLSD
226 Transfer complete
PASV
227 Entering Passive Mode (78,47,6,211,241,21).
RETR clickhouse01/clickhouse01-dev-20211015-1345/metadata.json
150 Opening BINARY mode data connection for clickhouse01/clickhouse01-dev-20211015-1345/metadata.json (982 bytes)
226 Transfer complete
PASV
227 Entering Passive Mode (78,47,6,211,194,124).
MLSD clickhouse01/clickhouse01-dev-20211014-1936
150 Opening BINARY mode data connection for MLSD
226 Transfer complete
PASV
227 Entering Passive Mode (78,47,6,211,234,91).
RETR clickhouse01/clickhouse01-dev-20211014-1936/metadata.json
150 Opening BINARY mode data connection for clickhouse01/clickhouse01-dev-20211014-1936/metadata.json (983 bytes)
226 Transfer complete
2021/10/15 15:49:32 debug prepare table concurrent semaphore with concurrency=1 len(tablesForUpload)=6 backup=clickhouse01-dev-20211015-1549 operation=upload
2021/10/15 15:49:32 debug start uploadTableData mycompany.daily_local with concurrency=1 len(table.Parts[...])=202
2021/10/15 15:49:33 debug start upload 58138 files to clickhouse01-dev-20211015-1549/shadow/mycompany/daily_local/default_1.tar.gz
PASV
227 Entering Passive Mode (78,47,6,211,196,108).
MLSD clickhouse01/clickhouse01-dev-20211015-1549/shadow/mycompany/daily_local
550 clickhouse01/clickhouse01-dev-20211015-1549/shadow/mycompany/daily_local: No such file or directory
2021/10/15 15:49:33 error CompressedStreamUpload return error: 550 clickhouse01/clickhouse01-dev-20211015-1549/shadow/mycompany/daily_local: No such file or directory
2021/10/15 15:49:33 error can't acquire semaphore during Upload: context canceled backup=clickhouse01-dev-20211015-1549 operation=upload
2021/10/15 15:49:33 debug start uploadTableData mycompany.daily_local_tmp with concurrency=1 len(table.Parts[...])=0
2021/10/15 15:49:33 debug finish uploadTableData mycompany.daily_local_tmp with concurrency=1 len(table.Parts[...])=0 metadataFiles=map[], uploadedBytes=0
CWD /
250 CWD command successful
MKD clickhouse01
550 clickhouse01: File exists
MKD clickhouse01/clickhouse01-dev-20211015-1549
257 "/clickhouse01/clickhouse01-dev-20211015-1549" - Directory successfully created
MKD clickhouse01/clickhouse01-dev-20211015-1549/metadata
257 "/clickhouse01/clickhouse01-dev-20211015-1549/metadata" - Directory successfully created
MKD clickhouse01/clickhouse01-dev-20211015-1549/metadata/mycompany
257 "/clickhouse01/clickhouse01-dev-20211015-1549/metadata/mycompany" - Directory successfully created
PASV
227 Entering Passive Mode (78,47,6,211,198,215).
STOR clickhouse01/clickhouse01-dev-20211015-1549/metadata/mycompany/daily_local_tmp.json
150 Opening BINARY mode data connection for clickhouse01/clickhouse01-dev-20211015-1549/metadata/mycompany/daily_local_tmp.json
226 Transfer complete
2021/10/15 15:49:33  info done                      backup=clickhouse01-dev-20211015-1549 duration=115ms operation=upload size=4.60KiB table=mycompany.daily_local_tmp
2021/10/15 15:49:33 error one of upload go-routine return error: one of uploadTableData go-routine return error: can't upload: 550 clickhouse01/clickhouse01-dev-20211015-1549/shadow/mycompany/daily_local: No such file or directory

Looks like the folder is not being created before trying to be accessed.
As I said, with our SFTP configuration, the backups are correctly updated.
For the sake of completion, I'll attach below the SFTP configuration file:

---
general:
    remote_storage: sftp
    disable_progress_bar: false
    backups_to_keep_local: 0
    backups_to_keep_remote: 0
clickhouse:
    username: "backup_user"
    password: "12341234"
    host: "localhost"
    port: 9000
    data_path: ""
    skip_tables:
        - system.*
    timeout: 5m
    freeze_by_part: false
    secure: false
    skip_verify: false
sftp:
    address: "u111111.your-storagebox.de"
    timeout: 2m
    username: "u111111"
    password: "222222"
    path: "clickhouse01"  # THIS PATH MUST EXIST. WON'T BE AUTOMATICALLY CREATED!
    compression_format: gzip
    compression_level: 1
    concurrency: 8
@Slach
Copy link
Collaborator

Slach commented Oct 15, 2021

Thanks a lot for reporting, look like we should add FTP testing on our github actions

@Slach Slach self-assigned this Oct 15, 2021
@amakeenk
Copy link

I faced the same problem with backup via FTP:

OS: ALT Server 10.0
clickhouse-backup-1.1.0-alt1

# clickhouse-backup create_remote test_backup
2021/10/22 10:04:09  info done                      backup=test_backup operation=create table=test_db.test_table
2021/10/22 10:04:09  info done                      backup=test_backup duration=150ms operation=create
2021/10/22 10:04:09 error CompressedStreamUpload return error: 550 test_backup/shadow/test_db/test_table: No such file or directory
2021/10/22 10:04:09 error one of upload go-routine return error: one of uploadTableData go-routine return error: can't upload: 550 test_backup/shadow/test_db/test_table: No such file or directory

@Slach
Copy link
Collaborator

Slach commented Oct 22, 2021

found the root reason, look like ProFTPD return 550 error for LIST command for not exists directory
i'll prepare fix ASAP

@Slach Slach mentioned this issue Oct 24, 2021
Merged
Slach added a commit that referenced this issue Oct 24, 2021
@Slach
Merge pull request #288 from Altinity/master
8567fc6 
# v1.2.0

INCOMPATIBLE CHANGES
- REST API `/backup/status` now return only latest executed command with status and error message

IMPROVEMENTS
- Added REST API `/backup/list/local` and `/backup/list/remote` to allow list backup types separately
- Decreased background backup creation time via REST API `/backup/create`, during avoid list remote backups for update metrics value 
- Decreased backup creation time, during avoid scan whole `system.tables` when set `table` query string parameter or `--tables` cli parameter     
- Added `last` and `filter` query string parameters to REST API `/backup/actions`, to avoid pass to client long JSON documents
- Improved `FTP` remote storage parallel upload / download
- Added `FTP_CONCURRENCY` to allow, by default MAX_CPU / 2 
- Added `FTP_DEBUG` setting, to allow debug FTP commands
- Added `FTP` to CI/CD on any commit
- Added race condition check to CI/CD

BUG FIXES
- environment variable `LOG_LEVEL` now apply to `clickhouse-backup server` properly
- fix #280, incorrect prometheus metrics measurement for `/backup/create`, `/backup/upload`, `/backup/download`
- fix #273, return `S3_PART_SIZE` back, but calculates it smartly
- fix #252, now you can pass `last` and `filter` query string parameters
- fix #246, incorrect error messages when use `REMOTE_STORAGE=none`
- fix #283, properly handle error message from `FTP` server 
- fix #268, properly restore legacy backup for schema without database name
- fix #287
@mmolinac mmolinac mentioned this issue Oct 25, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Slach Slach

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Release 1.2.0
3 participants
@Slach @mmolinac @amakeenk