backup-push does not work on the Postgresql 15/16 without $PGDATA in command line #1385
Comments
|
@sebasmannem Sebas, I think it's a problem of the remote backup. Also, as a side note - was it really a good idea to default to remote backup when run without path in command line? @IgorOhrimenko just use |
We can move the remote backup option to a separate flag, e.g. |
|
I think this is a good idea. It's kind of new feature compared to regular backup and we just activate it silently... maybe give a hint if no --remote is given? |
There is $PGDATA in the /var/lib/postgresql/.walg.json and |
|
You encounter one bug and one unexpected behavior:
|
|
#1396 currently allows to fix the first error, another error is triggered at a later step : Please provide steps to reproduce (if it's a bug)`local.json`Please add config and wal-g stdout/stderr logs for debug purposealso you can use WALG_LOG_LEVEL=DEVEL for logs collecting If you can, provide logs$ WALG_LOG_LEVEL=DEVEL ./wal-g --config local.clear.json backup-push
DEBUG: 2022/12/07 22:54:15.202555 --- COMPILED ENVIRONMENT VARS ---
PGBACKREST_STANZA=main
PGDATABASE=test_database
PGHOST=localhost
PGPASSWORD=--HIDDEN--
PGPORT=4321
PGUSER=replication
TOTAL_BG_UPLOADED_LIMIT=32
WALG_COMPRESSION_METHOD=zstd
WALG_DELTA_MAX_STEPS=6
WALG_DOWNLOAD_CONCURRENCY=10
WALG_FILE_PREFIX=data
WALG_INTEGRITY_MAX_DELAYED_WALS=0
WALG_LIBSODIUM_KEY_TRANSFORM=none
WALG_LOG_LEVEL=DEVEL
WALG_PG_WAL_SIZE=16
WALG_PREVENT_WAL_OVERWRITE=false
WALG_SERIALIZER_TYPE=json_default
WALG_SKIP_REDUNDANT_TARS=false
WALG_STORE_ALL_CORRUPT_BLOCKS=false
WALG_TAR_DISABLE_FSYNC=true
WALG_TAR_SIZE_THRESHOLD=1073741823
WALG_UPLOAD_CONCURRENCY=16
WALG_UPLOAD_DISK_CONCURRENCY=1
WALG_UPLOAD_QUEUE=2
WALG_UPLOAD_WAL_METADATA=NOMETADATA
WALG_USE_COPY_COMPOSER=false
WALG_USE_RATING_COMPOSER=false
WALG_USE_REVERSE_UNPACK=false
WALG_USE_WAL_DELTA=false
WALG_VERIFY_PAGE_CHECKSUMS=false
WALG_WITHOUT_FILES_METADATA=false
DEBUG: 2022/12/07 22:54:15.202705 Initializing tmp connection to read Postgres info
DEBUG: 2022/12/07 22:54:15.212066 Datadir: /home/gvarouchas/dev/pgsql/data
DEBUG: 2022/12/07 22:54:15.212207 Postgres version: 150000
DEBUG: 2022/12/07 22:54:15.212350 Postgres SystemIdentifier: 150000
INFO: 2022/12/07 22:54:15.212526 Running remote backup through Postgres connection.
INFO: 2022/12/07 22:54:15.212534 Features like delta backup are disabled, there might be a performance impact.
INFO: 2022/12/07 22:54:15.212538 To run with local backup functionalities, supply [db_directory].
DEBUG: 2022/12/07 22:54:15.212559 Uploading folder: &{data basebackups_005}
DEBUG: 2022/12/07 22:54:15.212570 Connecting to Postgres (replication connection)
INFO: 2022/12/07 22:54:15.216669 Starting remote backup
INFO: 2022/12/07 22:54:15.266169 Streaming remote backup
DEBUG: 2022/12/07 22:54:15.268538 Put base_0000000100000000000000EA/tar_partitions/part_001.tar.zst into basebackups_005
DEBUG: 2022/12/07 22:54:15.269049 Next file
INFO: 2022/12/07 22:54:15.269076 Adding data directory
ERROR: 2022/12/07 22:54:15.269275 CompressAndEncrypt: compress: archive/tar: invalid tar header
FS error : Unable to copy data to data/basebackups_005/base_0000000100000000000000EA/tar_partitions/part_001.tar.zst
github.com/wal-g/wal-g/pkg/storages/storage.NewError
/home/gvarouchas/code/github.com/LeGEC/wal-g/pkg/storages/storage/errors.go:27
github.com/wal-g/wal-g/pkg/storages/fs.NewError
/home/gvarouchas/code/github.com/LeGEC/wal-g/pkg/storages/fs/folder.go:17
github.com/wal-g/wal-g/pkg/storages/fs.(*Folder).PutObject
/home/gvarouchas/code/github.com/LeGEC/wal-g/pkg/storages/fs/folder.go:121
github.com/wal-g/wal-g/internal.(*Uploader).Upload
/home/gvarouchas/code/github.com/LeGEC/wal-g/internal/uploader.go:173
github.com/wal-g/wal-g/internal/databases/postgres.(*StreamingBaseBackup).Upload
/home/gvarouchas/code/github.com/LeGEC/wal-g/internal/databases/postgres/streaming_base_backup.go:135
github.com/wal-g/wal-g/internal/databases/postgres.(*BackupHandler).runRemoteBackup
/home/gvarouchas/code/github.com/LeGEC/wal-g/internal/databases/postgres/backup_push_handler.go:463
github.com/wal-g/wal-g/internal/databases/postgres.(*BackupHandler).createAndPushRemoteBackup
/home/gvarouchas/code/github.com/LeGEC/wal-g/internal/databases/postgres/backup_push_handler.go:374
github.com/wal-g/wal-g/internal/databases/postgres.(*BackupHandler).handleBackupPushRemote
/home/gvarouchas/code/github.com/LeGEC/wal-g/internal/databases/postgres/backup_push_handler.go:329
github.com/wal-g/wal-g/internal/databases/postgres.(*BackupHandler).HandleBackupPush
/home/gvarouchas/code/github.com/LeGEC/wal-g/internal/databases/postgres/backup_push_handler.go:310
github.com/wal-g/wal-g/cmd/pg.glob..func4
/home/gvarouchas/code/github.com/LeGEC/wal-g/cmd/pg/backup_push.go:97
github.com/spf13/cobra.(*Command).execute
/home/gvarouchas/code/github.com/LeGEC/wal-g/vendor/github.com/spf13/cobra/command.go:860
github.com/spf13/cobra.(*Command).ExecuteC
/home/gvarouchas/code/github.com/LeGEC/wal-g/vendor/github.com/spf13/cobra/command.go:974
github.com/spf13/cobra.(*Command).Execute
/home/gvarouchas/code/github.com/LeGEC/wal-g/vendor/github.com/spf13/cobra/command.go:902
github.com/wal-g/wal-g/cmd/pg.Execute
/home/gvarouchas/code/github.com/LeGEC/wal-g/cmd/pg/pg.go:45
main.main
/home/gvarouchas/code/github.com/LeGEC/wal-g/main/pg/main.go:8
runtime.main
/home/gvarouchas/go/src/runtime/proc.go:250
runtime.goexit
/home/gvarouchas/go/src/runtime/asm_amd64.s:1594
ERROR: 2022/12/07 22:54:15.269384 CompressAndEncrypt: compress: archive/tar: invalid tar header
FS error : Unable to copy data to data/basebackups_005/base_0000000100000000000000EA/tar_partitions/part_001.tar.zst
github.com/wal-g/wal-g/pkg/storages/storage.NewError
/home/gvarouchas/code/github.com/LeGEC/wal-g/pkg/storages/storage/errors.go:27
github.com/wal-g/wal-g/pkg/storages/fs.NewError
/home/gvarouchas/code/github.com/LeGEC/wal-g/pkg/storages/fs/folder.go:17
github.com/wal-g/wal-g/pkg/storages/fs.(*Folder).PutObject
/home/gvarouchas/code/github.com/LeGEC/wal-g/pkg/storages/fs/folder.go:121
github.com/wal-g/wal-g/internal.(*Uploader).Upload
/home/gvarouchas/code/github.com/LeGEC/wal-g/internal/uploader.go:173
github.com/wal-g/wal-g/internal/databases/postgres.(*StreamingBaseBackup).Upload
/home/gvarouchas/code/github.com/LeGEC/wal-g/internal/databases/postgres/streaming_base_backup.go:135
github.com/wal-g/wal-g/internal/databases/postgres.(*BackupHandler).runRemoteBackup
/home/gvarouchas/code/github.com/LeGEC/wal-g/internal/databases/postgres/backup_push_handler.go:463
github.com/wal-g/wal-g/internal/databases/postgres.(*BackupHandler).createAndPushRemoteBackup
/home/gvarouchas/code/github.com/LeGEC/wal-g/internal/databases/postgres/backup_push_handler.go:374
github.com/wal-g/wal-g/internal/databases/postgres.(*BackupHandler).handleBackupPushRemote
/home/gvarouchas/code/github.com/LeGEC/wal-g/internal/databases/postgres/backup_push_handler.go:329
github.com/wal-g/wal-g/internal/databases/postgres.(*BackupHandler).HandleBackupPush
/home/gvarouchas/code/github.com/LeGEC/wal-g/internal/databases/postgres/backup_push_handler.go:310
github.com/wal-g/wal-g/cmd/pg.glob..func4
/home/gvarouchas/code/github.com/LeGEC/wal-g/cmd/pg/backup_push.go:97
github.com/spf13/cobra.(*Command).execute
/home/gvarouchas/code/github.com/LeGEC/wal-g/vendor/github.com/spf13/cobra/command.go:860
github.com/spf13/cobra.(*Command).ExecuteC
/home/gvarouchas/code/github.com/LeGEC/wal-g/vendor/github.com/spf13/cobra/command.go:974
github.com/spf13/cobra.(*Command).Execute
/home/gvarouchas/code/github.com/LeGEC/wal-g/vendor/github.com/spf13/cobra/command.go:902
github.com/wal-g/wal-g/cmd/pg.Execute
/home/gvarouchas/code/github.com/LeGEC/wal-g/cmd/pg/pg.go:45
main.main
/home/gvarouchas/code/github.com/LeGEC/wal-g/main/pg/main.go:8
runtime.main
/home/gvarouchas/go/src/runtime/proc.go:250
runtime.goexit
/home/gvarouchas/go/src/runtime/asm_amd64.s:1594 |
IgorOhrimenko
changed the title
IgorOhrimenko
changed the title
|
A year has passed. and the same workaround is working: Relative #1248 |
It’s not workaround. It’s correct expected recommended syntax. The problem is we have a remote backup which tries to kick in… and that’s kind of a bug. |
tangtx3
pushed a commit
to tangtx3/wal-g
that referenced
this issue
Nov 16, 2023
* In the streamFromPostgres method, CopyData case adds data receiving processing from pgsql version 15 or later. * In the streamFromPostgres method, add NoticeResponse case. * Troubleshoot EOF problems caused by the last zeroBlock message sent from pgsql15 or later. * Add test for Supports remote backup push from PostgreSQL 15 or later
tangtx3
pushed a commit
to tangtx3/wal-g
that referenced
this issue
Nov 16, 2023
* In the streamFromPostgres method, CopyData case adds data receiving processing from pgsql version 15 or later. * In the streamFromPostgres method, add NoticeResponse case. * Troubleshoot EOF problems caused by the last zeroBlock message sent from pgsql15 or later. * Add test for Supports remote backup push from PostgreSQL 15 or later Signed-off-by: tangtx <tianxitang@harmonycloud.cn>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Database name
Postgresql
Issue description
Describe your problem
Command backup-push does not work on the Postgresql 15 with error:
I try last v2.0.1 and compiled master 3d22cab.
Please provide steps to reproduce
Please add config and wal-g stdout/stderr logs for debug purpose
/var/lib/postgresql/.walg.json
Full logs with "WALG_LOG_LEVEL": "DEVEL"
\dconfig
List of non-default configuration parameters Parameter | Value ----------------------------+---------------------------------------------------------- application_name | psql archive_command | exit 0 archive_mode | on archive_timeout | 1min cluster_name | cluster-psql-test-15 config_file | /var/lib/postgresql/15/cluster-psql-test/postgresql.conf data_directory | /var/lib/postgresql/15/cluster-psql-test default_text_search_config | pg_catalog.english effective_cache_size | 1685MB effective_io_concurrency | 200 hba_file | /var/lib/postgresql/15/cluster-psql-test/pg_hba.conf ident_file | /var/lib/postgresql/15/cluster-psql-test/pg_ident.conf lc_messages | en_US.UTF-8 lc_monetary | en_US.UTF-8 lc_numeric | en_US.UTF-8 lc_time | en_US.UTF-8 listen_addresses | 127.0.0.1,10.10.10.10 log_directory | /var/log/postgresql log_filename | cluster-psql-test.log logging_collector | on log_min_duration_statement | 5s log_rotation_age | 0 log_rotation_size | 0 log_timezone | Europe/Moscow maintenance_work_mem | 198MB max_connections | 500 max_wal_size | 2GB min_wal_size | 1GB pg_stat_statements.max | 10000 pg_stat_statements.track | all random_page_cost | 1.1 shared_buffers | 991MB shared_preload_libraries | pg_stat_statements TimeZone | Europe/Moscow unix_socket_directories | /var/run/postgresql/ wal_buffers | 16MB wal_keep_size | 2GB wal_level | logical wal_log_hints | on work_mem | 2MB (40 rows)The text was updated successfully, but these errors were encountered: