"ERROR: [101]: raised from remote-0 ssh protocol on 'db-pgprod01': NULL result required to complete request"

[naveenchowdaryon]

Hi Team,

1. pgBackRest version:

   pgBackRest 2.44

2. PostgreSQL version:

   (PostgreSQL) 13.12

3. Operating system/version - if you have more than one server (for example, a database server, a repository host server, one or more standbys), please specify each:

SUSE Linux Enterprise Server 15 SP5

4. Did you install pgBackRest from source or from a package?

   from Package

5. pgbackreset.conf in DB server

   #[global]
   #repo-path=/var/lib/pgbackrest
   [global]
   repo1-host=db-pgbackp01
   repo1-host-user=postgres
   process-max=2
   log-level-console=info
   log-level-file=debug
   [prod_backup]
   pg1-path=/var/lib/pgsql/data

   pgbackreset.conf in backup server

   #[global]
   #repo-path=/var/lib/pgbackrest
   [global]
   repo1-path=/backup/db_backup
   repo1-retention-full=2
   process-max=2
   log-level-console=info
   log-level-file=debug
   start-fast=y
   stop-auto=y
   [prod_backup]
   pg1-path=/var/lib/pgsql/data
   pg1-host=db-pgprod01
   pg1-host-user=postgres
   pg1-user=pgbackrest
   #pg1-socket-path=/tmp
   pg1-port = 5432
   #[main]
   #pg-path=/var/lib/pgsql/data

   postgresql.conf

   wal_level = 'logical'
   archive_command = 'pgbackrest --stanza=prod_backup archive-push %p'
   archive_mode = on
   listen_addresses = '*'
   max_wal_senders = 10
   port = 5432

6. Issue description:

Running pgbackrest from remote host.
pgbackrest server: db-pgbackp01
DB Server: db-pgprod01

Initially it was working fine after completing 5% backup, backup failing with fallowing error
"ERROR: [101]: raised from remote-0 ssh protocol on 'db-pgprod01': NULL result required to complete request"

postgres@db-pgbackp01:~> pgbackrest --stanza=prod_backup backup --type=full
2023-12-11 11:10:54.465 P00 INFO: backup command begin 2.44: --exec-id=13026-ad094e4f --log-level-console=info --log-level-file=debug --pg1-host=db-pgprod01 --pg1-host-user=postgres --pg1-path=/var/lib/pgsql/data --pg1-port=5432 --pg1-user=pgbackrest --process-max=2 --repo1-path=/backup/db_backup --repo1-retention-full=2 --stanza=prod_backup --start-fast --stop-auto --type=full
2023-12-11 11:10:55.539 P00 INFO: execute non-exclusive backup start: backup begins after the requested immediate checkpoint completes
2023-12-11 11:10:57.147 P00 INFO: backup start archive = 00000002000008C80000002E, lsn = 8C8/2E0006E8
2023-12-11 11:10:57.147 P00 INFO: check archive for prior segment 00000002000008C80000002D
ERROR: [101]: raised from remote-0 ssh protocol on 'db-pgprod01': NULL result required to complete request
2023-12-11 14:00:37.022 P00 INFO: backup command end: aborted with exception [101]

Already password less authentication there between backrest server to DB server and DB server to backreset server. Initially backup was successfully running, past few day's I am seeing this error message in backrest backup log.

2023-12-11 14:00:37.022 P00 DEBUG: command/exit::exitSafe: (result: 0, error: true, signalType: 0)
2023-12-11 14:00:37.022 P00 ERROR: [101]: raised from remote-0 ssh protocol on 'dest-bmcpkdbl00.itc.global.mahle': NULL result required to complete request
--------------------------------------------------------------------
If SUBMITTING AN ISSUE please provide the following information:

                                version: 2.44
                                command: backup
                                options: --exec-id=13026-ad094e4f --log-level-console=info --log-level-file=debug --pg1-host=db-pgprod01 --pg1-host-user=postgres --pg1-path=/var/lib/pgsql/data --pg1-port=5432 --pg1-user=pgbackrest --process-max=2 --repo1-path=/backup/bmcdb_backup --repo1-retention-full=2 --stanza=prod_backup --start-fast --stop-auto --type=full

                                stack trace:
                                postgres/client.c:pgClientQuery:350:(debug log level required for parameters)
                                db/protocol.c:dbQueryProtocol:(debug log level required for parameters)
                                protocol/server.c:protocolServerProcess:(debug log level required for parameters)
                                command/remote/remote.c:cmdRemote:(debug log level required for parameters)
                                main.c:main:(debug log level required for parameters)
                                --------------------------------------------------------------------

2023-12-11 14:00:37.022 P00 INFO: backup command end: aborted with exception [101]
2023-12-11 14:00:37.022 P00 DEBUG: common/lock::lockRelease: (failOnNoLock: false)
2023-12-11 14:00:37.022 P00 DEBUG: common/lock::lockRelease: => true
2023-12-11 14:00:37.022 P00 DEBUG: command/exit::exitSafe: => 101
2023-12-11 14:02:24.632 P00 DEBUG: main::main: => 101

Please help to resolve this issue.

Thanks.
Naveen

[pgstef]

Hi,

I don't immediately see anything obvious from your config or logs.

You're saying "after completing 5% backup" but the part of the log you're giving let think that it didn't even starting copying anything and was just waiting to check for WAL archives.

So first of all, have look at the PG logs for possible WAL archiving errors (or simply wal generated too fast compared to your current archiving speed)?

And could you retry your backup command with --log-subprocess --log-level-file=debug and give us the complete log generated?

Kind Regards

[dwsteele]

Looks to me like something is breaking the connection between pgBackRest and Postgres. If it is happening after a consistent amount of time, it it probably a firewall or some rule that is severing the connection. We do regular queries to keep the connection open, but that is not always honored by "close anything that has been open too long" rules.

[naveenchowdaryon]

Hi David, Yes, pgbackrest connection is terminated by corn that kills idle connections. After fixing it backup was successful. Thanks & Regards, Naveen Kumar .M, Sr. PostgreSQL Database Administrator, Mobile: 7755929449. *My attitude will always be based on how you treat me. *

…

On Wed, 13 Dec 2023 at 10:50 PM, David Steele ***@***.***> wrote: Looks to me like something is breaking the connection between pgBackRest and Postgres. If it is happening after a consistent amount of time, it it probably a firewall or some rule that is severing the connection. We do regular queries to keep the connection open, but that is not always honored by "close anything that has been open too long" rules. — Reply to this email directly, view it on GitHub <#2243 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKIT6KXYRRWOXVHFXCQCD53YJIBH7AVCNFSM6AAAAABAP22JV6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJUGYYDONJWHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>