ERROR: [082]: WAL segment 0000000100008FEB0000000C was not archived before the 60000ms timeout

[gideonisele]

Please provide the following information when submitting an issue (feature requests or general comments can skip this):

1. pgBackRest version:
   pgBackRest 2.38

2. PostgreSQL version:
   psql (13.6 (Ubuntu 13.6-1.pgdg20.04+1+b1))

3. Operating system/version - if you have more than one server (for example, a database server, a repository host server, one or more standbys), please specify each:
   NAME="Ubuntu"
   VERSION="20.04.4 LTS (Focal Fossa)"

4. Did you install pgBackRest from source or from a package?

I installed packages and dependencies: sudo apt-get install -y pgbackrest

9. Please attach the following as applicable:
   • pgbackrest.conf file(s)

cat /etc/pgbackrest.conf
[global]
backup-host=mothership
backup-user=postgres
repo-path=/var/lib/pgbackrest

[ops4]
pg1-path=/var/lib/pgsql/13/data

- `postgresql.conf` settings applicable to pgBackRest (`archive_command`, `archive_mode`, `listen_addresses`, `max_wal_senders`, `wal_level`, `port`)
- archive_command = 'pgbackrest --stanza=ops4 archive-push %p' 
- archive_mode = on 
- listen_addresses = '*'
- max_wal_senders = 3
- wal_level = replica

- errors in the postgresql log file before or during the time you experienced the issue,

log file

2022-05-16 10:08:49.278 EDT [191045] DETAIL: The failed archive command was: pgbackrest --stanza=ops4 archive-push pg_wal/0000000100008FE9000000FE
2022-05-16 10:08:49.279 EDT [191045] WARNING: archiving write-ahead log file "0000000100008FE9000000FE" failed too many times, will try again later

- log file in `/var/log/pgbackrest` for the commands run (e.g. `/var/log/pgbackrest/mystanza_backup.log`)

10. Describe the issue:

trying to backup to a remote server, but keep getting timeout error with WAL segment 0000000100008FEB0000000C was not archived before the 60000ms timeout.
I configured SSH connection without password on both machine, connection are successful.

pgbackrest --stanza=ops4 --config-path=/etc/pgbackrest --log-level-console=info check
2022-05-16 09:14:49.534 P00 INFO: check command begin 2.38: --compress-level-network=6 --config-path=/etc/pgbackrest --exec-id=18794-f85c57c6 --log-level-console=info --pg1-host=moddbops4 --pg1-host-user=postgres --pg1-path=/var/lib/pgsql/13/data/ --repo1-path=/var/lib/pgbackrest --stanza=ops4
2022-05-16 09:14:51.555 P00 INFO: check repo1 configuration (primary)
2022-05-16 09:14:51.763 P00 INFO: check repo1 archive for WAL (primary)
ERROR: [082]: WAL segment 0000000100008FEB0000000C was not archived before the 60000ms timeout
HINT: check the archive_command to ensure that all options are correct (especially --stanza).
HINT: check the PostgreSQL server log for errors.
HINT: run the 'start' command if the stanza was previously stopped.
2022-05-16 09:15:51.887 P00 INFO: check command end: aborted with exception [082]

I would appreciate a quick help on this issue.

Thanks

[pgstef]

Hi,

[global]
backup-host=mothership
backup-user=postgres
repo-path=/var/lib/pgbackrest

[ops4]
pg1-path=/var/lib/pgsql/13/data

I guess this is the configuration of your PG host. What is the configuration on your repo host (mothership)?

backup-host and backup-user are deprecated options. See https://pgbackrest.org/user-guide.html#repo-host/config.

You'd probably better using:

[global]
repo1-host=mothership
repo1-host-user=postgres
repo1-path=/var/lib/pgbackrest

[ops4]
pg1-path=/var/lib/pgsql/13/data

And on your mothership server:

[global]
repo1-path=/var/lib/pgbackrest

[demo]
pg1-host=your_postgres_host
pg1-path=/var/lib/pgsql/13/data
...

repo1-path must match in both configs. Then, the stanza-create and check must succeed.
You should see more errors in PG logs if you add --log-level-console=detail in your archive_command too.

Kind regards

[gideonisele]

On the storage host the configuration for pgbackrest =>
[ops4]
compress-level=6
compress-level-network=6
process-max=12
pg1-host=moddbops4
pg1-host-user=postgres
pg1-path=/var/lib/pgsql/13/data/

[11:01:10 1001 postgres@mothership ~]$ pgbackrest --stanza=ops4 --config-path=/etc/pgbackrest --log-level-console=detail check
2022-05-16 11:01:57.899 P00 INFO: check command begin 2.38: --compress-level-network=6 --config-path=/etc/pgbackrest --exec-id=12573-36c2c4fb --log-level-console=detail --pg1-host=moddbops4 --pg1-host-user=postgres --pg1-path=/var/lib/pgsql/13/data/ --repo1-path=/var/lib/pgbackrest --stanza=ops4
2022-05-16 11:02:00.029 P00 INFO: check repo1 configuration (primary)
2022-05-16 11:02:00.237 P00 INFO: check repo1 archive for WAL (primary)
ERROR: [082]: WAL segment 0000000100008FEB0000000F was not archived before the 60000ms timeout
HINT: check the archive_command to ensure that all options are correct (especially --stanza).
HINT: check the PostgreSQL server log for errors.
HINT: run the 'start' command if the stanza was previously stopped.
2022-05-16 11:03:00.361 P00 INFO: check command end: aborted with exception [082]

So I'm still getting the error msg after adding "--log-level-console=detail" .

[pgstef]

What's the global part of your config on mothership ?

HINT: check the PostgreSQL server log for errors.

Have you added the option I suggested in your PG archive_command ? And what does the PG logs say ?

[gideonisele]

The global configuration in mothership storage is
[global]
repo-path=/var/lib/pgbackrest
retention-full=3
start-fast=y

PG logs info
/var/lib/pgsql/13/data/log$ cat postgresql-Mon.log

2022-05-16 11:12:00.398 EDT [191045] LOG: archive command failed with exit code 45
2022-05-16 11:12:00.398 EDT [191045] DETAIL: The failed archive command was: pgbackrest --stanza=ops4 archive-push pg_wal/0000000100008FE9000000FE
2022-05-16 11:12:00.398 EDT [191045] WARNING: archiving write-ahead log file "0000000100008FE9000000FE" failed too many times, will try again later

On postgresql.conf
archive_command = 'pgbackrest --stanza=ops4 archive-push %p'

[pgstef]

Have you modified the configuration on the pg host as I suggested ?

[global]
repo1-host=mothership
repo1-host-user=postgres
repo1-path=/var/lib/pgbackrest

[ops4]
pg1-path=/var/lib/pgsql/13/data

And also, set: archive_command = 'pgbackrest --stanza=ops4 --log-level-console=detail archive-push %p'

You should definitively have more output in your PG logs. pgBackRest will push the errors encountered in the PG logs and we'll need those to be able to help you identify the problem.

[gideonisele]

So this is what I'm seeing in my log after setting archive_command = 'pgbackrest --stanza=ops4 --log-level-console=detail archive-push %p' in postgresql.conf file.

ERROR: [045]: WAL file '0000000100008FE9000000FE' already exists in the repo1 archive with a different checksum
2022-05-16 12:13:57.451 P00 INFO: archive-push command end: aborted with exception [045]
2022-05-16 12:13:57.454 EDT [1095763] LOG: archive command failed with exit code 45
2022-05-16 12:13:57.454 EDT [1095763] DETAIL: The failed archive command was: pgbackrest --stanza=ops4 --log-level-console=detail archive-push pg_wal/0000000100008FE9000000FE
2022-05-16 12:13:57.454 EDT [1095763] WARNING: archiving write-ahead log file "0000000100008FE9000000FE" failed too many times, will try again later

[dwsteele]

This error:

ERROR: [045]: WAL file '0000000100008FE9000000FE' already exists in the repo1 archive with a different checksum

means that the 0000000100008FE9000000FE segment already exists in the archive with a different checksum. This is usually bad. It generally means you have split brain or promoted a standby incorrectly and the timeline didn't change.

There is also a bug in older versions of Postgres that could cause this error after a crash, but 13.6 should not have the problem, though the fix is new enough that I can't say for sure.

There's not much to be done here except set archive_command = true and let PG clear out the bad WAL, then immediately make a new backup. Even if you deleted the segment from the archive you would probably not be able to play safely past this point.

[gideonisele]

I was able to resolved the issue based on the suggestion @dwsteele gave: set archive_command = true to clear the bad WAL.
Also cleared the WAL file in the archive 0000000100008FE9000000FE. and the issue was resolved.

Thanks @dwsteele