Version 2.2.1

New features:

• WAL Archive catalog. Now it is possible to view the state of every WAL timeline in archive. At this moment the following information can be provided via WAL archive catalog: timeline ID, parent timeline ID, switchpoint LSN, min/max segments, number of segments, resident size, compression ratio, number of backups, lost segments intervals.
  • Option '--archive' allows to view to WAL archive catalog, 'plain' and 'json' formats are supported. Can be used with 'show' command.
• Backup Pinning. Now it is possible to exclude a backup from established retention policy or, in other words, 'pin' it for a specified amount of time by setting 'expire-time' attribute. Already existing backup can be pinned by using new command 'set-backup'. Backup can also be pinned at the moment of creation by providing additional parameters to the 'backup' command.
  • Command 'set-backup' can be used for overwriting backup meta info. For now it can be used only for pinning or unpinning.
  • Option '--ttl' can be used to pin a backup for a specified amount of time. For example, '--ttl=30d' will pin a backup for 30 days. Can be used with 'set-backup' and 'backup' commands.
  • Option '--expire-time' can be used to pin a backup until the specified date. For example, "--expire-time='2020-01-01 00:00:01+03'". Can be used with 'set-backup' and 'backup' commands.
• WAL retention. Now it is possible to set 'a depth' of stored WAL archive measured in backups per timeline.
  • Option '--wal-depth' can be used to determine the number of latest valid backups per timeline that must retain the ability to perform PITR. WAL segments that cannot be applied to this backups will be removed by retention purge. WAL segments that are required by ARCHIVE backup for consistent recovery will be retained. Can be used with 'set-config', 'backup' and 'delete' command.
• Partial restore. Now it is possible to restore only the specified databases or exclude the specified databases from restore using additional options with the 'restore' command. Databases 'template0' and 'template1' are always restored and cannot be explicitly excluded from restore. New options '--db-exclude' and '--db-include' cannot be used together.
  • Option '--db-include' allows to specify a database name to restore. Can be specified multiple times. All other databases will be skipped (with exception of 'template0' and 'template1'). Can be used with 'restore' command.
  • Options '--db-exclude' allows to specify a database name to exclude from restore. Can be specified multiple times. All other databases will be restored. Databases 'template0' and 'template1' cannot be excluded. Can be used with 'restore' command.
• Archive host options. Now it is possible to specify ssh credentials of a host with WAL archive during restore, so the PostgreSQL parameter 'restore_command' can be generated correctly.
  • Option '--archive-host' can be used to set value for --remote-host parameter of archive-get command. Can be used with 'restore' and 'set-config' commands.
  • Option '--archive-port' can be used to set value for --remote-port parameter of 'archive-get' command. Can be used with 'restore' and 'set-config' commands.
  • Option '--archive-user' can be used to set value for --remote-user parameter of 'archive-get' command. Can be used with 'restore' and 'set-config' commands.
  • Option '--restore-command' can be used to set entire 'restore_command'. Can be used with 'restore' and 'set-config' commands.

Improvements:

• '--dry-run' flag is now honored by a WAL purge and a deletion of a specific backup.
• 'archive-push' command now use '.part' suffix for partially copied WAL segment instead of '.partial' to avoid collisions with '.partial' files produced by standby promotion.
• timeline ID now obtained via pg_control_checkpoint() instead of pg_control file.
• backup catalog in 'plain' format now has several changes:
  • column 'Current/Parent TLI' was renamed to 'TLI'
  • new column 'Zratio' contain compression ratio for compressed backups calculated as 'uncompressed-bytes' / 'data_bytes'
  • column 'WAL' was renamed to 'WAL Mode'
  • new column 'WAL' now contain the size of uncompressed WAL segments, required by the backup to reach consistency
• backup meta information now has several new attributes:
  • 'pgdata-size' contain PostgreSQL PGDATA size at the time backup was taken, this attribute can be used to determine the effectiveness of incremental backup
  • 'uncompressed-size' contain the total size of backup data files before compression is applied, this attribute can be used to determine the effectiveness of compression
  • 'expire-time' contain the expiration date if backup is pinned
• documentation has several new sections:
  • Viewing WAL Archive Information
  • Partial Restore
  • Pinning a Backup
  • WAL Retention
  • Howto

Bugfixes:

• WAL purge now has timeline awareness, previously it was possible for WAL purge to unintentionally delete branched child timeline
• merge of compressed backups on multiple threads now works correctly. Reported by Alexander Nikitin
• handling of invalid stop_lsn is improved. Reported by Alexander Nikitin
• SIGTERM and SIGINT are now handled correctly
• now recovery.conf is generated if '--recovery-target=immediate' is used
• after-backup validation now reports corruption corretly. Reported By Yuri Kurenkov
• backup attribute 'recovery-xid' now calculated correctly
• now non-ISO datestyle formats of PostgreSQL cluster are handled correctly
• WAL purge now work correctly if oldest backup has invalid status. Reported by Alex Ignatov
• retention purge now correctly handles invalid backups. Reported by Alex Ignatov
• previously timeout for WAL streaming was always equal to zero, now its value depends on --archive-timeout parameter. Reported by Alex Ignatov
• previously using of --recovery-target-lsn option lead to validation of all backups, now validation of PITR is performed. Reported by Alexey Shishkin
• now only valid backups can be used for PITR and validation of PITR. Reported by Alexey Shishkin
• 'pg_probackup --version' now prints version into stdout instead of stderr

Version 2.1.5

Bugfixes:

• Failed merges do not cause data file corruption anymore and can be successfully re-run. Previously, the 'out of space' error encountered when merging compressed backups could corrupt backup files. Reported by Ilya Skvortsov.
• When taking an archive backup from a standby, pg_probackup now always waits for delivery of WAL segments containing STOP LSN. Failing to do so could make the backup invalid. Reported by Alexander Nikitin.
• Server-side errors occurring during asynchronous execution of the pg_stop_backup() function are now reported correctly instead of being silently ignored. Reported by Alex Ignatov.
• Detecting a system ID mismatch when streaming or reading WAL files is now more thorough. Reported by Rostislav Pochevalov.
• You can now back up files that are not block size aligned. Previously, trying to backup such files could result in an error. Reported by Alex Ignatov.
• Overlapping incremental backup chains are now handled correctly during backup and retention purge. Reported by Rostislav Pochevalov.
• The del-instance command now correctly handles deletion of '.history' files.
• The checkdb command now reports correct schema names of indexes when used in the --amcheck mode.
• Restoring a backup on a remote server now uses correct flags when creating 'recovery.conf' file. Previously, trying to restore a backup on a remote server could result in an error.
• Previously backing up of PostgreSQL 9.5 was possible only under superuser role because of reliance on pg_switch_xlog() function which is superuser-only. Now PostgreSQL 9.5 can be backed up by unprivileged role.
• Detecting of system ID mismatch when streaming or reading WAL files is more thorough now, so receiving of 'alien' WAL segment can be detected as soon as possible.
• Permission mask of non-data files is enforced during restore now, so the original permission mask is not lost.
• Multiple typos are fixed. Reported by Alexander Lakhin.
• Build problems on Windows are fixed.

Improvements:

• Backup metadata is now updated every 10 seconds while the backup is being taken, so you can track the backup completion progress. Previously, the metadata was updated only at the start and at the end of the backup.
• SSH connections are now closed once data file transfer is complete.
• The latest documentation is now available directly on GitHub: https://postgrespro.github.io/pg_probackup

Version 2.1.3

Bugfixes:

• validate was giving a false-positive alarm for files disappeared during backup. Reported by Nikitin Alexander.
• validate was reporting incorrect latest possible recovery target after WAL validation.
• backup command was returning positive exit code even if after-backup validation found corruption. Now in this case backup will exit with an error.
• backup on Windows had a risk of ending with failure if deleted but pinned files were encountered. Fixed in pg_probackup and patch submitted to PostgreSQL community: https://www.postgresql.org/message-id/flat/a9c76882-27c7-9c92-7843-21d5521b70a9%40postgrespro.ru. Reported by Yuri Kurenkov
• restore used with --no-validate could lead to data loss because of lax behavior in case of file been missing. Now missing file is always considered as an error condition for restore.
• merge crash safety improved. Now the MERGING state is preserved until the FULL backup successfully receives a new ID during the merge operation.
• merge of PAGE backup with an external directory had a risk of data loss.
• Previously permission mask of restored files was not preserved and default PostgreSQL permission mask (0700) was enforced, which potentially could lead to inability to backup an instance, that was restored with pg_probackup (for PostgreSQL version >= 11). Reported by Mikhail Kulagin.

Improvements:

• Previously restore of incremental chain was inefficient in case of deleted files: such files were copied and then deleted. Now they are just not copied at all.

Version 2.1.2

Bugfixes:

• do not use find_my_exec(), because it is relying on PATH, which can be unavailable. It could lead to broken archiving on Debian-like distributions if relative path to pg_probackup was used in archive_command. More details: #39

Version 2.1.1

New features:

• Remote backup and restore using SSH. It uses an operation model when backup catalog is located on local machine, while PostgreSQL is located on remote machine, the only exceptions are archive-push and archive-get commands which are always launched from database server side. For remote mode to work a local ssh binary is used. All --remote-* and --ssh-* options can be used with add-instance, set-config, backup, restore, archive-push and archive-get commands. Currently remote operations are not supported on Windows.

  • Option --remote-proto allows to specify a protocol to use for remote operations. Currently only ssh and none values are supported. If option --remote-host is specified then default value for --remote-proto is ssh.
  • Option --remote-host allows to specify remote host IP address or hostname to connect in remote mode.
  • Option --remote-port allows to specify remote host port to connect in remote mode. Default value: 22.
  • Option --remote-user allows to specify remote host user to connect in remote mode. Default value: current user.
  • Option --remote-path allows to specify a directory path with pg_probackup binary on remote machine.
  • Option --ssh-options allows to specify a string of options for ssh. Default: none.

• Merge partially-expired incremental chains during backups retention purge. This is the analog of Oracle Incrementally Updated Backups feature.

  • Flag --merge-expired allows to merge partially-expired incremental backup chains. Can be used only if retention-window policy is set. Can be safely used with --delete-expired flag. Can be used only with backup and delete commands.
  • Flag --dry-run allows to perform a trial retention run without any changes. Can be used only with backup and delete commands.

• Support of external directories. It can be used for adding to backup any files and directories located outside of PGDATA, e.g. configs, logs or scripts.

  • Option --external-dirs allows to specify multiple directories separated by colon that should be copied during backup. Default value: 'none'. Can be used only with set-config and backup commands. Example: --external-dirs=/etc/postgresql/9.6:/opt/scripts.
  • Option --external-mapping similarly to option --tablespace-mapping allows to relocate specified external directory to a new location during restore.Can be specified multiple times. Can be used only with restore command. Example: --external-mapping=/etc/postgresql/9.6=/tmp/configs --external-mapping=/opt/scripts=/tmp/scripts
  • Flag --skip-external-dirs allows to skip restoring of external directories during restore. Can be used only with restore command.

• New command checkdb provides additional facilities to check your PostgreSQL instance. It allows to validate all data files located in PGDATA via block checksums matching and page header sanity checks. Optionally all indexes in all databases in PostgreSQL instance can be logically verified using extensions amcheck or amcheck_next.

  • Flag --amcheck allows logical verification of all indexes in specified PostgreSQL instance if no corruption was found during data files checking. It requires either amcheck or amcheck_next extension to be available. Uses bt_index_check() function from said extensions.
  • Flag --skip-block-validation, which already exists in validate command, completely disables data files validation. Can be used only with --amcheck flag.
  • Flag --heapallindexed allows to check that all heap tuples that should be indexed are actually indexed. Can be used for PG11 or, if amcheck_next extension is installed, for any PG version. Can be used only with --amcheck flag.

• Flag --temp-slot allows user to use temporary replication slot for STREAM backups. Default temp slot name is pg_probackup_slot which can be changed via --slot option.

Improvements:

• Windows support is now considered stable.
• Additional support of threads by validate command. Now it use multiple threads defined by -j option for checking WAL files.
• Flag --no-validate can now be used with backup command. It allows not to force validation after successful backup.
• Now invalid backups will be deleted during retention purge, if they are not guarded by retention.
• Show merge time after backup merging by show command.
• Options --master-* and --replica-timeout are deprecated. Deprecated options are still usable for the purpose of backward compatibility.
• Now an absolute path to pg_probackup binary is used in restore_command option of recovery.conf file generated by restore command.
• Options --time, --xid, --lsn, --timeline and flags --inclusive, --immediate are deprecated. Use --recovery-target-time, --recovery-target-xid, --recovery-target-lsn, --recovery-target-inclusive, --recovery-target-timeline and --recovery-target options instead. Deprecated options and flags are still usable for the purpose of backward compatibility.
• Option --recovery-target has two allowed values: immediate which is identical to deprecated flag --immediate and latest which stands for default recovery behavior - recover to the end of the all available WAL log.
• Usage of parent_link in merge, restore and backup commands instead of time sorting makes incremental chains more robust to time lapses and other anomalies.

Bugfixes:

• Update program_version of the target backup, recalculate CRC of a unchanged file after merge command.
• Changing backup metadata is now atomic.
• During merge command destination backup now inherit WAL mode from parent.
• Allow to delete backups with missing or emtpy .control files.
• More lenient handling of .rotation file been empty or containing garbage, WARNING is issued now instead of ERROR.

Version 2.0.27

- Bugfix: reading of compressed WAL files was very slow, it happened because xlogreader was in habit of reading XLOG page twice, which breaks zlib optimization for sequential access. See https://commitfest.postgresql.org/22/1994 for additional info. Reported by Alex Ignatov.
- Bugfix: previously path to PGDATA was not canonicalized. On Windows this could lead to producing empty backups. Additionally during investigation another bug was found: readdir() silently ignored 'permission denied' errors. See https://www.postgresql.org/message-id/2cad7829-8d66-e39c-b937-ac825db5203d%40postgrespro.ru for additional info. Reported by Yuri Kurenkov.
- Bugfix: archive-push didn`t use O_EXCL flag when creating '.partial' file, in rare case of two PostgreSQL instances concurrently pushing the same WAL segment it could lead to corruption of pushed WAL segment.
- Minor bugfix: disappeared during backup files were marked as 'not changed', now they just evicted from file list.
- Minor bugfix: skip 'log' directory during backup for PG >= 10. Reported by triwada.
- Improvement: previously locking was done at instance level, because of that concurrent operations were impossible, now it is done at the backup level, so restore of backup A won`t interfere with validation of backup B.
- Improvement: previously restore was relying on sort order when restoring incremental backups, now it`s relying on parent chain.
- Minor improvement: when using show command "Time" for RUNNING backups now calculated on the fly.
- Minor improvement: START LSN of backup now written to backup meta right after receiving, not at the end of the backup. It will be needed for "resume" feature.

Version 2.0.26

- Bugfix: fixed undefined behaviour in case of undefined compression algorithm
- Bugfix: correctly calculate backup size after MERGE
- Bugfix: fixed MERGE of compressed and uncompressed backups
- Bugfix: remove unnecessary remaining files after MERGE
- Bugfix: consider target LSN (--lsn parameter) during validating WAL
- Improvement: check backup program_version during VALIDATE, do not support forward compatibility
- Improvement: improve Windows support
- Improvement: improve support of tablespaces within PGDATA

Version 2.0.25

 - Bugfix: changing log-level-file and log-level-console via set-config now works correctly
 - Bugfix: WAL files crc comparison done by archive-push is now works correctly, previously it`s was always giving mismatch
 - Impromevent: PG_STOP_BACKUP_TIMEOUT now governed by archive-timeout option
 - Impromevent: CRC-32c now used to calculate crc for data and WAL files, previously a slower CRC-32 was used
 - Impromevent: cold replica can be backuped without false-positive WAL timeouts
 - Impromevent: for PG >= 10 pg_stop_backup now called without wait_for_archive flag
 - Impromevent: minor fixes for Windows build

Version 2.0.24

        - Major bugfix: incorrect handling of badly compressed blocks, previously there was a risk to restore block in uncompressed state, if compressed size was equal or larger than BLCKSZ
        - Impromevent: backup from replica >= 9.6 no longer need connection to master
        - Workaround: wrong minRecPoint in PostgreSQL thanks to commit 8d68ee6(block from future), overwrite minRecPoint with latest applied LSN
        - Impromevent: merge is now considered stable feature
        - Impromevent: validation now use more conservative and paranoid approach to file validation, during validation pg_probackup also check block checksumm, make sanity check based on block header information and try to detect blocks from future
        - New validate/restore options:
           '--skip-block-validation' - disable aforementioned approach to file validation
        - Multiple minor fixes

Version 2.0.22

- Calculate checksum for backuped files using CRC-32, validate pre 2.0.22 backups using CRC-32C
- Do not backup unchanged _vm,_fsm, pg_clog, pg_multixact, etc files
- Write program-version only during backup
- Minor fixes and bug fixes