-
Notifications
You must be signed in to change notification settings - Fork 365
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Feature/patroni release 1 2 3 (#121)
* Some optimisations around restore_command = wal-e 1) wal-e wal-fetch is incredibly slow 2) When we start postgres it will call the restore_command for every xlog segment needed to reach consistent state even if there is a valid file in the pg_xlog dicrectory. 3) Even if the wal-e is prefetching wal segments it it is rather expencive to move them into pg_xlog by wal-e. All these facts are making useless streaming of xlogs with pg_basebackup, because afterwards it will call restore_command anyway :( In order to optimize this process we will stream xlogs not into default pg_xlog but into separate directory and create a wrapper restore_command script, which will: Check if there is xlog already available on disk either in directory created by pg_basebackup or in the pg_xlog/.wal-e/prefetch directory. If the file is there - just move it. If it is not - call wal-e with usual options. * Bump up Patroni and WAL-E versions.
- Loading branch information
1 parent
012d217
commit 1666be9
Showing
4 changed files
with
78 additions
and
8 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| #!/bin/bash | ||
|
|
||
| PARSED=$(getopt --options v --longoptions connstring:,retries:,datadir: --name "$0" -- "$@" 2> /dev/null) | ||
| eval set -- "$PARSED" | ||
|
|
||
| RETRIES=2 | ||
|
|
||
| while [[ $# -gt 0 ]]; do | ||
| case $1 in | ||
| --datadir ) | ||
| DATA_DIR=$2 | ||
| shift | ||
| ;; | ||
| --connstring ) | ||
| CONNSTR=$2 | ||
| shift | ||
| ;; | ||
| --retries ) | ||
| RETRIES=$2 | ||
| shift | ||
| ;; | ||
| * ) | ||
| ;; | ||
| esac | ||
| shift | ||
| done | ||
|
|
||
| [[ -z $DATA_DIR || -z $CONNSTR || ! $RETRIES =~ ^[1-9]$ ]] && exit 1 | ||
|
|
||
| ATTEMPT=0 | ||
| while [[ $((ATTEMPT++)) -le $RETRIES ]]; do | ||
| rm -fr "${DATA_DIR}" | ||
| pg_basebackup --pgdata="${DATA_DIR}" --xlog-method=stream --dbname="${CONNSTR}" | ||
| EXITCODE=$? | ||
| if [[ $EXITCODE == 0 ]]; then | ||
| XLOG_FAST=$(dirname $DATA_DIR)/xlog_fast | ||
| rm -fr $XLOG_FAST | ||
| mv ${DATA_DIR}/pg_xlog $XLOG_FAST | ||
| rm -fr $XLOG_FAST/archive_status | ||
| mkdir ${DATA_DIR}/pg_xlog | ||
| break | ||
| elif [[ $ATTEMPT -le $RETRIES ]]; then | ||
| sleep $((ATTEMPT*10)) | ||
| fi | ||
| done | ||
| exit $EXITCODE |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| #!/bin/bash | ||
|
|
||
| readonly xlog_filename=$1 | ||
| readonly xlog_destination=$2 | ||
|
|
||
| [[ -z $xlog_filename || -z $xlog_destination ]] && exit 1 | ||
|
|
||
| readonly xlog_dir=$(dirname $xlog_destination) | ||
| readonly xlog_fast_source=$(dirname $(dirname $(realpath $xlog_dir)))/xlog_fast/$xlog_filename | ||
| readonly wale_prefetch_source=${xlog_dir}/.wal-e/prefetch/${xlog_filename} | ||
|
|
||
| if [[ -f $xlog_fast_source ]]; then | ||
| exec mv "${xlog_fast_source}" "${xlog_destination}" | ||
| elif [[ -f $wale_prefetch_source ]]; then | ||
| exec mv "${wale_prefetch_source}" "${xlog_destination}" | ||
| else | ||
| exec wal-e --aws-instance-profile wal-fetch "${xlog_filename}" "${xlog_destination}" | ||
| fi |