Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Invent WALG_PREFETCH_DIR #905

Merged
merged 1 commit into from
Mar 26, 2021
Merged

Invent WALG_PREFETCH_DIR #905

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
5 changes: 5 additions & 0 deletions PostgreSQL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,11 @@ Concurrency values can be configured using:

To configure how many goroutines to use during ```backup-fetch``` and ```wal-fetch```, use `WALG_DOWNLOAD_CONCURRENCY`. By default, WAL-G uses the minimum of the number of files to extract and 10.

* `WALG_PREFETCH_DIR`

By default WAL prefetch is storing prefetched data in pg_wal directory. This ensures that WAL can be easily moved from prefetch location to actual WAL consumption directory. But it may have negative consequences if you use it with pg_rewind in PostgreSQL 13.
PostgreSQL 13 is able to invoke restore_command during pg_rewind. Prefetched WAL can generate false failure of pg_rewind. To avoid it you can either turn off prefetch during rewind (set WALG_DOWNLOAD_CONCURRENCY = 1) or place wal prefetch folder outside PGDATA. For details see [this pgsql-hackers thread](https://postgr.es/m/CAFh8B=kW8yY3yzA1=-w8BT90ejDoELhU+zho7F7k4J6D_6oPFA@mail.gmail.com).

* `WALG_UPLOAD_CONCURRENCY`

To configure how many concurrency streams to use during backup uploading, use `WALG_UPLOAD_CONCURRENCY`. By default, WAL-G uses 16 streams.
Expand Down
2 changes: 2 additions & 0 deletions internal/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ const (
NameStreamCreateCmd = "WALG_STREAM_CREATE_COMMAND"
NameStreamRestoreCmd = "WALG_STREAM_RESTORE_COMMAND"
MaxDelayedSegmentsCount = "WALG_INTEGRITY_MAX_DELAYED_WALS"
PrefetchDir = "WALG_PREFETCH_DIR"

MongoDBUriSetting = "MONGODB_URI"
MongoDBLastWriteUpdateInterval = "MONGODB_LAST_WRITE_UPDATE_INTERVAL"
Expand Down Expand Up @@ -176,6 +177,7 @@ var (
PgSlotName: true,
PgWalSize: true,
"PGPASSFILE": true,
PrefetchDir: true,

// Swift
"WALG_SWIFT_PREFIX": true,
Expand Down
7 changes: 6 additions & 1 deletion internal/wal_fetch_handler.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"encoding/binary"
"encoding/json"
"fmt"
"github.com/spf13/viper"
"io"
"io/ioutil"
"os"
Expand Down Expand Up @@ -51,7 +52,11 @@ func HandleWALFetch(folder storage.Folder, walFileName string, location string,
folder = folder.GetSubFolder(utility.WalPath)
location = utility.ResolveSymlink(location)
if triggerPrefetch {
defer forkPrefetch(walFileName, location)
prefetchLocation := location
if viper.IsSet(PrefetchDir) {
prefetchLocation = viper.GetString(PrefetchDir)
}
defer forkPrefetch(walFileName, prefetchLocation)
}

_, _, running, prefetched := getPrefetchLocations(path.Dir(location), walFileName)
Expand Down