Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add --until-binlog-last-modified-time option to `wal-g-mysql binlog… #1154

Merged
merged 3 commits into from
Nov 22, 2021
Merged

Add --until-binlog-last-modified-time option to `wal-g-mysql binlog… #1154

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
92b5c7d
Add `--until-binlog-last-modified-time` option to `wal-g-mysql binlog…
ostinru Nov 19, 2021
6d9c7da
fix copy-paste error
ostinru Nov 19, 2021
1d6b0e3
Hakuna Matata!
ostinru Nov 19, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
9 changes: 8 additions & 1 deletion cmd/mysql/binlog_fetch.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,12 @@ import (

const fetchSinceFlagShortDescr = "backup name starting from which you want to fetch binlogs"
const fetchUntilFlagShortDescr = "time in RFC3339 for PITR"
const fetchUntilBinlogLastModifiedFlagShortDescr = "time in RFC3339 that is used to prevent wal-g from replaying" +
" binlogs that was created/modified after this time"

var fetchBackupName string
var fetchUntilTS string
var fetchUntilBinlogLastModifiedTS string

// binlogPushCmd represents the cron command
var binlogFetchCmd = &cobra.Command{
Expand All @@ -24,7 +27,7 @@ var binlogFetchCmd = &cobra.Command{
Run: func(cmd *cobra.Command, args []string) {
folder, err := internal.ConfigureFolder()
tracelog.ErrorLogger.FatalOnError(err)
mysql.HandleBinlogFetch(folder, fetchBackupName, fetchUntilTS)
mysql.HandleBinlogFetch(folder, fetchBackupName, fetchUntilTS, fetchUntilBinlogLastModifiedTS)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the difference between fetchUntilTS and fetchUntilBinlogLastModifiedTS ?
can we use fetchUntilTs instead of fetchUntilBinlogLastModifiedTS

Copy link
Contributor Author

@ostinru ostinru Nov 21, 2021 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here I am introducing new should distinguish two timestamps:
fetchUntilTS - is about time of individual transactions
fetchUntilBinlogLastModifiedTS - is about time when binlog was created

There are 2 properties:
fetchUntilTS < fetchUntilBinlogLastModifiedTS - because transactions dumped to binlog before binlog upload
--stop-datetime < fetchUntilBinlogLastModifiedTS - it is new rule that fixes multi-host restores (check detailed explanation in another comment)

},
PersistentPreRun: func(cmd *cobra.Command, args []string) {
internal.RequiredSettings[internal.MysqlBinlogDstSetting] = true
Expand All @@ -39,5 +42,9 @@ func init() {
"until",
utility.TimeNowCrossPlatformUTC().Format(time.RFC3339),
fetchUntilFlagShortDescr)
binlogFetchCmd.PersistentFlags().StringVar(&fetchUntilBinlogLastModifiedTS,
"until-binlog-last-modified-time",
"",
fetchUntilBinlogLastModifiedFlagShortDescr)
cmd.AddCommand(binlogFetchCmd)
}
7 changes: 6 additions & 1 deletion cmd/mysql/binlog_replay.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,12 @@ import (

const replaySinceFlagShortDescr = "backup name starting from which you want to fetch binlogs"
const replayUntilFlagShortDescr = "time in RFC3339 for PITR"
const replayUntilBinlogLastModifiedFlagShortDescr = "time in RFC3339 that is used to prevent wal-g from replaying" +
" binlogs that was created/modified after this time"

var replayBackupName string
var replayUntilTS string
var replayUntilBinlogLastModifiedTS string

var binlogReplayCmd = &cobra.Command{
Use: "binlog-replay",
Expand All @@ -23,7 +26,7 @@ var binlogReplayCmd = &cobra.Command{
Run: func(cmd *cobra.Command, args []string) {
folder, err := internal.ConfigureFolder()
tracelog.ErrorLogger.FatalOnError(err)
mysql.HandleBinlogReplay(folder, replayBackupName, replayUntilTS)
mysql.HandleBinlogReplay(folder, replayBackupName, replayUntilTS, replayUntilBinlogLastModifiedTS)
},
PersistentPreRun: func(cmd *cobra.Command, args []string) {
internal.RequiredSettings[internal.MysqlBinlogReplayCmd] = true
Expand All @@ -36,5 +39,7 @@ func init() {
binlogReplayCmd.PersistentFlags().StringVar(&replayBackupName, "since", "LATEST", replaySinceFlagShortDescr)
binlogReplayCmd.PersistentFlags().StringVar(&replayUntilTS, "until",
utility.TimeNowCrossPlatformUTC().Format(time.RFC3339), replayUntilFlagShortDescr)
binlogReplayCmd.PersistentFlags().StringVar(&replayUntilBinlogLastModifiedTS, "until-binlog-last-modified-time",
"", replayUntilBinlogLastModifiedFlagShortDescr)
cmd.AddCommand(binlogReplayCmd)
}
13 changes: 13 additions & 0 deletions docs/MySQL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,13 @@ or
wal-g binlog-fetch --since LATEST --until "2006-01-02T15:04:05Z07:00"
```

You can stop wal-g from fetching newly created/modified binlogs by specifying `--until-binlog-last-modified-time` option.
This may be useful to achieve exact clones of the same database in scenarios when new binlogs are uploaded concurrently whith your restore process.

```bash
wal-g binlog-replay --since LATEST --until "2006-01-02T15:04:05Z07:00" --until-binlog-last-modified-time "2006-01-02T15:04:05Z07:00"
```

### ``binlog-replay``

Fetches binlogs from storage and passes them to `WALG_MYSQL_BINLOG_REPLAY_COMMAND` to replay on running MySQL server.
Expand All @@ -161,6 +168,12 @@ or
wal-g binlog-replay --since LATEST --until "2006-01-02T15:04:05Z07:00"
```

You can stop wal-g from applying newly created/modified binlogs by specifying `--until-binlog-last-modified-time` option.
This may be useful to achieve exact clones of the same database in scenarios when new binlogs are uploaded concurrently whith your restore process.

```bash
wal-g binlog-replay --since LATEST --until "2006-01-02T15:04:05Z07:00" --until-binlog-last-modified-time "2006-01-02T15:04:05Z07:00"
```

Typical configurations
-----
Expand Down
6 changes: 3 additions & 3 deletions internal/databases/mysql/binlog_fetch_handler.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,17 +41,17 @@ func (ih *indexHandler) createIndexFile() error {
return nil
}

func HandleBinlogFetch(folder storage.Folder, backupName string, untilTS string) {
func HandleBinlogFetch(folder storage.Folder, backupName string, untilTS string, untilBinlogLastModifiedTS string) {
dstDir, err := internal.GetLogsDstSettings(internal.MysqlBinlogDstSetting)
tracelog.ErrorLogger.FatalOnError(err)

startTS, endTS, err := getTimestamps(folder, backupName, untilTS)
startTS, endTS, endBinlogTS, err := getTimestamps(folder, backupName, untilTS, untilBinlogLastModifiedTS)
tracelog.ErrorLogger.FatalOnError(err)

handler := newIndexHandler(dstDir)

tracelog.InfoLogger.Printf("Fetching binlogs since %s until %s", startTS, endTS)
err = fetchLogs(folder, dstDir, startTS, endTS, handler)
err = fetchLogs(folder, dstDir, startTS, endTS, endBinlogTS, handler)
tracelog.ErrorLogger.FatalfOnError("Failed to fetch binlogs: %v", err)

err = handler.createIndexFile()
Expand Down
21 changes: 13 additions & 8 deletions internal/databases/mysql/binlog_replay_handler.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,37 +72,42 @@ func (rh *replayHandler) handleBinlog(binlogPath string) error {
}
}

func HandleBinlogReplay(folder storage.Folder, backupName string, untilTS string) {
func HandleBinlogReplay(folder storage.Folder, backupName string, untilTS string, untilBinlogLastModifiedTS string) {
dstDir, err := internal.GetLogsDstSettings(internal.MysqlBinlogDstSetting)
tracelog.ErrorLogger.FatalOnError(err)

startTS, endTS, err := getTimestamps(folder, backupName, untilTS)
startTS, endTS, endBinlogTS, err := getTimestamps(folder, backupName, untilTS, untilBinlogLastModifiedTS)
tracelog.ErrorLogger.FatalOnError(err)

handler := newReplayHandler(endTS)

tracelog.InfoLogger.Printf("Fetching binlogs since %s until %s", startTS, endTS)
err = fetchLogs(folder, dstDir, startTS, endTS, handler)
err = fetchLogs(folder, dstDir, startTS, endTS, endBinlogTS, handler)
tracelog.ErrorLogger.FatalfOnError("Failed to fetch binlogs: %v", err)

err = handler.wait()
tracelog.ErrorLogger.FatalfOnError("Failed to apply binlogs: %v", err)
}

func getTimestamps(folder storage.Folder, backupName, untilTS string) (time.Time, time.Time, error) {
func getTimestamps(folder storage.Folder, backupName, untilTS, untilBinlogLastModifiedTS string) (time.Time, time.Time, time.Time, error) {
backup, err := internal.GetBackupByName(backupName, utility.BaseBackupPath, folder)
if err != nil {
return time.Time{}, time.Time{}, errors.Wrap(err, "Unable to get backup")
return time.Time{}, time.Time{}, time.Time{}, errors.Wrap(err, "Unable to get backup")
}

startTS, err := getBinlogSinceTS(folder, backup)
if err != nil {
return time.Time{}, time.Time{}, err
return time.Time{}, time.Time{}, time.Time{}, err
}

endTS, err := utility.ParseUntilTS(untilTS)
if err != nil {
return time.Time{}, time.Time{}, err
return time.Time{}, time.Time{}, time.Time{}, err
}
return startTS, endTS, nil

endBinlogTS, err := utility.ParseUntilTS(untilBinlogLastModifiedTS)
if err != nil {
return time.Time{}, time.Time{}, time.Time{}, err
}
return startTS, endTS, endBinlogTS, nil
}
9 changes: 6 additions & 3 deletions internal/databases/mysql/mysql.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,12 +184,12 @@ type binlogHandler interface {
handleBinlog(binlogPath string) error
}

func fetchLogs(folder storage.Folder, dstDir string, startTS time.Time, endTS time.Time, handler binlogHandler) error {
func fetchLogs(folder storage.Folder, dstDir string, startTS, endTS, endBinlogTS time.Time, handler binlogHandler) error {
logFolder := folder.GetSubFolder(BinlogPath)
includeStart := true
outer:
for {
logsToFetch, err := getLogsCoveringInterval(logFolder, startTS, includeStart)
logsToFetch, err := getLogsCoveringInterval(logFolder, startTS, includeStart, endBinlogTS)
includeStart = false
if err != nil {
return err
Expand Down Expand Up @@ -261,7 +261,7 @@ func getBinlogSinceTS(folder storage.Folder, backup internal.Backup) (time.Time,
}

// getLogsCoveringInterval lists the operation logs that cover the interval
func getLogsCoveringInterval(folder storage.Folder, start time.Time, includeStart bool) ([]storage.Object, error) {
func getLogsCoveringInterval(folder storage.Folder, start time.Time, includeStart bool, endBinlogTS time.Time) ([]storage.Object, error) {
logFiles, _, err := folder.ListFolder()
if err != nil {
return nil, err
Expand All @@ -271,6 +271,9 @@ func getLogsCoveringInterval(folder storage.Folder, start time.Time, includeStar
})
var logsToFetch []storage.Object
for _, logFile := range logFiles {
if logFile.GetLastModified().After(endBinlogTS) {
continue // don't fetch binlogs from future
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shold not affect the final result, as mysqbinlog started as

mysqlbinlog --stop-datetime="$WALG_MYSQL_BINLOG_END_TS"

and should not replay transactions happened after fetchUntilTS

Copy link
Contributor Author

@ostinru ostinru Nov 21, 2021 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This patch fixes following issue:

Live MySQL cluster A constantly uploads binlogs. We are trying to restore new cluster (let's call it B) from S3 on some point in time, e.g. NOW() (or future time).
Then we can have following scenario:
'Fast' servers restore MySQL on Snapshot + binlog.123 (Binlog 123 is the latest observed binlog for 'fast' hosts).
'Slow' servers may restore MySQL on Snapshot+binlog.124 (Because cluster A have uploaded one more binlog during 'slow' restore).
As a result we have different GTIDs in a cluster. It is easier to prevent such scenario rather than fixing it afterwards.

The root cause of this issue is simple: newly uploaded binlog file may contain transactions that happend before user-specified --stop-datetime (it is arbitrary time, not a "$WALG_MYSQL_BINLOG_END_TS"). I can imagine scenario when binlog is uploaded once a day (after calling FLUSH BINARY LOGS). In this scenario for any user-specified timestamp from that day we will have transactions in new binlog.

To prevent different GTID-sets we should:

  • point in time for this limits should be NOW() or in past.
  • introduce hard limits on which binlogs we are replaying

This 2 rules guarantees that all restored clones will have same binlog files replayed and as a result - same GTID-sets.

if start.Before(logFile.GetLastModified()) || includeStart && start.Equal(logFile.GetLastModified()) {
logsToFetch = append(logsToFetch, logFile)
}
Expand Down