Skip to content

Add documentation for replication client#579

Merged
morgo merged 4 commits intoblock:mainfrom
morgo:mtocker-document-repl-client
Jan 29, 2026
Merged

Add documentation for replication client#579
morgo merged 4 commits intoblock:mainfrom
morgo:mtocker-document-repl-client

Conversation

@morgo
Copy link
Collaborator

@morgo morgo commented Jan 27, 2026

A Pull Request should be associated with an Issue.

We wish to have discussions in Issues. A single issue may be targeted by multiple PRs.
If you're offering a new feature or fixing anything, we'd like to know beforehand in Issues,
and potentially we'll be able to point development in a particular direction.
Further notes in https://github.com/block/spirit/blob/main/.github/CONTRIBUTING.md

Provides documentation for the replication client package.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new README documenting how the pkg/repl replication client works and how its subscription implementations handle binlog deltas during migrations.

Changes:

  • Introduces pkg/repl/README.md explaining the replication client’s purpose and architecture.
  • Documents the three subscription implementations (delta map, delta queue, buffered map) with tradeoffs and examples.
  • Describes key operational behaviors: watermark optimization, checkpointing, and cutover coordination.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

// Safe to cutover now
```

The `client.Flush()` will retry in a loop until the number of pending changes is considered trivial (currently <10K). It is important to handle errors correctly here, because `FlushUnderTableLock` may fail if it can't flush the pending changes fast enough. This is your cue to abandon the cutover operation for now, and try again when the server is under less load.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there force-cutover behavior that can help here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Force cutover applies to the lock acquisition. This threshold is about once we have the lock, what we can reasonably do before we decide that we're taking too long.

The repl.FlushUnderTableLock() call requires that you have to have pre-created the lock, which is done here:

tableLock, err := dbconn.NewTableLock(ctx, c.db, tablesToLock, c.dbConfig, c.logger)
if err != nil {
return err
}
defer utils.CloseAndLogWithContext(ctx, tableLock)
if err := c.feed.FlushUnderTableLock(ctx, tableLock); err != nil {
return err
}

(This is where force / non-force matters).

@morgo morgo enabled auto-merge January 29, 2026 18:36
@morgo morgo merged commit 8ea7311 into block:main Jan 29, 2026
7 checks passed
@morgo morgo deleted the mtocker-document-repl-client branch January 29, 2026 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants