Logical Loops

Logical Loops

NOTE: This page is a Work in Progress

The logical package ( GoDoc ) contains the core functionality for building a logical-replication pipeline. Product-specific functionality is encapsulated within a Dialect implementation that leaves lifecycle, state, and error handling to the common logical code.

The general design of the logical package assumes that given integration will advance from one "consistent point" to another as part of consuming some stream of replication events. That is, there is a source of data (e.g. transaction log or queryable API) where the consistent point describes some partial progress.

In the simplest case, a Dialect provides two services to the loop: A source of events, Dialect.ReadInto(), and a consumer of those events, Dialect.Process(). The ReadInto() and Process() methods are connected via a loop-provided channel.

API

• Dialect.ReadInto(): This method will query the replication source, starting from the consistent point, in order to send dialect-specific events into the provided channel. If ReadInto() exits with an error, the loop will restart it after injecting a rollback message into the channel.
• Dialect.Process(): This method will read from the loop-provided channel to call Events.OnBegin() to provide batches of data to be committed to the target. The Process() method may call State.SetConsistentPoint() from time to time in order to pass data to ReadInto() during restarts. If Process() exits, the loop will restart.
• Backfiller.BackfillInto(): An optional capability that allows the Dialect to support backfilling mode.
• Lessor.Acquire(): An optional capability that requires the logical loop to hold a (global) lease in order to proceed. This is used when exactly one instance of a loop should be run at a time.

Topics:

• Backfill
• Error handling
• SetConsistentPoint
• Leasing
• Events Pipeline
• Mutation sorting
• Lifecycle / shutdown