Skip to content

Commit

Permalink
DBZ-7699 Document new snapshot modes for Informix connector
Browse files Browse the repository at this point in the history
  • Loading branch information
@mfvitale
mfvitale committed Mar 28, 2024
1 parent 0ff787f commit 9cdaa38
Showing 1 changed file with 165 additions and 10 deletions.
175 changes: 165 additions & 10 deletions documentation/modules/ROOT/pages/connectors/informix.adoc
View file
Expand Up @@ -97,6 +97,7 @@ As a result, the {prodname} Informix connector cannot retrieve the entire histor
To enable the connector to establish a baseline for the current state of the database, the first time that the connector starts, it performs an initial _consistent snapshot_ of the tables that are in capture mode.
For each change that the snapshot captures, the connector emits a `read` event to the Kafka topic for the captured table.

[[default-workflow-for-performing-an-initial-snapshot]]
.Default workflow that the {prodname} Informix connector uses to perform an initial snapshot

The following workflow lists the steps that {prodname} takes to create a snapshot.
Expand Down Expand Up @@ -157,6 +158,48 @@ After the connector completes the initial snapshot, it continues streaming from

If the connector stops again for any reason, after it restarts, it resumes streaming changes from where it previously left off.

[id="informix-connector-snapshot-mode-options"]
.Settings for `snapshot.mode` connector configuration property
[cols="30%a,70%a",options="header"]
|===
|Setting |Description
|`always`
|The connector performs a snapshot every time that it starts.
After the snapshot completes, the connector begins to stream event records for subsequent database changes.
|`initial`
|The connector performs a database snapshot as described in the xref:default-workflow-for-performing-an-initial-snapshot[default workflow for creating an initial snapshot].
After the snapshot completes, the connector begins to stream event records for subsequent database changes.
|`initial_only`
|The connector performs a database snapshot.
After the snapshot completes, the connector stops, and does not stream event records for subsequent database changes.
|`schema_only`
|Deprecated, see `no_data`.
|`no_data`
|The connector captures the structure of all relevant tables, performing all the steps described in the xref:default-workflow-for-performing-an-initial-snapshot[default snapshot workflow], except that it does not create `READ` events to represent the data set at the point of the connector's start-up (Step 7.b).
|`recovery`
|Set this option to restore a database schema history topic that is lost or corrupted.
After a restart, the connector runs a snapshot that rebuilds the topic from the source tables.
You can also set the property to periodically prune a database schema history topic that experiences unexpected growth. +
+
WARNING: Do not use this mode to perform a snapshot if schema changes were committed to the database after the last connector shutdown.
|`when_needed`
|After the connector starts, it performs a snapshot only if it detects one of the following circumstances:
* It cannot detect any topic offsets.
* A previously recorded offset specifies a log position that is not available on the server.

ifdef::community[]
|`configuration_based`
|Set the snapshot mode to `configuration_based` to control snapshot behavior through the set of connector properties that have the prefix 'snapshot.mode.configuration.based'.
endif::community[]

ifdef::community[]
|`custom`
|The `custom` snapshot mode lets you inject your own implementation of the `io.debezium.spi.snapshot.Snapshotter` interface.
endif::community[]

|===


// ModuleID: informix-description-of-why-initial-snapshots-capture-the-schema-history-for-all-tables
// Title: Description of why initial snapshots capture the schema history for all tables
// Type: concept
Expand Down Expand Up @@ -357,9 +400,15 @@ include::{partialsdir}/modules/all-connectors/proc-stopping-an-incremental-snaps

include::{partialsdir}/modules/all-connectors/proc-stopping-an-incremental-snapshot-kafka.adoc[leveloffset=+1]

ifdef::community[]
[[connector-custom-snapshot]]
=== Custom snapshotter SPI
include::{partialsdir}/modules/all-connectors/custom-snapshotter-spi.adoc[leveloffset=+3]
endif::community[]

// Type: concept
[id="informix-blocking-snapshots"]
==== Blocking snapshots
=== Blocking snapshots

include::{partialsdir}/modules/all-connectors/con-connector-blocking-snapshot.adoc[leveloffset=+3]

Expand Down Expand Up @@ -2208,18 +2257,124 @@ For example, +
isbn.schema.name: io.debezium.informix.type.Isbn

|[[informix-property-snapshot-mode]]<<informix-property-snapshot-mode, `+snapshot.mode+`>>
|`initial`
|Specifies the criteria for performing a snapshot when the connector starts. +
Specify one of the following values:
|_initial_
|Specifies the criteria for performing a snapshot when the connector starts: +

`always`:: The connector performs a snapshot every time that it starts.
The snapshot includes the structure and data of the captured tables.
Specify this value to populate topics with a complete representation of the data from the captured tables every time that the connector starts.
After the snapshot completes, the connector begins to stream event records for subsequent database changes.
`initial`:: The connector performs a database snapshot as described in the xref:default-workflow-for-performing-an-initial-snapshot[default workflow for creating an initial snapshot].
After the snapshot completes, the connector begins to stream event records for subsequent database changes.
`initial_only`:: The connector performs a database a snapshot only when no offsets have been recorded for the logical server name.
After the snapshot completes, the connector stops.
It does not transition to streaming event records for subsequent database changes.
`schema_only`:: Deprecated, see `no_data`.
`no_data`:: The connector runs a snapshot that captures the structure of all relevant tables, performing all the steps described in the xref:default-workflow-for-performing-an-initial-snapshot[default snapshot workflow], except that it does not create `READ` events to represent the data set at the point of the connector's start-up (Step 7.b).
`recovery`:: Set this option to restore a database schema history topic that is lost or corrupted.
After a restart, the connector runs a snapshot that rebuilds the topic from the source tables.
You can also set the property to periodically prune a database schema history topic that experiences unexpected growth. +
+
WARNING: Do not use this mode to perform a snapshot if schema changes were committed to the database after the last connector shutdown.
`when_needed`:: After the connector starts, it performs a snapshot only if it detects one of the following circumstances:
* It cannot detect any topic offsets.
* A previously recorded offset specifies a log position that is not available on the server.

ifdef::community[]
`configuration_based`:: With this option, you control snapshot behavior through a set of connector properties that have the prefix 'snapshot.mode.configuration.based'.

endif::community[]

ifdef::community[]
`custom`:: The `custom` snapshot mode lets you inject your own implementation of the `io.debezium.spi.snapshot.Snapshotter` interface.
Set the `snapshot.mode.custom.name` configuration property to the name provided by the `name()` method of your implementation.

For more information, see xref:connector-custom-snapshot[custom snapshotter SPI].
endif::community[]

`initial`:: For tables in capture mode, the connector takes a snapshot of the schema for the table and the data in the table, and then transitions to streaming data for subsequent changes.
This option is useful for populating Kafka topics with a complete representation of the existing table data.
ifdef::community[]
|[[informix-property-snapshot-mode-configuration-based-snapshot-data]]<<informix-property-configuration-based-snapshot-data, `+snapshot.mode.configuration.based.snapshot.data+`>>
|false
|If the `snapshot.mode` is set to `configuration_based`, set this property to specify whether the connector includes table data when it performs a snapshot.
endif::community[]

`initial_only`:: Takes a snapshot of structure and data as with the `initial` option, but does not transition to streaming changes after the snapshot completes.
ifdef::community[]
|[[informix-property-snapshot-mode-configuration-based-snapshot-schema]]<<informix-property-configuration-based-snapshot-schema, `+snapshot.mode.configuration.based.snapshot.schema+`>>
|false
|If the `snapshot.mode` is set to `configuration_based`, set this property to specify whether the connector includes the table schema when it performs a snapshot.
endif::community[]

`schema_only`:: For tables in capture mode, the connector takes a snapshot of only the schema for the table.
This is useful when you are not interested in capturing past data and want the connector to emit to Kafka topics only the changes that happen after the current time.
After the snapshot is complete, the connector continues by reading change events from the database's redo logs.
ifdef::community[]
|[[informix-property-snapshot-mode-configuration-based-start-stream]]<<informix-property-configuration-based-start-stream, `+snapshot.mode.configuration.based.start.stream+`>>
|false
|If the `snapshot.mode` is set to `configuration_based`, set this property to specify whether the connector begins to stream change events after a snapshot completes.
endif::community[]

ifdef::community[]
|[[informix-property-snapshot-mode-configuration-based-snapshot-on-schema-error]]<<informix-property-configuration-based-snapshot-on-schema-error, `+snapshot.mode.configuration.based.snapshot.on.schema.error+`>>
|false
|If the `snapshot.mode` is set to `configuration_based`, set this property to specify whether the connector includes table schema in a snapshot if the schema history topic is not available.
endif::community[]

ifdef::community[]
|[[informix-property-snapshot-mode-configuration-based-snapshot-on-data-error]]<<informix-property-configuration-based-snapshot-on-data-error, `+snapshot.mode.configuration.based.snapshot.on.data.error+`>>
|false
|If the `snapshot.mode` is set to `configuration_based`, this property specifies whether the connector attempts to snapshot table data if it does not find the last committed offset in the transaction log. +
Set the value to `true` to instruct the connector to perform a new snapshot.
endif::community[]

ifdef::community[]
|[[informix-property-snapshot-mode-custom-name]]<<informix-property-snapshot-mode-custom-name, `+snapshot.mode.custom.name+`>>
|No default
| If `snapshot.mode` is set to `custom`, use this setting to specify the name of the custom implementation that is provided in the `name()` method that is defined in the 'io.debezium.spi.snapshot.Snapshotter' interface.
After a connector restart, {prodname} calls the specified custom implementation to determine whether to perform a snapshot.
For more information, see xref:connector-custom-snapshot[custom snapshotter SPI].
endif::community[]

|[[informix-property-snapshot-locking-mode]]<<informix-property-snapshot-locking-mode, `+snapshot.locking.mode+`>>
|_exclusive_
a|Controls whether and for how long the connector holds a table lock.
Table locks prevent other database clients from performing certain table operations during a snapshot.
You can set the following values:

`exclusive`:: Controls how the connector holds locks on tables while performing the schema snapshot when `snapshot.isolation.mode` is `REPEATABLE_READ` or `EXCLUSIVE`. +
The connector holds a table lock that ensures exclusive table access during only the initial phase of the snapshot in which the connector reads the database schema and other metadata.
In subsequent phases of the snapshot, the connector uses a flashback query, which requires no locks, to select all rows from each table.

`share`:: Controls how the connector holds locks on tables while performing the schema snapshot when `snapshot.isolation.mode` is `REPEATABLE_READ` or `EXCLUSIVE`. +
The connector holds a read table lock that ensures read table access during only the initial phase of the snapshot in which the connector reads the database schema and other metadata.
In subsequent phases of the snapshot, the connector uses a flashback query, which requires no locks, to select all rows from each table.

ifdef::community[]
`custom`:: The connector performs a snapshot according to the implementation specified by the xref:informix-property-snapshot-locking-mode-custom-name[`snapshot.locking.mode.custom.name`] property, which is a custom implementation of the `io.debezium.spi.snapshot.SnapshotLock` interface.
endif::community[]

ifdef::community[]
|[[informix-property-snapshot-locking-mode-custom-name]]<<informix-property-snapshot-locking-mode-custom-name, `+snapshot.locking.mode.custom.name+`>>
|No default
| When `snapshot.locking.mode` is set to `custom`, use this setting to specify the name of the custom locking implementation provided in the `name()` method that is defined by the 'io.debezium.spi.snapshot.SnapshotLock' interface.
For more information, see xref:connector-custom-snapshot[custom snapshotter SPI].
endif::community[]

|[[informix-property-snapshot-query-mode]]<<informix-property-snapshot-query-mode, `+snapshot.query.mode+`>>
|`select_all`
|Specifies how the connector queries data while performing a snapshot. +
Set one of the following options:

`select_all`:: The connector performs a `select all` query by default, optionally adjusting the columns selected based on the column include and exclude list configurations.

ifdef::community[]
`custom`:: The connector performs a snapshot query according to the implementation specified by the xref:informix-property-snapshot-snapshot-query-mode-custom-name[`snapshot.query.mode.custom.name`] property, which defines a custom implementation of the `io.debezium.spi.snapshot.SnapshotQuery` interface. +
endif::community[]

This setting enables you to manage snapshot content in a more flexible manner compared to using the xref:informix-property-snapshot-select-statement-overrides[`snapshot.select.statement.overrides`] property.

ifdef::community[]
|[[informix-property-snapshot-snapshot-query-mode-custom-name]]<<informix-property-snapshot-query-mode-custom-name, `+snapshot.query.mode.custom.name+`>>
|No default
| When xref:informix-property-snapshot-query-mode[`snapshot.query.mode`] is set to `custom`, use this setting to specify the name of the custom query implementation provided in the `name()` method that is defined by the 'io.debezium.spi.snapshot.SnapshotQuery' interface.
For more information, see xref:connector-custom-snapshot[custom snapshotter SPI].
endif::community[]

|[[informix-property-snapshot-isolation-mode]]<<informix-property-snapshot-isolation-mode, `+snapshot.isolation.mode+`>>
|`repeatable_read`
Expand Down

0 comments on commit 9cdaa38

Please sign in to comment.