Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DBZ-4588 restructure custom converters doc #3487

Merged
26 changes: 26 additions & 0 deletions documentation/modules/ROOT/pages/connectors/cassandra.adoc
Expand Up @@ -751,6 +751,8 @@ The following table describes how the connector maps each of the Cassandra data

**TODO**: add logical types

If the default data type conversions do not meet your needs, you can {link-prefix}:{link-custom-converters}#custom-converters[create a custom converter] for the connector.

[[cassandra-when-things-go-wrong]]
=== When Things Go Wrong

Expand Down Expand Up @@ -960,6 +962,30 @@ Cassandra connector has built-in support for JMX metrics. The Cassandra driver a
|false
|Determines whether or not the CommitLogProcessor should re-process error commit logs.

|[[cassandra-property-converters]]<<cassandra-property-converters, `converters`>>
|No default
|Enumerates a comma-separated list of the symbolic names of the {link-prefix}:{link-custom-converters}#custom-converters[custom converter] instances that the connector can use.
For example, +

`isbn`

You must set the `converters` property to enable the connector to use a custom converter.

For each converter that you configure for a connector, you must also add a `.type` property, which specifies the fully-qualifed name of the class that implements the converter interface.
The `.type` property uses the following format: +

`_<converterSymbolicName>_.type` +

For example, +

isbn.type: io.debezium.test.IsbnConverter

If you want to further control the behavior of a configured converter, you can add one or more configuration parameters to pass values to the converter.
To associate any additional configuration parameter with a converter, prefix the parameter names with the symbolic name of the converter.
For example, +

isbn.schema.name: io.debezium.cassandra.type.Isbn

|[[cassandra-property-offset-backing-store-dir]]<<cassandra-property-offset-backing-store-dir, `offset.backing.store.dir`>>
|
|The directory to store offset tracking files.
Expand Down
25 changes: 25 additions & 0 deletions documentation/modules/ROOT/pages/connectors/db2.adoc
Expand Up @@ -1102,6 +1102,7 @@ When a row is deleted, the _delete_ event value still works with log compaction,
Db2's data types are described in https://www.ibm.com/support/knowledgecenter/en/SSEPGG_11.5.0/com.ibm.db2.luw.sql.ref.doc/doc/r0008483.html[Db2 SQL Data Types].

The Db2 connector represents changes to rows with events that are structured like the table in which the row exists. The event contains a field for each column value. How that value is represented in the event depends on the Db2 data type of the column. This section describes these mappings.
If the default data type conversions do not meet your needs, you can {link-prefix}:{link-custom-converters}#custom-converters[create a custom converter] for the connector.

ifdef::product[]
Details are in the following sections:
Expand Down Expand Up @@ -2158,6 +2159,30 @@ The following _advanced_ configuration properties have defaults that work in mos
|===
|Property |Default |Description

|[[db2-property-converters]]<<db2-property-converters, `converters`>>
|No default
|Enumerates a comma-separated list of the symbolic names of the {link-prefix}:{link-custom-converters}#custom-converters[custom converter] instances that the connector can use.
For example, +

`isbn`

You must set the `converters` property to enable the connector to use a custom converter.

For each converter that you configure for a connector, you must also add a `.type` property, which specifies the fully-qualifed name of the class that implements the converter interface.
The `.type` property uses the following format: +

`_<converterSymbolicName>_.type` +

For example, +

isbn.type: io.debezium.test.IsbnConverter

If you want to further control the behavior of a configured converter, you can add one or more configuration parameters to pass values to the converter.
To associate any additional configuration parameter with a converter, prefix the parameter names with the symbolic name of the converter. +
For example, +

isbn.schema.name: io.debezium.db2.type.Isbn

|[[db2-property-snapshot-mode]]<<db2-property-snapshot-mode, `+snapshot.mode+`>>
|`initial`
|Specifies the criteria for performing a snapshot when the connector starts: +
Expand Down
39 changes: 32 additions & 7 deletions documentation/modules/ROOT/pages/connectors/mysql.adoc
Expand Up @@ -1310,8 +1310,10 @@ Columns that store strings are defined in MySQL with a character set and collati

The connector can map MySQL data types to both _literal_ and _semantic_ types.

* *Literal type*: how the value is represented using Kafka Connect schema types
* *Semantic type*: how the Kafka Connect schema captures the meaning of the field (schema name)
* *Literal type*: how the value is represented using Kafka Connect schema types.
* *Semantic type*: how the Kafka Connect schema captures the meaning of the field (schema name).

If the default data type conversions do not meet your needs, you can {link-prefix}:{link-custom-converters}#custom-converters[create a custom converter] for the connector.

ifdef::product[]
Details are in the following sections:
Expand Down Expand Up @@ -1604,12 +1606,13 @@ The `BOOLEAN` column is internally mapped to the `TINYINT(1)` data type.
When the table is created during streaming then it uses proper `BOOLEAN` mapping as {prodname} receives the original DDL.
During snapshots, {prodname} executes `SHOW CREATE TABLE` to obtain table definitions that return `TINYINT(1)` for both `BOOLEAN` and `TINYINT(1)` columns. {prodname} then has no way to obtain the original type mapping and so maps to `TINYINT(1)`.

ifdef::community[]
The operator can configure the out-of-the-box xref:{link-custom-converters}[`TinyIntOneToBooleanConverter` custom converter] that would either map all `TINYINT(1)` columns to `BOOLEAN` or if the `selector` parameter is set then a subset of columns could be enumerated using comma-separated regular expressions.
endif::community[]

Following is an example configuration:
To enable you to convert source columns to Boolean data types, {prodname} provides a `TinyIntOneToBooleanConverter` {link-prefix}:{link-custom-converters}#custom-converters[custom converter] that you can use in one of the following ways:

* Map all `TINYINT(1)` columns to `BOOLEAN` types.
* Enumerate a subset of columns by using a comma-separated list of regular expressions. +
To use this type of conversion, you must set the xref:mysql-property-converters[`converters`] configuration property with the `selector` parameter, as shown in the following example:
+
[source]
----
converters=boolean
boolean.type=io.debezium.connector.mysql.converters.TinyIntOneToBooleanConverter
Expand Down Expand Up @@ -2629,6 +2632,28 @@ The following table describes xref:{link-mysql-connector}#mysql-advanced-connect
|`true`
|A Boolean value that specifies whether a separate thread should be used to ensure that the connection to the MySQL server/cluster is kept alive.

|[[mysql-property-converters]]<<mysql-property-converters, `converters`>>
|No default
|Enumerates a comma-separated list of the symbolic names of the {link-prefix}:{link-custom-converters}#custom-converters[custom converter] instances that the connector can use. +
For example, `boolean`. +
This property is required to enable the connector to use a custom converter.

For each converter that you configure for a connector, you must also add a `.type` property, which specifies the fully-qualifed name of the class that implements the converter interface.
The `.type` property uses the following format: +

`_<converterSymbolicName>_.type` +

For example, +

boolean.type: io.debezium.connector.mysql.converters.TinyIntOneToBooleanConverter

If you want to further control the behavior of a configured converter, you can add one or more configuration parameters to pass values to the converter.
To associate these additional configuration parameter with a converter, prefix the paraemeter name with the symbolic name of the converter. +
+
For example, to define a `selector` parameter that specifies the subset of columns that the `boolean` converter processes, add the following property: +

boolean.selector=db1.table1.*, db1.table2.column1

|[[mysql-property-table-ignore-builtin]]<<mysql-property-table-ignore-builtin, `+table.ignore.builtin+`>>
|`true`
|A Boolean value that specifies whether built-in system tables should be ignored. This applies regardless of the table include and exclude lists. By default, system tables are excluded from having their changes captured, and no events are generated when changes are made to any system tables.
Expand Down
33 changes: 29 additions & 4 deletions documentation/modules/ROOT/pages/connectors/oracle.adoc
Expand Up @@ -1110,6 +1110,8 @@ Literal types:: Describe how the value is literally represented, using one of th

Semantic types:: Describe how the Kafka Connect schema captures the _meaning_ of the field, by using the name of the Kafka Connect schema for the field.

If the default data type conversions do not meet your needs, you can {link-prefix}:{link-custom-converters}#custom-converters[create a custom converter] for the connector.

For some Oracle large object (CLOB, NCLOB, and BLOB) and numeric data types, you can manipulate the way that the connector performs the type mapping by changing default configuration property settings.
For more information about how {prodname} properties control mappings for these data types, see xref:oracle-binary-character-lob-types[Binary and Character LOB types] and xref:oracle-numeric-types[Numeric types].

Expand Down Expand Up @@ -1353,11 +1355,12 @@ When the `decimal.handling.mode` property is set to `string`, the connector repr
Oracle does not natively have support for a `BOOLEAN` data type; however,
it is common practice to use other data types with certain semantics to simulate the concept of a logical `BOOLEAN` data type.

The operator can configure the out-of-the-box `NumberOneToBooleanConverter` custom converter that would either map all `NUMBER(1)` columns to a `BOOLEAN` or if the `selector` parameter is set,
then a subset of columns could be enumerated using a comma-separated list of regular expressions.

Following is an example configuration:
To enable you to convert source columns to Boolean data types, {prodname} provides a `NumberOneToBooleanConverter` {link-prefix}:{link-custom-converters}#custom-converters[custom converter] that you can use in one of the following ways:

* Map all `NUMBER(1)` columns to a `BOOLEAN` type.
* Enumerate a subset of columns by using a comma-separated list of regular expressions. +
To use this type of conversion, you must set the xref:oracle-property-converters[`converters`] configuration property with the `selector` parameter, as shown in the following example:
+
[source]
----
converters=boolean
Expand Down Expand Up @@ -2314,6 +2317,28 @@ The following configuration properties are _required_ unless a default value is
|No default
|The name of the Java class for the connector. Always use a value of `io.debezium.connector.oracle.OracleConnector` for the Oracle connector.

|[[oracle-property-converters]]<<oracle-property-converters, `converters`>>
|No default
|Enumerates a comma-separated list of the symbolic names of the {link-prefix}:{link-custom-converters}#custom-converters[custom converter] instances that the connector can use. +
For example, `boolean`. +
This property is required to enable the connector to use a custom converter.

For each converter that you configure for a connector, you must also add a `.type` property, which specifies the fully-qualifed name of the class that implements the converter interface.
The `.type` property uses the following format: +

`_<converterSymbolicName>_.type` +

For example, +

boolean.type: io.debezium.connector.oracle.converters.NumberOneToBooleanConverter

If you want to further control the behavior of a configured converter, you can add one or more configuration parameters to pass values to the converter.
To associate any additional configuration parameters with a converter, prefix the parameter names with the symbolic name of the converter. +
+
For example, to define a `selector` parameter that specifies the subset of columns that the `boolean` converter processes, add the following property: +

boolean.selector: .*MYTABLE.FLAG,.*.IS_ARCHIVED

|[[oracle-property-tasks-max]]<<oracle-property-tasks-max, `+tasks.max+`>>
|`1`
|The maximum number of tasks that should be created for this connector. The Oracle connector always uses a single task and therefore does not use this value, so the default is always acceptable.
Expand Down
26 changes: 26 additions & 0 deletions documentation/modules/ROOT/pages/connectors/postgresql.adoc
Expand Up @@ -1323,6 +1323,8 @@ The PostgreSQL connector represents changes to rows with events that are structu

* _semantic type_ describes how the Kafka Connect schema captures the _meaning_ of the field using the name of the Kafka Connect schema for the field.

If the default data type conversions do not meet your needs, you can {link-prefix}:{link-custom-converters}#custom-converters[create a custom converter] for the connector.

ifdef::product[]
Details are in the following sections:

Expand Down Expand Up @@ -2911,6 +2913,30 @@ The following _advanced_ configuration properties have defaults that work in mos
|Default
|Description

|[[postgresql-property-converters]]<<postgresql-property-converters, `converters`>>
|No default
|Enumerates a comma-separated list of the symbolic names of the {link-prefix}:{link-custom-converters}#custom-converters[custom converter] instances that the connector can use.
For example, +

`isbn`

You must set the `converters` property to enable the connector to use a custom converter.

For each converter that you configure for a connector, you must also add a `.type` property, which specifies the fully-qualifed name of the class that implements the converter interface.
The `.type` property uses the following format: +

`_<converterSymbolicName>_.type` +

For example, +

isbn.type: io.debezium.test.IsbnConverter

If you want to further control the behavior of a configured converter, you can add one or more configuration parameters to pass values to the converter.
To associate any additional configuration parameter with a converter, prefix the parameter names with the symbolic name of the converter. +
For example, +

isbn.schema.name: io.debezium.postgresql.type.Isbn

|[[postgresql-property-snapshot-mode]]<<postgresql-property-snapshot-mode, `+snapshot.mode+`>>
|`initial`
|Specifies the criteria for performing a snapshot when the connector starts: +
Expand Down
26 changes: 26 additions & 0 deletions documentation/modules/ROOT/pages/connectors/sqlserver.adoc
Expand Up @@ -1147,6 +1147,8 @@ The connector can map SQL Server data types to both _literal_ and _semantic_ typ
Literal type:: Describes how the value is literally represented by using Kafka Connect schema types, namely `INT8`, `INT16`, `INT32`, `INT64`, `FLOAT32`, `FLOAT64`, `BOOLEAN`, `STRING`, `BYTES`, `ARRAY`, `MAP`, and `STRUCT`.
Semantic type:: Describes how the Kafka Connect schema captures the _meaning_ of the field using the name of the Kafka Connect schema for the field.

If the default data type conversions do not meet your needs, you can {link-prefix}:{link-custom-converters}#custom-converters[create a custom converter] for the connector.

ifdef::product[]

For more information about data type mappings, see the following sections:
Expand Down Expand Up @@ -2249,6 +2251,30 @@ The following _advanced_ configuration properties have good defaults that will w
|Default
|Description

|[[sqlserver-property-converters]]<<sqlserver-property-converters, `converters`>>
|No default
|Enumerates a comma-separated list of the symbolic names of the {link-prefix}:{link-custom-converters}#custom-converters[custom converter] instances that the connector can use.
For example, +

`isbn`

You must set the `converters` property to enable the connector to use a custom converter.

For each converter that you configure for a connector, you must also add a `.type` property, which specifies the fully-qualifed name of the class that implements the converter interface.
The `.type` property uses the following format: +

`_<converterSymbolicName>_.type` +

For example, +

isbn.type: io.debezium.test.IsbnConverter

If you want to further control the behavior of a configured converter, you can add one or more configuration parameters to pass values to the converter.
To associate any additional configuration parameter with a converter, prefix the parameter names with the symbolic name of the converter.
For example, +

isbn.schema.name: io.debezium.sqlserver.type.Isbn

|[[sqlserver-property-snapshot-mode]]<<sqlserver-property-snapshot-mode, `+snapshot.mode+`>>
|_initial_
|A mode for taking an initial snapshot of the structure and optionally data of captured tables.
Expand Down
26 changes: 26 additions & 0 deletions documentation/modules/ROOT/pages/connectors/vitess.adoc
Expand Up @@ -745,6 +745,8 @@ When a row is deleted, the _delete_ event value still works with log compaction,

The Vitess connector represents changes to rows with events that are structured like the table in which the row exists. The event contains a field for each column value. How that value is represented in the event depends on the Vitess data type of the column. This section describes these mappings.

If the default data type conversions do not meet your needs, you can {link-prefix}:{link-custom-converters}#custom-converters[create a custom converter] for the connector.

[id="vitess-basic-types"]
=== Basic types

Expand Down Expand Up @@ -1229,6 +1231,30 @@ The following _advanced_ configuration properties have defaults that work in mos
|Default
|Description

|[[vitess-property-converters]]<<vitess-property-converters, `converters`>>
|No default
|Enumerates a comma-separated list of the symbolic names of the {link-prefix}:{link-custom-converters}#custom-converters[custom converter] instances that the connector can use.
For example, +

`isbn`

You must set the `converters` property to enable the connector to use a custom converter.

For each converter that you configure for a connector, you must also add a `.type` property, which specifies the fully-qualifed name of the class that implements the converter interface.
The `.type` property uses the following format: +

`_<converterSymbolicName>_.type` +

For example, +

isbn.type: io.debezium.test.IsbnConverter

If you want to further control the behavior of a configured converter, you can add one or more configuration parameters to pass values to the converter.
To associate any additional configuration parameter with a converter, prefix the parameter names with the symbolic name of the converter.
For example, +

isbn.schema.name: io.debezium.vitess.type.Isbn

|[[vitess-property-event-processing-failure-handling-mode]]<<vitess-property-event-processing-failure-handling-mode, `+event.processing.failure.handling.mode+`>>
|`fail`
| Specifies how the connector should react to exceptions during processing of events: +
Expand Down