Skip to content

Commit

Permalink
update docs
Browse files Browse the repository at this point in the history
  • Loading branch information
@whhe
whhe committed Mar 13, 2024
1 parent a4f0875 commit 32fa202
Showing 1 changed file with 93 additions and 27 deletions.
120 changes: 93 additions & 27 deletions docs/content/docs/connectors/cdc-connectors/oceanbase-cdc.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,22 +28,82 @@ under the License.

The OceanBase CDC connector allows for reading snapshot data and incremental data from OceanBase. This document describes how to set up the OceanBase CDC connector to run SQL queries against OceanBase.

## OceanBase CDC Solutions

Glossary:

- *OceanBase CE*: OceanBase Community Edition. It's compatible with MySQL and has been open sourced at https://github.com/oceanbase/oceanbase.
- *OceanBase EE*: OceanBase Enterprise Edition. It supports two compatibility modes: MysQL and Oracle. See https://en.oceanbase.com.
- *OceanBase Cloud*: OceanBase Enterprise Edition on Cloud. See https://en.oceanbase.com/product/cloud.
- *OceanBase Log Proxy CE*: OceanBase Log Proxy Community Edition. It's a proxy service which can fetch the commit log data of OceanBase cluster. It has been open sourced at https://github.com/oceanbase/oblogproxy.
- *OceanBase Log Proxy EE*: OceanBase Log Proxy Enterprise Edition. It's only available for users of OceanBase Cloud, you can contact the provider support for more details.
- *OceanBase Binlog Service CE*: OceanBase Binlog Service Community Edition. It's a solution composed of several open source components, which is compatible with MySQL Replication Protocol. See [docs (Chinese)](https://www.oceanbase.com/docs/community-oblogproxy-doc-1000000000531983).
- *OceanBase Binlog Service EE*: OceanBase Binlog Service Enterprise Edition. It's only available for users of Alibaba Cloud, see [docs](https://www.alibabacloud.com/help/en/apsaradb-for-oceanbase/latest/binlog-overview).
- *OceanBase Driver*: The Jdbc driver for OceanBase, which supports both MySQL mode and Oracle mode of all OceanBase versions. It's open sourced at https://github.com/oceanbase/obconnector-j.

Solutions:

<div class="wy-table-responsive">
<table class="colwidths-auto docutils">
<thead>
<tr>
<th class="text-left">Database</th>
<th class="text-left">Supported Driver</th>
<th class="text-left">CDC Connector</th>
<th class="text-left">Other Required Components</th>
</tr>
</thead>
<tbody>
<tr>
<td rowspan="2">OceanBase CE</td>
<td>
MySQL Driver: 5.1.47 (builtin), 8.0.x <br>
OceanBase Driver: 2.4.x
</td>
<td>OceanBase CDC Connector</td>
<td>OceanBase Log Proxy CE (CDC Mode)</td>
</tr>
<tr>
<td>MySQL Driver: 8.0.27 (builtin)</td>
<td>MySQL CDC Connector</td>
<td>OceanBase Binlog Service CE</td>
</tr>
<tr>
<td rowspan="2">OceanBase EE (MySQL Mode)</td>
<td>
MySQL Driver: 5.1.47 (builtin), 8.0.x <br>
OceanBase Driver: 2.4.x
</td>
<td>OceanBase CDC Connector</td>
<td>OceanBase Log Proxy EE (CDC Mode)</td>
</tr>
<tr>
<td>MySQL Driver: 8.0.27 (builtin)</td>
<td>MySQL CDC Connector</td>
<td>OceanBase Binlog Service EE</td>
</tr>
<tr>
<td>OceanBase EE (Oracle Mode)</td>
<td>OceanBase Driver: 2.4.x</td>
<td>OceanBase CDC Connector</td>
<td>OceanBase Log Proxy EE (CDC Mode)</td>
</tr>
</tbody>
</table>
</div>

Note:
- If you prefer to use MySQL CDC Connector, you can check the docs for more details at [mysql-cdc](mysql-cdc.md).
- When you want to use a driver other than the builtin driver, you need to manually introduce it to your application.

Dependencies
------------

In order to set up the OceanBase CDC connector, the following table provides dependency information for both projects using a build automation tool (such as Maven or SBT) and SQL Client with SQL JAR bundles.

{{< artifact flink-connector-oceanbase-cdc >}}

If you want to use OceanBase JDBC driver to connect to the enterprise edition database, you should also include the following dependency in your class path.
### Maven dependency

```xml
<dependency>
<groupId>com.oceanbase</groupId>
<artifactId>oceanbase-client</artifactId>
<version>2.4.2</version>
</dependency>
```
{{< artifact flink-connector-oceanbase-cdc >}}

### SQL Client JAR

Expand Down Expand Up @@ -84,7 +144,7 @@ Setup OceanBase and LogProxy Server
mysql> show parameters like 'obconfig_url';
```

5. Setup OceanBase LogProxy. For users of OceanBase Community Edition, you can follow the [quick start](https://github.com/oceanbase/oblogproxy#getting-started).
5. Setup OceanBase LogProxy. For users of OceanBase Community Edition, you can follow the [docs (Chinese)](https://www.oceanbase.com/docs/community-oblogproxy-doc-1000000000531984).

How to create a OceanBase CDC table
----------------
Expand Down Expand Up @@ -162,7 +222,7 @@ Connector Options
The OceanBase CDC Connector contains some options for both sql and stream api as the following sheet.

*Note*: The connector supports two ways to specify the table list to listen to, and will get the union of the results when both way are used at the same time.
1. Use `database-name` and `table-name` to match database and table names in regex. As the `obcdc` (former `liboblog`) only supports `fnmatch` now, we can't use regex directly to filter change events, so these two options can only be used in `initial` startup mode.
1. Use `database-name` and `table-name` to match database and table names in regex.
2. Use `table-list` to match the exact value of database and table names.

<div class="highlight">
Expand All @@ -186,11 +246,11 @@ The OceanBase CDC Connector contains some options for both sql and stream api as
</tr>
<tr>
<td>scan.startup.mode</td>
<td>required</td>
<td style="word-wrap: break-word;">(none)</td>
<td>optional</td>
<td style="word-wrap: break-word;">initial</td>
<td>String</td>
<td>Specify the startup mode for OceanBase CDC consumer, valid enumerations are
<code>'initial'</code>,<code>'latest-offset'</code> or <code>'timestamp'</code>.
<code>'initial'</code>,<code>'latest-offset'</code>,<code>'timestamp'</code> or <code>'snapshot'</code>.
</td>
</tr>
<tr>
Expand All @@ -216,24 +276,24 @@ The OceanBase CDC Connector contains some options for both sql and stream api as
</tr>
<tr>
<td>tenant-name</td>
<td>required</td>
<td>optional</td>
<td style="word-wrap: break-word;">(none)</td>
<td>String</td>
<td>Tenant name of OceanBase to monitor, should be exact value.</td>
<td>Tenant name of OceanBase to monitor, should be exact value. Required when 'scan.startup.mode' is not 'snapshot'.</td>
</tr>
<tr>
<td>database-name</td>
<td>optional</td>
<td style="word-wrap: break-word;">(none)</td>
<td>String</td>
<td>Database name of OceanBase to monitor, should be regular expression. Only can be used with 'initial' mode.</td>
<td>Database name of OceanBase to monitor, should be regular expression.</td>
</tr>
<tr>
<td>table-name</td>
<td>optional</td>
<td style="word-wrap: break-word;">(none)</td>
<td>String</td>
<td>Table name of OceanBase to monitor, should be regular expression. Only can be used with 'initial' mode.</td>
<td>Table name of OceanBase to monitor, should be regular expression.</td>
</tr>
<tr>
<td>table-list</td>
Expand All @@ -244,14 +304,14 @@ The OceanBase CDC Connector contains some options for both sql and stream api as
</tr>
<tr>
<td>hostname</td>
<td>optional</td>
<td>required</td>
<td style="word-wrap: break-word;">(none)</td>
<td>String</td>
<td>IP address or hostname of the OceanBase database server or OceanBase Proxy server.</td>
</tr>
<tr>
<td>port</td>
<td>optional</td>
<td>required</td>
<td style="word-wrap: break-word;">(none)</td>
<td>Integer</td>
<td>Integer port number to connect to OceanBase. It can be the SQL port of OceanBase server, which is 2881 by default, or the port of OceanBase proxy service, which is 2883 by default.</td>
Expand All @@ -272,17 +332,17 @@ The OceanBase CDC Connector contains some options for both sql and stream api as
</tr>
<tr>
<td>logproxy.host</td>
<td>required</td>
<td>optional</td>
<td style="word-wrap: break-word;">(none)</td>
<td>String</td>
<td>Hostname or IP address of OceanBase log proxy service.</td>
<td>Hostname or IP address of OceanBase log proxy service. Required when 'scan.startup.mode' is not 'snapshot'.</td>
</tr>
<tr>
<td>logproxy.port</td>
<td>required</td>
<td>optional</td>
<td style="word-wrap: break-word;">(none)</td>
<td>Integer</td>
<td>Port number of OceanBase log proxy service.</td>
<td>Port number of OceanBase log proxy service. Required when 'scan.startup.mode' is not 'snapshot'.</td>
</tr>
<tr>
<td>logproxy.client.id</td>
Expand Down Expand Up @@ -360,14 +420,19 @@ The following format metadata can be exposed as read-only (VIRTUAL) columns in a
<tbody>
<tr>
<td>tenant_name</td>
<td>STRING NOT NULL</td>
<td>STRING</td>
<td>Name of the tenant that contains the row.</td>
</tr>
<tr>
<td>database_name</td>
<td>STRING NOT NULL</td>
<td>STRING</td>
<td>Name of the database that contains the row.</td>
</tr>
<tr>
<td>schema_name</td>
<td>STRING</td>
<td>Name of the schema that contains the row.</td>
</tr>
<tr>
<td>table_name</td>
<td>STRING NOT NULL</td>
Expand Down Expand Up @@ -430,6 +495,7 @@ The config option `scan.startup.mode` specifies the startup mode for OceanBase C
- `initial`: Performs an initial snapshot on the monitored table upon first startup, and continue to read the latest commit log.
- `latest-offset`: Never to perform snapshot on the monitored table upon first startup and just read the latest commit log since the connector is started.
- `timestamp`: Never to perform snapshot on the monitored table upon first startup and just read the commit log from the given `scan.startup.timestamp`.
- `snapshot`: Only perform snapshot on the monitored table.

### Consume Commit Log

Expand Down

0 comments on commit 32fa202

Please sign in to comment.