From 10806188720ad8e25d68e7713142e3a3c202706d Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Fri, 21 Jun 2024 14:19:32 +0100 Subject: [PATCH 1/3] DOC-3910 added SSL properties for MySQL and PostgreSQL --- .../ingest/data-pipelines/data-pipelines.md | 54 ++++++++++++++++++- 1 file changed, 52 insertions(+), 2 deletions(-) diff --git a/content/integrate/redis-data-integration/ingest/data-pipelines/data-pipelines.md b/content/integrate/redis-data-integration/ingest/data-pipelines/data-pipelines.md index c36ec9ba0b..470d2c4d2b 100644 --- a/content/integrate/redis-data-integration/ingest/data-pipelines/data-pipelines.md +++ b/content/integrate/redis-data-integration/ingest/data-pipelines/data-pipelines.md @@ -148,13 +148,17 @@ configuration contains the following data: - `sink`: All advanced properties for writing to RDI (TLS, memory threshold, etc). See the Debezium [Redis stream properties](https://debezium.io/documentation/reference/stable/operations/debezium-server.html#_redis_stream) page for the full set of available properties. - - `source`: All advanced connector properties (for example, RAC nodes). See the + - `source`: All advanced connector properties (for example, RAC nodes). + See [Database-specific connection properties](#db-connect-props) below and also + see the Debezium [Connectors](https://debezium.io/documentation/reference/stable/connectors/) - page for the full set of available properties. + pages for more information about the properties available for each database type. - `quarkus`: All advanced properties for Debezium server, such as the log level. See the Quarkus [Configuration options](https://quarkus.io/guides/all-config) docs for the full set of available properties. + + ### Targets Use this section to provide the connection details for the target Redis @@ -165,6 +169,52 @@ with a unique name that you are free to choose (here, we have used `host` and `port` of the server. You can also supply connection credentials and TLS/mTLS secrets here if you use them. +### Database-specific connection properties {#db-connect-props} + +Use the following properties in the [`sources.advanced.source`](#sources) section +of `config.yaml` for more control over RDI's connection to your database. + +#### MySQL/MariaDB + +See the +[Debezium SSL mode properties](https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-property-database-ssl-mode) +for a full list of properties specific to MySQL/MariaDB. + +- [`database.ssl.keystore`](https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-property-database-ssl-keystore): + (Optional) The location of the key store file. Use this for two-way authentication between + your client and the MySQL/MariaDB Server. +- [`database.ssl.keystore.password`](https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-property-database-ssl-keystore-password): + (Optional) The password for the key store file. You only need this if you have also configured + `database.ssl.keystore`. +- [`database.ssl.truststore`](https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-property-database-ssl-truststore): + The location of the trust store file to use for server certificate verification. +- [`database.ssl.truststore.password`](https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-property-database-ssl-truststore-password): + The password for the trust store file. This is required both to check the integrity of the truststore + and to unlock it. + +#### PostgreSQL + +See the +[Debezium connector properties](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-connector-properties) +for a full list of properties specific to PostgreSQL. + +- [`database.sslcert`](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-property-database-sslcert): + The file path for the client's SSL certificate for the client. See + [Database Connection Control Functions](https://www.postgresql.org/docs/current/libpq-connect.html) + in the PostgreSQL docs for more information. +- [`database.sslkey`](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-property-database-sslkey): + The file path for the client's SSL private key. See + [Database Connection Control Functions](https://www.postgresql.org/docs/current/libpq-connect.html) + in the PostgreSQL docs for more information. +- [`database.sslpassword`](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-property-database-sslpassword): + The password for the client's private key file as specified `database.sslkey`. See + [Database Connection Control Functions](https://www.postgresql.org/docs/current/libpq-connect.html) + in the PostgreSQL docs for more information. +- [`database.sslrootcert`](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-property-database-sslrootcert): + The file path for the root certificate(s) used to validate the server. See + [Database Connection Control Functions](https://www.postgresql.org/docs/current/libpq-connect.html) + in the PostgreSQL docs for more information. + ## Job files You can optionally supply one or more job files that specify how you want to From bc9e98f1b9ce74eeb610c98530145d8a9d3ad280 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Mon, 24 Jun 2024 10:10:05 +0100 Subject: [PATCH 2/3] DOC-3910 new advanced config property info --- .../ingest/data-pipelines/data-pipelines.md | 45 +++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/content/integrate/redis-data-integration/ingest/data-pipelines/data-pipelines.md b/content/integrate/redis-data-integration/ingest/data-pipelines/data-pipelines.md index 470d2c4d2b..41d6f9ac26 100644 --- a/content/integrate/redis-data-integration/ingest/data-pipelines/data-pipelines.md +++ b/content/integrate/redis-data-integration/ingest/data-pipelines/data-pipelines.md @@ -215,6 +215,51 @@ for a full list of properties specific to PostgreSQL. [Database Connection Control Functions](https://www.postgresql.org/docs/current/libpq-connect.html) in the PostgreSQL docs for more information. +#### Oracle + +See the Kafka +[configuration docs](https://kafka.apache.org/documentation.html#configuration) +for a full list of properties relevant to Oracle configuration. +Where a property has a `` element, you can set the role to be +either `producer` or `consumer`, as appropriate. + +- [`schema.history.internal..security.protocol`](https://kafka.apache.org/documentation.html#consumerconfigs_security.protocol): + The protocol for communicating with brokers. This can take the values + `PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, and `SASL_SSL`. +- [`schema.history.internal..ssl.keystore.location`](https://kafka.apache.org/documentation.html#producerconfigs_ssl.keystore.location): + The file path for the keystore. +- [`schema.history.internal..ssl.keystore.password`](https://kafka.apache.org/documentation.html#producerconfigs_ssl.keystore.password): + The password for the keystore file. You only need this you have also set + `schema.history.internal..ssl.keystore.location`. +- [`schema.history.internal..ssl.truststore.location`](https://kafka.apache.org/documentation.html#producerconfigs_ssl.truststore.location): + The file path for the truststore. +- [`schema.history.internal..ssl.truststore.password`](https://kafka.apache.org/documentation.html#producerconfigs_ssl.truststore.password): + The password for the trust store file. If you don't set a password, RDI will still use the trust store + file specified in `schema.history.internal..ssl.truststore.location` but without integrity + checking. +- [`schema.history.internal..ssl.key.password`](https://kafka.apache.org/documentation.html#producerconfigs_ssl.key.password): + The password for the private key in the keystore file specified in + `schema.history.internal..ssl.keystore.location`. +- [`database.dbname`](https://debezium.io/documentation/reference/stable/connectors/oracle.html#oracle-property-database-dbname): + The name of the database you want to connect to. If you are using a container database environment, + then you should set this to the name of the root container database (CDB), rather than an included + pluggable database (PDB). +- [`database.pdb.name`](https://debezium.io/documentation/reference/stable/connectors/oracle.html#oracle-property-database-pdb-name): + The name of the Oracle pluggable database you want to connect to. You can only use this with + container database (CDB) installations. +- [`decimal.handling.mode`](https://debezium.io/documentation/reference/stable/connectors/oracle.html#oracle-property-decimal-handling-mode): + This specifies the data format for floating point values in `NUMBER`, `DECIMAL` and `NUMERIC` + columns. This can take the values `precise` (store values with any number of decimal places), + `double` (use double-precision floating point), or `string` (encode numbers as strings). + The default value is `precise`. See + [Oracle numeric types](https://debezium.io/documentation/reference/stable/connectors/oracle.html#oracle-numeric-types) + for more information about decimal handling. +- `key.converter.schemas.enable` and `value.converter.schemas.enable`: + Boolean values specifying whether or not you want to add JSON schemas to + serialized data. See Kafka's [connect transforms](https://kafka.apache.org/documentation/#connect_transforms) + docs for an example. + + ## Job files You can optionally supply one or more job files that specify how you want to From edfed378cd6ee5f49a4c6401a604a80ac5e0e8bb Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 27 Jun 2024 17:02:32 +0100 Subject: [PATCH 3/3] DOC-3910 minor fixes elsewhere --- .../ingest/data-pipelines/data-pipelines.md | 4 +++- .../redis-data-integration/ingest/installation/_index.md | 3 ++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/content/integrate/redis-data-integration/ingest/data-pipelines/data-pipelines.md b/content/integrate/redis-data-integration/ingest/data-pipelines/data-pipelines.md index 41d6f9ac26..12ac992624 100644 --- a/content/integrate/redis-data-integration/ingest/data-pipelines/data-pipelines.md +++ b/content/integrate/redis-data-integration/ingest/data-pipelines/data-pipelines.md @@ -345,7 +345,9 @@ The main sections of these files are: - `expire`: Positive integer value indicating a number of seconds for the key to expire. If you don't specify this property, the key will never expire. -{{< note >}}You must specify a `transform` section, or a `key` section under `output`, or both.{{< /note >}} +{{< note >}}In a job file, the `transform` section is optional, but if you don't specify +a `transform`, you must specify custom key logic in `output.with.key`. You can include +both of these sections if you want both a custom transform and a custom key.{{< /note >}} Another example below shows how you can rename the `fname` field to `first_name` in the table `emp` using the diff --git a/content/integrate/redis-data-integration/ingest/installation/_index.md b/content/integrate/redis-data-integration/ingest/installation/_index.md index 72e2b18db5..cd36ba3e08 100644 --- a/content/integrate/redis-data-integration/ingest/installation/_index.md +++ b/content/integrate/redis-data-integration/ingest/installation/_index.md @@ -52,7 +52,8 @@ Each of the RDI VMs should have: Follow the steps below for each of your VMs: -1. Download the RDI installer from the [Redis download center](https://redis.io/downloads) +1. Download the RDI installer from the [Redis download center](https://app.redislabs.com/#/rlec-downloads) + (under the *Modules, Tools & Integration* dropdown) and extract it to your preferred installation folder. 1. Go to the installation folder: