From 2d9f5aaa8c6eb6c2cdcb21d62ae4d8426713e34d Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Fri, 5 Sep 2025 06:57:17 +0300 Subject: [PATCH 1/4] RDSC-3996 Add RDI 1.14.1 release notes. Update the Google Spanner documentation wrt new default option for using the service account --- .../data-pipelines/prepare-dbs/spanner.md | 39 ++++++++++++++----- .../release-notes/rdi-1-14-1.md | 36 +++++++++++++++++ 2 files changed, 65 insertions(+), 10 deletions(-) create mode 100644 content/integrate/redis-data-integration/release-notes/rdi-1-14-1.md diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md index a196e7ae37..9f840f5859 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md @@ -60,9 +60,10 @@ for more details, including additional configuration options and dialect-specifi ## 3. Create a service account -To allow RDI to access the Spanner instance, you'll need to create a service account with the -appropriate permissions. This service account will then be provided to RDI as a secret for -authentication. +To allow RDI to access the Spanner instance, you'll need to create a service account with the +appropriate permissions. By default, RDI uses Google Cloud Workload Identity authentication +when the service account is assigned to the GKE cluster. Alternatively, you can provide the +service account credentials as a Kubernetes secret (see step 4 for details). 1. Create the service account @@ -109,15 +110,27 @@ authentication. --project=YOUR_PROJECT_ID ``` -## 4. Set up secrets for Kubernetes deployment +### Authentication methods -Before deploying the RDI pipeline, you need to configure the necessary secrets for both the source -and target databases. Instructions for setting up the target database secrets are available in the +RDI supports two authentication methods for accessing Spanner: + +1. **Workload Identity (default)**: The service account is assigned to the GKE cluster, and RDI + automatically uses the cluster's identity to authenticate. This is the recommended approach + as it's more secure and doesn't require managing credential files. + +2. **Service account credentials file**: You provide the service account key file as a Kubernetes + secret. This method requires setting `use_credentials_file: true` in your RDI configuration. + +## 4. Set up secrets for Kubernetes deployment (optional) + +Before deploying the RDI pipeline, you need to configure the necessary secrets for the target +database. Instructions for setting up the target database secrets are available in the [RDI deployment guide]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets-for-k8shelm-deployment-using-kubectl-command" >}}). -In addition to the target database secrets, you'll also need to create a Spanner-specific secret -named `source-db-credentials`. This secret should contain the service account key file generated -during the Spanner setup phase. Use the command below to create it: +**Optional**: If you prefer to use service account credentials file instead of Workload Identity +authentication, you'll need to create a Spanner-specific secret named `source-db-credentials`. +This secret should contain the service account key file generated during the Spanner setup phase. +Use the command below to create it: ```bash kubectl create secret generic source-db-credentials --namespace=rdi \ @@ -125,9 +138,14 @@ kubectl create secret generic source-db-credentials --namespace=rdi \ --save-config --dry-run=client -o yaml | kubectl apply -f - ``` -Be sure to adjust the file path (`~/spanner-reader-account.json`) if your service account key is +Be sure to adjust the file path (`~/spanner-reader-account.json`) if your service account key is stored elsewhere. +{{< note >}} +If you create the `source-db-credentials` secret, you must also set `use_credentials_file: true` +in your RDI configuration to use the credentials file instead of Workload Identity authentication. +{{< /note >}} + ## 5. Configure RDI for Spanner When configuring your RDI pipeline for Spanner, use the following example configuration in your @@ -142,6 +160,7 @@ sources: project_id: your-project-id instance_id: your-spanner-instance database_id: your-spanner-database + # use_credentials_file: false # Set to true if using service account credentials file instead of Workload Identity change_streams: change_stream_all: {} diff --git a/content/integrate/redis-data-integration/release-notes/rdi-1-14-1.md b/content/integrate/redis-data-integration/release-notes/rdi-1-14-1.md new file mode 100644 index 0000000000..1d0f6300d6 --- /dev/null +++ b/content/integrate/redis-data-integration/release-notes/rdi-1-14-1.md @@ -0,0 +1,36 @@ +--- +Title: Redis Data Integration release notes 1.14.1 (August 2025) +alwaysopen: false +categories: +- docs +- operate +- rs +description: | + New RDI API v2 with enhanced pipeline management. + Improved Oracle RAC support with configuration scaffolding. + Enhanced metrics and monitoring capabilities. + Better TLS/mTLS support across components. +linkTitle: 1.14.1 (August 2025) +toc: 'true' +weight: 978 +--- + +{{< note >}}This maintenance release replaces the 1.14.0 release.{{< /note >}} + +RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to: + +- Meet the required speed and scale of read queries and provide an excellent and predictable user experience. +- Save resources and time when building pipelines and coding data transformations. +- Reduce the total cost of ownership by saving money on expensive database read replicas. + +RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism. +It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required. + +## What's New in 1.14.1 + +- Support for Google Cloud Workload Identity authentication when a service account is assigned to the GKE cluster +- Fixed RDI API job validation that was incorrectly failing when schemas are not explicitly specified in source configuration, even though the configuration was valid + +## Limitations + +RDI can write data to a Redis Active-Active database. However, it doesn't support writing data to two or more Active-Active replicas. Writing data from RDI to several Active-Active replicas could easily harm data integrity as RDI is not synchronous with the source database commits. From 50aec7fb8fc1169cc8db6771840b13cbcf2562e9 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Fri, 5 Sep 2025 09:30:44 +0100 Subject: [PATCH 2/4] Updated RDI version in config.toml --- config.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config.toml b/config.toml index 92a28cc0d2..bc62683d48 100644 --- a/config.toml +++ b/config.toml @@ -55,7 +55,7 @@ rdi_redis_gears_version = "1.2.6" rdi_debezium_server_version = "2.3.0.Final" rdi_db_types = "cassandra|mysql|oracle|postgresql|sqlserver" rdi_cli_latest = "latest" -rdi_current_version = "1.14.0" +rdi_current_version = "1.14.1" [params.clientsConfig] "Python"={quickstartSlug="redis-py"} From db1cc5484f0ed66a72c822bd485384aa5a29b90f Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Mon, 8 Sep 2025 05:14:03 +0300 Subject: [PATCH 3/4] Apply suggestions from code review Co-authored-by: Yaron Parasol Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .../data-pipelines/prepare-dbs/spanner.md | 6 +++--- .../redis-data-integration/release-notes/rdi-1-14-1.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md index 9f840f5859..0b1be30dd1 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md @@ -61,7 +61,7 @@ for more details, including additional configuration options and dialect-specifi ## 3. Create a service account To allow RDI to access the Spanner instance, you'll need to create a service account with the -appropriate permissions. By default, RDI uses Google Cloud Workload Identity authentication +appropriate permissions. By default, RDI uses Google Cloud Workload Identity authentication. In this case RDI will assume the [service account is assigned to the GKE cluster](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#enable_on_clusters_and_node_pools) when the service account is assigned to the GKE cluster. Alternatively, you can provide the service account credentials as a Kubernetes secret (see step 4 for details). @@ -127,7 +127,7 @@ Before deploying the RDI pipeline, you need to configure the necessary secrets f database. Instructions for setting up the target database secrets are available in the [RDI deployment guide]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets-for-k8shelm-deployment-using-kubectl-command" >}}). -**Optional**: If you prefer to use service account credentials file instead of Workload Identity +**Optional**: If you prefer to use a service account credentials file instead of Workload Identity authentication, you'll need to create a Spanner-specific secret named `source-db-credentials`. This secret should contain the service account key file generated during the Spanner setup phase. Use the command below to create it: @@ -160,7 +160,7 @@ sources: project_id: your-project-id instance_id: your-spanner-instance database_id: your-spanner-database - # use_credentials_file: false # Set to true if using service account credentials file instead of Workload Identity + # use_credentials_file: false # Default: uses Workload Identity. Set to true to use service account credentials file instead change_streams: change_stream_all: {} diff --git a/content/integrate/redis-data-integration/release-notes/rdi-1-14-1.md b/content/integrate/redis-data-integration/release-notes/rdi-1-14-1.md index 1d0f6300d6..22ee09c137 100644 --- a/content/integrate/redis-data-integration/release-notes/rdi-1-14-1.md +++ b/content/integrate/redis-data-integration/release-notes/rdi-1-14-1.md @@ -12,7 +12,7 @@ description: | Better TLS/mTLS support across components. linkTitle: 1.14.1 (August 2025) toc: 'true' -weight: 978 +weight: 977 --- {{< note >}}This maintenance release replaces the 1.14.0 release.{{< /note >}} From 9c5de3fcc8bce87e42b45743b6a03b31f904740c Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Mon, 8 Sep 2025 05:15:00 +0300 Subject: [PATCH 4/4] Remove unnecessary line after edits --- .../data-pipelines/prepare-dbs/spanner.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md index 0b1be30dd1..59f93e5139 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md @@ -61,8 +61,7 @@ for more details, including additional configuration options and dialect-specifi ## 3. Create a service account To allow RDI to access the Spanner instance, you'll need to create a service account with the -appropriate permissions. By default, RDI uses Google Cloud Workload Identity authentication. In this case RDI will assume the [service account is assigned to the GKE cluster](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#enable_on_clusters_and_node_pools) -when the service account is assigned to the GKE cluster. Alternatively, you can provide the +appropriate permissions. By default, RDI uses Google Cloud Workload Identity authentication. In this case RDI will assume the [service account is assigned to the GKE cluster](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#enable_on_clusters_and_node_pools). Alternatively, you can provide the service account credentials as a Kubernetes secret (see step 4 for details). 1. Create the service account