From 407674243cd0d98379b535a89c530f4802201ace Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Tue, 1 Jul 2025 16:52:57 +0300 Subject: [PATCH 1/5] RDSC-3785: Document known issue for RDI upgrade from < 1.8.0 to >= 1.8.0 --- .../installation/upgrade.md | 20 +++++++++++++++++ .../release-notes/rdi-1-8-0.md | 22 +++++++++++++++++++ 2 files changed, 42 insertions(+) diff --git a/content/integrate/redis-data-integration/installation/upgrade.md b/content/integrate/redis-data-integration/installation/upgrade.md index 5aebf9add8..9967806760 100644 --- a/content/integrate/redis-data-integration/installation/upgrade.md +++ b/content/integrate/redis-data-integration/installation/upgrade.md @@ -82,6 +82,26 @@ If there is an active pipeline, upgrade RDI on the active VM first. This will cause a short pipeline downtime of up to two minutes. Afterwards, upgrade RDI on the passive VM. This will not cause any downtime. +{{< warning >}} +When upgrading from RDI < 1.8.0 to RDI >= 1.8.0 in a VM HA setup, both RDI instances may incorrectly consider themselves active after the upgrade. This occurs because the upgrade process doesn't update the `rdi:ha:lock` value from the legacy `cluster-1` hardcoded identifier, causing both clusters to assume they are the active cluster. + +**Symptoms:** + +- The upgraded passive node will start collector and processor components +- Collector may enter a crash loop as it fails to connect to the source +- Both clusters will restart in a loop + +**Workaround:** + +After upgrading, manually set a unique cluster ID for one of the installations (preferred is the passive instance): + +```bash +kubectl edit cm -n rdi rdi-sys-config +``` + +Then add the line `RDI_CLUSTER_ID: cluster-2` to distinguish between the clusters. +{{< /warning >}} + ## Upgrading a Kubernetes installation Follow the steps below to upgrade an existing diff --git a/content/integrate/redis-data-integration/release-notes/rdi-1-8-0.md b/content/integrate/redis-data-integration/release-notes/rdi-1-8-0.md index 1bcacfe6fa..dc4b927023 100644 --- a/content/integrate/redis-data-integration/release-notes/rdi-1-8-0.md +++ b/content/integrate/redis-data-integration/release-notes/rdi-1-8-0.md @@ -87,3 +87,25 @@ The RDI operator has been significantly enhanced in the following areas: ## Limitations RDI can write data to a Redis Active-Active database. However, it doesn't support writing data to two or more Active-Active replicas. Writing data from RDI to several Active-Active replicas could easily harm data integrity as RDI is not synchronous with the source database commits. + +## Known Issues + +### High Availability upgrade issue + +When upgrading from RDI < 1.8.0 to RDI >= 1.8.0 in an HA setup, both RDI instances may incorrectly consider themselves active after the upgrade. This occurs because the upgrade process doesn't update the `rdi:ha:lock` value from the legacy `cluster-1` identifier, causing both clusters to assume they are the active cluster. + +**Symptoms:** + +- The upgraded passive node will start collector and processor components +- Collector may enter a crash loop as it fails to connect to the source +- Both clusters will restart in a loop + +**Workaround:** + +After upgrading, manually set a unique cluster ID for one of the installations by editing the configmap: + +```bash +kubectl edit cm -n rdi rdi-sys-config +``` + +Then add the following line `RDI_CLUSTER_ID: cluster-2` to distinguish between the clusters. From 91f3ec6233fc4a34d63020a0b8549ac8b58133ca Mon Sep 17 00:00:00 2001 From: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> Date: Tue, 1 Jul 2025 15:57:51 +0100 Subject: [PATCH 2/5] Update content/integrate/redis-data-integration/installation/upgrade.md Co-authored-by: Stoyan Rachev --- .../integrate/redis-data-integration/installation/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/installation/upgrade.md b/content/integrate/redis-data-integration/installation/upgrade.md index 9967806760..99ceb83db2 100644 --- a/content/integrate/redis-data-integration/installation/upgrade.md +++ b/content/integrate/redis-data-integration/installation/upgrade.md @@ -83,7 +83,7 @@ This will cause a short pipeline downtime of up to two minutes. Afterwards, upgrade RDI on the passive VM. This will not cause any downtime. {{< warning >}} -When upgrading from RDI < 1.8.0 to RDI >= 1.8.0 in a VM HA setup, both RDI instances may incorrectly consider themselves active after the upgrade. This occurs because the upgrade process doesn't update the `rdi:ha:lock` value from the legacy `cluster-1` hardcoded identifier, causing both clusters to assume they are the active cluster. +When upgrading from RDI < 1.8.0 to RDI >= 1.8.0 in a VM HA setup, both RDI instances may incorrectly consider themselves active after the upgrade. This occurs because the upgrade process doesn't change the cluster id value from its default `cluster-1`, causing both clusters to assume they are the active cluster. **Symptoms:** From 0ff6aae4031de50079e4bca2e5cd40d7021d06b2 Mon Sep 17 00:00:00 2001 From: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> Date: Tue, 1 Jul 2025 15:58:42 +0100 Subject: [PATCH 3/5] Update content/integrate/redis-data-integration/installation/upgrade.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .../redis-data-integration/installation/upgrade.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/integrate/redis-data-integration/installation/upgrade.md b/content/integrate/redis-data-integration/installation/upgrade.md index 99ceb83db2..431163bd1f 100644 --- a/content/integrate/redis-data-integration/installation/upgrade.md +++ b/content/integrate/redis-data-integration/installation/upgrade.md @@ -95,11 +95,11 @@ When upgrading from RDI < 1.8.0 to RDI >= 1.8.0 in a VM HA setup, both RDI insta After upgrading, manually set a unique cluster ID for one of the installations (preferred is the passive instance): -```bash -kubectl edit cm -n rdi rdi-sys-config -``` +1. Locate the RDI configuration file on the VM host. The file is typically located at `/etc/rdi/rdi-sys-config.yaml`. +2. Open the configuration file in a text editor. For example: -Then add the line `RDI_CLUSTER_ID: cluster-2` to distinguish between the clusters. + ```bash + sudo nano /etc/rdi/rdi-sys-config.yaml {{< /warning >}} ## Upgrading a Kubernetes installation From 4275e8219b113afb7b80f9245fff3aa2073a3400 Mon Sep 17 00:00:00 2001 From: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> Date: Tue, 1 Jul 2025 15:59:00 +0100 Subject: [PATCH 4/5] Update content/integrate/redis-data-integration/installation/upgrade.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .../integrate/redis-data-integration/installation/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/installation/upgrade.md b/content/integrate/redis-data-integration/installation/upgrade.md index 431163bd1f..48592b3ecf 100644 --- a/content/integrate/redis-data-integration/installation/upgrade.md +++ b/content/integrate/redis-data-integration/installation/upgrade.md @@ -93,7 +93,7 @@ When upgrading from RDI < 1.8.0 to RDI >= 1.8.0 in a VM HA setup, both RDI insta **Workaround:** -After upgrading, manually set a unique cluster ID for one of the installations (preferred is the passive instance): +After upgrading, manually set a unique cluster ID for one of the installations (preferably on the passive instance): 1. Locate the RDI configuration file on the VM host. The file is typically located at `/etc/rdi/rdi-sys-config.yaml`. 2. Open the configuration file in a text editor. For example: From f0b3e0f911fb1e8f0e0f6c382f81e30815ddd8e7 Mon Sep 17 00:00:00 2001 From: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> Date: Tue, 1 Jul 2025 15:59:09 +0100 Subject: [PATCH 5/5] Update content/integrate/redis-data-integration/release-notes/rdi-1-8-0.md --- .../redis-data-integration/release-notes/rdi-1-8-0.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/release-notes/rdi-1-8-0.md b/content/integrate/redis-data-integration/release-notes/rdi-1-8-0.md index dc4b927023..4f164c81b6 100644 --- a/content/integrate/redis-data-integration/release-notes/rdi-1-8-0.md +++ b/content/integrate/redis-data-integration/release-notes/rdi-1-8-0.md @@ -108,4 +108,8 @@ After upgrading, manually set a unique cluster ID for one of the installations b kubectl edit cm -n rdi rdi-sys-config ``` -Then add the following line `RDI_CLUSTER_ID: cluster-2` to distinguish between the clusters. +Then add the following line to distinguish between the clusters: + +```bash +RDI_CLUSTER_ID: cluster-2 +```