From 8676aca0da579cb172a017050a4a8b79ec3b8095 Mon Sep 17 00:00:00 2001
From: Scott Wicken <1562170+swicken@users.noreply.github.com>
Date: Wed, 8 Jul 2026 10:24:45 -0400
Subject: [PATCH 1/2] fix(opensearch): preserve OS index store on phase
rollback during in-flight reindex (#36471)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Rolling FEATURE_FLAG_OPEN_SEARCH_PHASE back to 0 while a full reindex was
draining the journal wiped the active OS working/live rows from the
indicies store and stranded a partial .os reindex pair on the cluster,
which a later boot catchup adopted as active — phase-2 searches then
silently returned a fraction of the content.
- Scope IndiciesFactory.point()'s delete to its own NULL-version rows so
the legacy ES store update can never remove the OS versioned rows it
does not manage (root cause of the store wipe; in phases 1/2 the wipe
was masked only because the OS mirror re-wrote the rows from a stale
VersionedIndicesCache).
- Treat OS reindex slots found during a Phase-0 switchover or abort as a
stranded mid-reindex rollback: clear the slots (a leftover slot makes
isInFullReindex() true again on a later flip to Phase 2 and would
switch ES over onto null pointers) and delete the partial physical
.os indices so no boot catchup can ever adopt them.
- Cover the rollback in OpenSearchUpgradeSuite
(ContentletIndexAPIImplMidReindexRollbackIT) and document the
operational rule in OPENSEARCH_MIGRATION.md.
---
docs/backend/OPENSEARCH_MIGRATION.md | 18 +
.../business/ContentletIndexAPIImpl.java | 58 +++
.../business/IndiciesFactory.java | 7 +-
.../com/dotcms/OpenSearchUpgradeSuite.java | 2 +
...ntletIndexAPIImplMidReindexRollbackIT.java | 334 ++++++++++++++++++
5 files changed, 418 insertions(+), 1 deletion(-)
create mode 100644 dotcms-integration/src/test/java/com/dotcms/content/elasticsearch/business/ContentletIndexAPIImplMidReindexRollbackIT.java
diff --git a/docs/backend/OPENSEARCH_MIGRATION.md b/docs/backend/OPENSEARCH_MIGRATION.md
index 9a0a4a7caf7b..92a1c231b2e5 100644
--- a/docs/backend/OPENSEARCH_MIGRATION.md
+++ b/docs/backend/OPENSEARCH_MIGRATION.md
@@ -670,6 +670,24 @@ rollback, with no impact on normal operation.
> **Status**: Option 1 (runbook) is the only mitigation currently in place. Option 2 and 3 are
> not yet implemented — tracked as technical debt before Phase 2 goes to production.
+#### Phase rollback during an in-flight full reindex (#36471)
+
+Rolling `FEATURE_FLAG_OPEN_SEARCH_PHASE` back to 0 **while a full reindex is draining the
+journal** is a distinct hazard from the mapping drift above. The phase is re-read per journal
+batch, so the remaining entries index to ES only and the OS reindex pair freezes partially
+populated. The ES switchover then completes in Phase 0.
+
+**Fixed behavior:** the Phase-0 switchover (and abort) now treats this state as an OS reindex
+abort — the active OS working/live rows survive in the store (the legacy `indicies` update is
+scoped to its own NULL-version rows), the OS reindex slots are cleared, and the partial physical
+`.os` pair is deleted from the cluster so a later boot catchup can never adopt it as active. The
+abort is logged at WARN with the deleted index names.
+
+**Operational rule:** the OS pair that survives the rollback is the *old* one — it stops
+receiving writes in Phase 0 and drifts exactly as described above. Before re-activating Phase 2,
+trigger a full reindex so OS is rebuilt in a dual-write phase. Prefer letting an in-flight
+reindex finish (or aborting it explicitly) over flipping the phase mid-drain.
+
---
### Fan-out routing with divergent index names — resolved (#35640)
diff --git a/dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/ContentletIndexAPIImpl.java b/dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/ContentletIndexAPIImpl.java
index 99856ef93ba7..e2ca60c7219f 100644
--- a/dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/ContentletIndexAPIImpl.java
+++ b/dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/ContentletIndexAPIImpl.java
@@ -1424,6 +1424,11 @@ public synchronized boolean fullReindexSwitchover(Connection conn, final boolean
} catch (Exception osEx) {
Logger.warn(this, "Could not mirror reindex switchover to OS store", osEx);
}
+ } else {
+ // Phase 0 with OS reindex slots present = the phase flag was rolled back while
+ // a dual-write full reindex was in flight (#36471). The OS pair is partial and
+ // must never survive as adoptable state.
+ abortStrandedOsReindex();
}
// Async: merge index segments and expand replicas on the newly active indices,
@@ -1523,6 +1528,55 @@ private boolean fullReindexSwitchoverOS(final boolean forceSwitch) throws Except
return true;
}
+ /**
+ * Aborts a stranded OS reindex left behind by a Phase-0 rollback during an in-flight
+ * dual-write full reindex (#36471). When the phase flag is rolled back mid-journal-drain,
+ * the OS reindex pair stops receiving writes and stays partial; the OS store still holds
+ * the reindex slots pointing at it. Left alone, those slots make {@code isInFullReindex()}
+ * report true again on a later flip to Phase 2 (triggering a switchover over null ES
+ * pointers), and the partial indices are the exact {@code .os} twins of the promoted ES
+ * names — what a boot catchup would mirror-adopt as active, silently serving a fraction
+ * of the content.
+ *
+ * Clears the slots first (the safety-critical part — active working/live pointers are
+ * preserved), then deletes the partial physical indices best-effort. Never throws: this
+ * runs inside the ES switchover/abort, which must not be undone by OS housekeeping.
+ */
+ private void abortStrandedOsReindex() {
+ try {
+ final Optional osExisting =
+ versionedIndicesAPI.loadDefaultVersionedIndices();
+ final Optional reindexWorking =
+ osExisting.flatMap(VersionedIndices::reindexWorking);
+ final Optional reindexLive =
+ osExisting.flatMap(VersionedIndices::reindexLive);
+ if (reindexWorking.isEmpty() && reindexLive.isEmpty()) {
+ return;
+ }
+ Logger.warn(this, "Migration phase was rolled back to 0 during a full reindex:"
+ + " aborting the OS reindex — clearing the OS reindex slots and deleting the"
+ + " partial indices [" + reindexWorking.orElse("none") + ", "
+ + reindexLive.orElse("none") + "] so they can never be adopted as active"
+ + " (#36471)");
+
+ final VersionedIndicesImpl.Builder osBuilder = VersionedIndicesImpl.builder();
+ osExisting.flatMap(VersionedIndices::working).ifPresent(osBuilder::working);
+ osExisting.flatMap(VersionedIndices::live).ifPresent(osBuilder::live);
+ // reindexWorking / reindexLive intentionally omitted → cleared
+ versionedIndicesAPI.saveIndices(osBuilder.build());
+
+ for (final Optional name : List.of(reindexWorking, reindexLive)) {
+ name.ifPresent(idx -> Try.run(() -> operationsOS.indexAPI().delete(idx))
+ .onFailure(e -> Logger.warn(this,
+ "Could not delete partial OS reindex index " + idx
+ + " — delete it manually", e)));
+ }
+ } catch (Exception osEx) {
+ Logger.warn(this, "Could not abort the stranded OS reindex (#36471) — the OS store"
+ + " may still hold reindex slots pointing at partial indices", osEx);
+ }
+ }
+
/**
* Optimizes (force-merges) the newly-promoted indices after a reindex switchover, targeting
* each provider with the names it actually holds: ES with its bare names, OS with its
@@ -2899,6 +2953,10 @@ public void fullReindexAbort() {
} catch (Exception osEx) {
Logger.warn(this, "Could not clear OS reindex slots during abort", osEx);
}
+ } else {
+ // Same mid-reindex rollback state as the switchover path (#36471), reached when
+ // the operator aborts instead of letting the journal drain.
+ abortStrandedOsReindex();
}
} catch (Exception e) {
diff --git a/dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/IndiciesFactory.java b/dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/IndiciesFactory.java
index bc550f228ae2..0270b2530c71 100644
--- a/dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/IndiciesFactory.java
+++ b/dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/IndiciesFactory.java
@@ -86,7 +86,12 @@ public void point(final IndiciesInfo newInfo) throws DotDataException {
}
DotConnect dc = new DotConnect();
final String insertSQL = "INSERT INTO indicies VALUES(?,?)";
- final String deleteSQL = "DELETE from indicies where index_type=? or index_name=?";
+ // Scoped to index_version IS NULL: this legacy store only owns the ES rows (the same
+ // rows loadIndicies reads). The OS migration rows carry a non-NULL index_version in the
+ // shared table and are managed by VersionedIndicesAPI — an unscoped delete-by-type wipes
+ // them on every ES switchover, which in Phase 0 orphans the OS index store (#36471).
+ final String deleteSQL =
+ "DELETE from indicies where (index_type=? or index_name=?) and index_version is null";
for (IndexType type : IndexType.values()) {
final String indexType = type.toString().toLowerCase();
final String newValue = Try.of(() -> (String) PropertyUtils
diff --git a/dotcms-integration/src/test/java/com/dotcms/OpenSearchUpgradeSuite.java b/dotcms-integration/src/test/java/com/dotcms/OpenSearchUpgradeSuite.java
index 25bf6a0a5aad..cc807479af9a 100644
--- a/dotcms-integration/src/test/java/com/dotcms/OpenSearchUpgradeSuite.java
+++ b/dotcms-integration/src/test/java/com/dotcms/OpenSearchUpgradeSuite.java
@@ -2,6 +2,7 @@
import com.dotcms.content.elasticsearch.business.MigrationPhaseStoreBootstrapIT;
import com.dotcms.content.elasticsearch.business.ContentletIndexAPIImplPhaseSwitchIntegrationTest;
+import com.dotcms.content.elasticsearch.business.ContentletIndexAPIImplMidReindexRollbackIT;
import com.dotcms.content.elasticsearch.business.ContentletIndexAPIImplMigrationIntegrationTest;
import com.dotcms.content.index.opensearch.ContentFactoryIndexOperationsOSIntegrationTest;
import com.dotcms.content.index.opensearch.ContentletIndexOperationsOSIntegrationTest;
@@ -49,6 +50,7 @@
OSClientConfigTest.class,
ContentletIndexAPIImplMigrationIntegrationTest.class,
ContentletIndexAPIImplPhaseSwitchIntegrationTest.class,
+ ContentletIndexAPIImplMidReindexRollbackIT.class,
MigrationPhaseStoreBootstrapIT.class,
OSSearchAPIImplIntegrationTest.class,
OSSiteSearchAPIIntegrationTest.class,
diff --git a/dotcms-integration/src/test/java/com/dotcms/content/elasticsearch/business/ContentletIndexAPIImplMidReindexRollbackIT.java b/dotcms-integration/src/test/java/com/dotcms/content/elasticsearch/business/ContentletIndexAPIImplMidReindexRollbackIT.java
new file mode 100644
index 000000000000..3dd4fd3c2a69
--- /dev/null
+++ b/dotcms-integration/src/test/java/com/dotcms/content/elasticsearch/business/ContentletIndexAPIImplMidReindexRollbackIT.java
@@ -0,0 +1,334 @@
+package com.dotcms.content.elasticsearch.business;
+
+import static com.dotcms.content.index.IndexConfigHelper.MigrationPhase.FLAG_KEY;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import com.dotcms.DataProviderWeldRunner;
+import com.dotcms.IntegrationTestBase;
+import com.dotcms.content.index.IndexAPIImpl;
+import com.dotcms.content.index.IndexTag;
+import com.dotcms.content.index.VersionedIndices;
+import com.dotcms.content.index.VersionedIndicesImpl;
+import com.dotcms.content.index.opensearch.OSIndexAPIImpl;
+import com.dotcms.util.IntegrationTestInitService;
+import com.dotmarketing.business.APILocator;
+import com.dotmarketing.exception.DotDataException;
+import com.dotmarketing.util.Config;
+import com.dotmarketing.util.Logger;
+import io.vavr.control.Try;
+import java.util.List;
+import java.util.Optional;
+import java.util.UUID;
+import javax.enterprise.context.ApplicationScoped;
+import javax.inject.Inject;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+
+/**
+ * Regression tests for #36471 —
+ * rolling the migration phase back to 0 while a dual-write full reindex is in flight must not
+ * wipe the OS index store, and must not strand a partial OS reindex pair that a later boot
+ * catchup could adopt as active.
+ *
+ * Scenario under test
+ * Phase 1, full reindex running: reindex slots exist on both engines. The operator flips
+ * {@code FEATURE_FLAG_OPEN_SEARCH_PHASE} back to 0 mid-journal-drain, then the ES switchover
+ * (or an abort) completes in Phase 0. Before the fix:
+ *
+ * - {@code IndiciesFactory.point()} deleted rows by {@code index_type} across ALL
+ * {@code index_version} values, wiping the active OS working/live rows (4 rows → 2);
+ * - the partial OS reindex pair stayed on the cluster unknown to the DB, and a later boot
+ * catchup mirror-adopted it as active — silently serving a fraction of the content.
+ *
+ *
+ * Fixed contract (asserted here)
+ *
+ * - The OS working/live rows survive a Phase-0 switchover and abort untouched.
+ * - The OS reindex slots are cleared (a stranded slot would make {@code isInFullReindex()}
+ * true again on a later flip to Phase 2 and trigger a switchover over null ES pointers).
+ * - The partial physical OS reindex indices are deleted from the cluster — nothing left
+ * for a boot catchup to adopt.
+ *
+ *
+ * Run command
+ *
+ * ./mvnw verify -pl :dotcms-integration \
+ * -Dcoreit.test.skip=false \
+ * -Dopensearch.upgrade.test=true \
+ * -Dit.test=ContentletIndexAPIImplMidReindexRollbackIT
+ *
+ */
+@ApplicationScoped
+@RunWith(DataProviderWeldRunner.class)
+public class ContentletIndexAPIImplMidReindexRollbackIT extends IntegrationTestBase {
+
+ // ── Unique run suffix prevents cross-run index name collisions ─────────────
+ private static final String RUN_ID =
+ UUID.randomUUID().toString().replace("-", "").substring(0, 8);
+
+ /**
+ * Timestamps embedded in the reindex index names. They must be (a) parseable as the
+ * trailing {@code _yyyyMMddHHmmss} suffix — {@code fullReindexSwitchover}'s minimum-runtime
+ * guard parses them — and (b) far in the past so the guard ({@code
+ * REINDEX_THREAD_MINIMUM_RUNTIME_IN_SEC}, default 30s) does not defer the switchover.
+ */
+ private static final String OLD_TS = "20200101000000";
+ private static final String REINDEX_TS = "20200102000000";
+
+ // ── ES names (logical; stored cluster-prefixed) ─────────────────────────────
+ private static final String ES_WORKING = "working_rbk" + RUN_ID + "_" + OLD_TS;
+ private static final String ES_LIVE = "live_rbk" + RUN_ID + "_" + OLD_TS;
+ private static final String ES_REINDEX_WK = "working_rbk" + RUN_ID + "_" + REINDEX_TS;
+ private static final String ES_REINDEX_LV = "live_rbk" + RUN_ID + "_" + REINDEX_TS;
+
+ // ── OS names (logical; stored cluster-prefixed + .os-tagged) ────────────────
+ private static final String OS_WORKING = IndexTag.OS.tag("working_rbk" + RUN_ID + "_" + OLD_TS);
+ private static final String OS_LIVE = IndexTag.OS.tag("live_rbk" + RUN_ID + "_" + OLD_TS);
+ private static final String OS_REINDEX_WK = IndexTag.OS.tag("working_rbk" + RUN_ID + "_" + REINDEX_TS);
+ private static final String OS_REINDEX_LV = IndexTag.OS.tag("live_rbk" + RUN_ID + "_" + REINDEX_TS);
+
+ @Inject
+ private OSIndexAPIImpl osIndexAPI;
+
+ // ── Saved DB state — restored in @After ────────────────────────────────────
+ @SuppressWarnings("deprecation")
+ private IndiciesInfo savedEsInfo;
+ private Optional savedOsIndices;
+
+ @BeforeClass
+ public static void prepare() throws Exception {
+ IntegrationTestInitService.getInstance().init();
+ }
+
+ @Before
+ public void setUp() throws Exception {
+ savedEsInfo = APILocator.getIndiciesAPI().loadIndicies();
+ savedOsIndices = APILocator.getVersionedIndicesAPI().loadDefaultVersionedIndices();
+ cleanupTestIndices();
+ }
+
+ @After
+ public void tearDown() {
+ Config.setProperty(FLAG_KEY, null);
+ cleanupTestIndices();
+
+ Try.run(() -> APILocator.getIndiciesAPI().point(savedEsInfo))
+ .onFailure(e -> Logger.warn(this,
+ "tearDown: could not restore ES pointers: " + e.getMessage()));
+ savedOsIndices.ifPresent(v ->
+ Try.run(() -> APILocator.getVersionedIndicesAPI().saveIndices(v))
+ .onFailure(e -> Logger.warn(this,
+ "tearDown: could not restore OS pointers: " + e.getMessage())));
+ }
+
+ // =========================================================================
+ // Root fix — IndiciesFactory.point() must only manage NULL-version rows
+ // =========================================================================
+
+ /**
+ * Given: the OS store holds working/live rows (version {@code os-3.x}) alongside the
+ * legacy ES rows (NULL version) in the shared {@code indicies} table.
+ * When: the legacy ES store is re-pointed via {@code IndiciesAPI.point()} — the exact
+ * store update every ES switchover and abort performs.
+ * Then: the OS rows are untouched. Before the fix the unscoped delete-by-type inside
+ * {@code point()} wiped them (#36471, step "the .os rows die").
+ */
+ @Test
+ public void test_legacyPoint_preservesOsVersionedRows() throws DotDataException {
+ setPhase(0);
+ pointOsStore(OS_WORKING, OS_LIVE, null, null);
+ pointEsStore(ES_WORKING, ES_LIVE, null, null);
+
+ // Re-point ES to a new pair — this is what the Phase-0 switchover does.
+ pointEsStore(ES_REINDEX_WK, ES_REINDEX_LV, null, null);
+
+ final Optional os =
+ APILocator.getVersionedIndicesAPI().loadDefaultVersionedIndices();
+ assertTrue("OS store record must survive an ES point()", os.isPresent());
+ assertEquals("OS working row must survive an ES point()",
+ Optional.of(osPhysical(OS_WORKING)), os.get().working());
+ assertEquals("OS live row must survive an ES point()",
+ Optional.of(osPhysical(OS_LIVE)), os.get().live());
+
+ Logger.info(this, "✅ point() preserved the OS versioned rows");
+ }
+
+ // =========================================================================
+ // Phase-0 switchover after a mid-reindex rollback
+ // =========================================================================
+
+ /**
+ * Given: the state a mid-drain rollback leaves behind — Phase 0, ES store with active
+ * working/live plus reindex slots, OS store with active working/live plus reindex
+ * slots pointing at a PARTIAL physical pair on the OS cluster.
+ * When: the ES switchover completes in Phase 0.
+ * Then: ES promotes its reindex pair as usual; the OS working/live rows survive; the OS
+ * reindex slots are cleared; and the partial physical OS pair is deleted from the
+ * cluster so no later boot catchup can adopt it.
+ */
+ @Test
+ public void test_phase0Switchover_midReindexRollback_preservesOsStoreAndAbortsOsReindex()
+ throws Exception {
+ seedMidReindexRollbackState();
+
+ final boolean switched =
+ APILocator.getContentletIndexAPI().fullReindexSwitchover(true);
+ assertTrue("Phase-0 switchover must complete", switched);
+
+ // ES: reindex pair promoted, slots cleared — the normal Phase-0 contract.
+ final IndiciesInfo esInfo = APILocator.getIndiciesAPI().loadIndicies();
+ assertEquals("ES working must be the promoted reindex-working index",
+ esPhysical(ES_REINDEX_WK), esInfo.getWorking());
+ assertEquals("ES live must be the promoted reindex-live index",
+ esPhysical(ES_REINDEX_LV), esInfo.getLive());
+
+ assertOsReindexAborted();
+ Logger.info(this, "✅ Phase-0 switchover preserved the OS store and aborted the OS reindex");
+ }
+
+ // =========================================================================
+ // Phase-0 abort after a mid-reindex rollback
+ // =========================================================================
+
+ /**
+ * Same rollback state as the switchover test, but the operator aborts the reindex instead
+ * of letting the journal drain. The ES store keeps its active pair with the reindex slots
+ * cleared, and the OS side is aborted identically.
+ */
+ @Test
+ public void test_phase0Abort_midReindexRollback_preservesOsStoreAndAbortsOsReindex()
+ throws Exception {
+ seedMidReindexRollbackState();
+
+ APILocator.getContentletIndexAPI().fullReindexAbort();
+
+ // ES: active pair preserved, reindex slots cleared — the normal abort contract.
+ final IndiciesInfo esInfo = APILocator.getIndiciesAPI().loadIndicies();
+ assertEquals("ES working must stay the active index after abort",
+ esPhysical(ES_WORKING), esInfo.getWorking());
+ assertEquals("ES live must stay the active index after abort",
+ esPhysical(ES_LIVE), esInfo.getLive());
+
+ assertOsReindexAborted();
+ Logger.info(this, "✅ Phase-0 abort preserved the OS store and aborted the OS reindex");
+ }
+
+ // =========================================================================
+ // Helpers
+ // =========================================================================
+
+ /**
+ * Builds the exact store + cluster state a mid-journal-drain rollback leaves behind:
+ * both stores fully populated with active and reindex slots, the partial OS reindex pair
+ * physically present on the OS cluster, and the phase flag already rolled back to 0.
+ */
+ private void seedMidReindexRollbackState() throws Exception {
+ setPhase(1);
+
+ // Physical indices: the ES reindex pair (promoted by the switchover) and the partial
+ // OS reindex pair (the orphans-to-be). The old active pairs are pointer-only — the
+ // switchover never touches them physically.
+ esImpl().createIndex(ES_REINDEX_WK, 1);
+ esImpl().createIndex(ES_REINDEX_LV, 1);
+ osIndexAPI.createIndex(OS_REINDEX_WK, 1);
+ osIndexAPI.createIndex(OS_REINDEX_LV, 1);
+
+ pointEsStore(ES_WORKING, ES_LIVE, ES_REINDEX_WK, ES_REINDEX_LV);
+ pointOsStore(OS_WORKING, OS_LIVE, OS_REINDEX_WK, OS_REINDEX_LV);
+
+ // The rollback: operator flips the phase flag back to 0 mid-drain.
+ setPhase(0);
+ }
+
+ /** The fixed post-rollback OS contract shared by the switchover and abort tests. */
+ private void assertOsReindexAborted() throws Exception {
+ final Optional os =
+ APILocator.getVersionedIndicesAPI().loadDefaultVersionedIndices();
+ assertTrue("OS store record must survive the Phase-0 rollback", os.isPresent());
+ assertEquals("OS working row must survive untouched",
+ Optional.of(osPhysical(OS_WORKING)), os.get().working());
+ assertEquals("OS live row must survive untouched",
+ Optional.of(osPhysical(OS_LIVE)), os.get().live());
+ assertTrue("OS reindex-working slot must be cleared",
+ os.get().reindexWorking().isEmpty());
+ assertTrue("OS reindex-live slot must be cleared",
+ os.get().reindexLive().isEmpty());
+
+ assertFalse("partial OS reindex-working index must be deleted from the cluster",
+ osIndexAPI.indexExists(OS_REINDEX_WK));
+ assertFalse("partial OS reindex-live index must be deleted from the cluster",
+ osIndexAPI.indexExists(OS_REINDEX_LV));
+ }
+
+ private static void setPhase(final int ordinal) {
+ Config.setProperty(FLAG_KEY, String.valueOf(ordinal));
+ }
+
+ private static ESIndexAPI esImpl() {
+ return ((IndexAPIImpl) APILocator.getESIndexAPI()).esImpl();
+ }
+
+ /** Cluster-prefixed physical form of an ES logical name (what the ES store persists). */
+ private static String esPhysical(final String logicalName) {
+ return esImpl().getNameWithClusterIDPrefix(logicalName);
+ }
+
+ /** Cluster-prefixed physical form of an OS logical name (what the OS store persists). */
+ private String osPhysical(final String logicalName) {
+ return osIndexAPI.getNameWithClusterIDPrefix(logicalName);
+ }
+
+ /** Points the legacy ES store, preserving nothing — deterministic slot state. */
+ private static void pointEsStore(final String working, final String live,
+ final String reindexWorking, final String reindexLive) throws DotDataException {
+ final IndiciesInfo.Builder builder = new IndiciesInfo.Builder();
+ builder.setWorking(esPhysical(working));
+ builder.setLive(esPhysical(live));
+ if (reindexWorking != null) {
+ builder.setReindexWorking(esPhysical(reindexWorking));
+ }
+ if (reindexLive != null) {
+ builder.setReindexLive(esPhysical(reindexLive));
+ }
+ APILocator.getIndiciesAPI().point(builder.build());
+ }
+
+ /** Points the OS versioned store directly (names are already {@code .os}-tagged). */
+ private void pointOsStore(final String working, final String live,
+ final String reindexWorking, final String reindexLive) throws DotDataException {
+ final VersionedIndicesImpl.Builder builder = VersionedIndicesImpl.builder();
+ builder.working(osPhysical(working));
+ builder.live(osPhysical(live));
+ if (reindexWorking != null) {
+ builder.reindexWorking(osPhysical(reindexWorking));
+ }
+ if (reindexLive != null) {
+ builder.reindexLive(osPhysical(reindexLive));
+ }
+ APILocator.getVersionedIndicesAPI().saveIndices(builder.build());
+ }
+
+ private void cleanupTestIndices() {
+ for (final String name : List.of(ES_WORKING, ES_LIVE, ES_REINDEX_WK, ES_REINDEX_LV)) {
+ Try.run(() -> {
+ if (esImpl().indexExists(name)) {
+ esImpl().delete(name);
+ }
+ }).onFailure(e -> Logger.warn(this,
+ "Cleanup ES index '" + name + "': " + e.getMessage()));
+ }
+ for (final String name : List.of(OS_WORKING, OS_LIVE, OS_REINDEX_WK, OS_REINDEX_LV)) {
+ Try.run(() -> {
+ if (osIndexAPI.indexExists(name)) {
+ osIndexAPI.delete(name);
+ }
+ }).onFailure(e -> Logger.warn(this,
+ "Cleanup OS index '" + name + "': " + e.getMessage()));
+ }
+ }
+}
From 5049c6e9a103e87739e87ba862d4e973c0373171 Mon Sep 17 00:00:00 2001
From: Scott Wicken <1562170+swicken@users.noreply.github.com>
Date: Wed, 8 Jul 2026 11:15:10 -0400
Subject: [PATCH 2/2] fix(opensearch): handle slots-only OS record in
stranded-reindex abort (#36471)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
When the OS store record holds reindex slots but no active working/live
(reachable when the active OS pair is deleted through the index-management
flow mid-reindex), rebuilding the record from working/live alone produced
an empty record that saveIndices rejects — the throw skipped the physical
deletes and left exactly the stranded slots + partial indices the abort
exists to remove. Remove the version row instead (same treatment as
clearOsStorePointer, #35640) so the deletes always run, and preserve a
siteSearch pointer if one exists. Covered by a fourth IT case.
---
.../business/ContentletIndexAPIImpl.java | 12 ++++-
...ntletIndexAPIImplMidReindexRollbackIT.java | 47 +++++++++++++++++++
2 files changed, 58 insertions(+), 1 deletion(-)
diff --git a/dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/ContentletIndexAPIImpl.java b/dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/ContentletIndexAPIImpl.java
index e2ca60c7219f..cbfe95d2fefb 100644
--- a/dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/ContentletIndexAPIImpl.java
+++ b/dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/ContentletIndexAPIImpl.java
@@ -1562,8 +1562,18 @@ private void abortStrandedOsReindex() {
final VersionedIndicesImpl.Builder osBuilder = VersionedIndicesImpl.builder();
osExisting.flatMap(VersionedIndices::working).ifPresent(osBuilder::working);
osExisting.flatMap(VersionedIndices::live).ifPresent(osBuilder::live);
+ osExisting.flatMap(VersionedIndices::siteSearch).ifPresent(osBuilder::siteSearch);
// reindexWorking / reindexLive intentionally omitted → cleared
- versionedIndicesAPI.saveIndices(osBuilder.build());
+ final VersionedIndices rebuilt = osBuilder.build();
+ if (rebuilt.hasAnyIndex()) {
+ versionedIndicesAPI.saveIndices(rebuilt);
+ } else {
+ // The reindex slots were the only OS pointers (e.g. the active OS pair was
+ // deleted via the index-management flow mid-reindex). saveIndices contractually
+ // rejects an empty record — remove the version row instead, same as
+ // clearOsStorePointer (#35640), so the physical deletes below still run.
+ versionedIndicesAPI.removeVersion(VersionedIndices.OPENSEARCH_3X);
+ }
for (final Optional name : List.of(reindexWorking, reindexLive)) {
name.ifPresent(idx -> Try.run(() -> operationsOS.indexAPI().delete(idx))
diff --git a/dotcms-integration/src/test/java/com/dotcms/content/elasticsearch/business/ContentletIndexAPIImplMidReindexRollbackIT.java b/dotcms-integration/src/test/java/com/dotcms/content/elasticsearch/business/ContentletIndexAPIImplMidReindexRollbackIT.java
index 3dd4fd3c2a69..ef78498573e5 100644
--- a/dotcms-integration/src/test/java/com/dotcms/content/elasticsearch/business/ContentletIndexAPIImplMidReindexRollbackIT.java
+++ b/dotcms-integration/src/test/java/com/dotcms/content/elasticsearch/business/ContentletIndexAPIImplMidReindexRollbackIT.java
@@ -218,6 +218,53 @@ public void test_phase0Abort_midReindexRollback_preservesOsStoreAndAbortsOsReind
Logger.info(this, "✅ Phase-0 abort preserved the OS store and aborted the OS reindex");
}
+ // =========================================================================
+ // Phase-0 switchover when the reindex slots are the ONLY OS pointers
+ // =========================================================================
+
+ /**
+ * Given: the OS store record holds reindex slots but no active working/live (reachable when
+ * the active OS pair is deleted via the index-management flow mid-reindex — that flow
+ * preserves the reindex slots). The partial pair exists physically. Phase rolled back
+ * to 0.
+ * When: the ES switchover completes in Phase 0.
+ * Then: the OS version record is removed entirely (an empty record cannot be saved —
+ * {@code saveIndices} rejects it, and a thrown save must not skip the physical
+ * deletes), and the partial pair is deleted from the cluster.
+ */
+ @Test
+ public void test_phase0Switchover_reindexSlotsOnly_removesRecordAndDeletesPartialPair()
+ throws Exception {
+ setPhase(1);
+ esImpl().createIndex(ES_REINDEX_WK, 1);
+ esImpl().createIndex(ES_REINDEX_LV, 1);
+ osIndexAPI.createIndex(OS_REINDEX_WK, 1);
+ osIndexAPI.createIndex(OS_REINDEX_LV, 1);
+
+ pointEsStore(ES_WORKING, ES_LIVE, ES_REINDEX_WK, ES_REINDEX_LV);
+
+ // OS store: reindex slots only — no working/live to preserve.
+ APILocator.getVersionedIndicesAPI().saveIndices(VersionedIndicesImpl.builder()
+ .reindexWorking(osPhysical(OS_REINDEX_WK))
+ .reindexLive(osPhysical(OS_REINDEX_LV))
+ .build());
+
+ setPhase(0);
+
+ final boolean switched =
+ APILocator.getContentletIndexAPI().fullReindexSwitchover(true);
+ assertTrue("Phase-0 switchover must complete", switched);
+
+ assertTrue("OS version record must be removed when no active pointers remain",
+ APILocator.getVersionedIndicesAPI().loadDefaultVersionedIndices().isEmpty());
+ assertFalse("partial OS reindex-working index must be deleted from the cluster",
+ osIndexAPI.indexExists(OS_REINDEX_WK));
+ assertFalse("partial OS reindex-live index must be deleted from the cluster",
+ osIndexAPI.indexExists(OS_REINDEX_LV));
+
+ Logger.info(this, "✅ Phase-0 switchover removed the slots-only OS record and the partial pair");
+ }
+
// =========================================================================
// Helpers
// =========================================================================