fix(migration): guard activateIndex against a missing OpenSearch counterpart (#36360)

[fabrizzio-dotCMS]

Problem

Activating an old/inactive index is dotCMS's rollback mechanism (revert to a previous reindex). But if that index predates the migration, it never went through the OpenSearch create fan-out, so it has no OpenSearch counterpart. ContentletIndexAPIImpl.activateIndex is phase-aware but only repoints both stores by name (toPhysicalName → cluster_X.<name>.os) — no indexExists, no create, no reconcile. Reactivating such an index during the migration is unguarded:

• Phase 1: silent divergence (OS points at a .os index that doesn't exist; shadow writes swallowed).
• Phase 2: the ES read-fallback masks it (logs ERROR per read).
• Phase 3 (the cliff): ES is decommissioned, no fallback → OS points at a missing index → empty results / errors = "content lost" for the customer.

delete already has a phase-aware guard (assertIndexNotActive); activate had none.

Fix — a narrow guard

During the migration (phases 1/2/3), activateIndex now refuses to activate an index whose OpenSearch counterpart does not exist, throwing DotStateException (→ 400) with a clear message.

• It is narrow: a rollback to a migration-era index that DOES have its .os copy still works. Only the genuinely-broken case (no OS copy) is blocked.
• Phase 0 is not guarded (no OpenSearch store).
• Escape hatch: ALLOW_ACTIVATE_INDEX_WITHOUT_OS_MIRROR=true forces the activation (accepting that OpenSearch is left pointing at a missing index until rebuilt) — mirroring the ALLOW_ACTIVE_INDEX_DELETE override on the delete guard.

Known trade-off (deliberate, "simple first")

A hard block conflicts with rollback: you can't roll back to a pre-migration index during the migration without first rebuilding it (or setting the override). The non-blocking alternative — allow the rollback, repoint ES instantly (stays transparent), and rebuild the OpenSearch copy asynchronously via the existing reindex machinery, with the readiness gate refusing Phase 3 until it's ready — is the intended follow-up. This PR is the cheap, safe first step that stops the silent Phase-3 detonation.

Tests

• ContentletIndexAPIImplActivateGuardTest (6, surefire): blocks in phases 1/2/3 when the OS counterpart is missing; allows when it's present, in Phase 0, and under the override flag. Set-backed engine fakes control OS existence — no cluster needed.
• Fixed the shared FakeContentletIndexOperations.indexAPI() (previously threw) to report existence, so the existing phase-2/3 activateIndex tests in ContentletIndexAPIImplPhaseTest (14) keep passing under the new precondition.

CI note: migration integration tests that activate an index in a dual/OS phase now require the OS counterpart to exist (real bootstrap normally provides it). Watch the OpenSearch upgrade / phase suites.

Notes

• Independent of the readiness endpoint (feat(migration): internal migration-readiness endpoint (content + site search) (#36360) #36849) and the mirror-reconciliation PR (fix(sitesearch): reconcile ES/OS index mirrors on the write path (#36360) #36825); based on main. Tied to [QA-G17] OpenSearch Migration — Site Search Functional Validation (Phases 0–3, $sitesearch viewtool) #36360 (no new issue).

🤖 Generated with Claude Code

[claude]

Claude finished @fabrizzio-dotCMS's task in 2m 21s —— View job

---

Code Review — activateIndex OpenSearch-counterpart guard

• Read the diff (ContentletIndexAPIImpl, ESIndexResource, both tests)
• Trace callers / blast radius of activateIndex
• Check guard placement, phase semantics, error paths
• Verify test coverage

New Issues

• 🟡 Medium: ContentletIndexAPIImpl.java:3301 — Fail-closed existence check conflates "OS unreachable" with "OS counterpart missing." Try.of(() -> operationsOS.indexAPI().indexExists(osCounterpart)).getOrElse(false) treats any exception (connection timeout, cluster red, transient OpenSearch outage) as "the index does not exist" and blocks the activation. Since activateIndex is the rollback mechanism, a transient OS hiccup during Phases 1/2 — where ES is still primary and fully functional — would block a legitimate emergency ES rollback, exactly when an operator most needs it. The ALLOW_ACTIVATE_INDEX_WITHOUT_OS_MIRROR override mitigates it, but only if the operator knows to reach for it mid-incident.
  Assumption: indexExists throws (rather than returning false) on cluster connectivity failures.
  What to verify: the exception behavior of operationsOS.indexAPI().indexExists() on a red/unreachable cluster; consider distinguishing "confirmed absent" from "could not determine" (e.g. only block on a confirmed false, and log-and-allow on a thrown exception).

• 🟡 Medium: ContentletIndexAPIImpl.java:3280 — Audit log claims the activation happened before the guard can reject it. The Logger.info(..., "Index activation (" + indexName + ") performed by user...") runs unconditionally, ahead of the guard block at line 3298. When the guard throws, the audit trail records "activation performed" for an activation that was actually blocked, which is misleading for anyone reading logs during an incident. Consider moving the audit line after the guard, or wording it as "requested" until it succeeds.

Notes (non-blocking)

• The guard does not sit on the normal reindex switchover path — fullReindexSwitchover repoints slots directly and never calls activateIndex, so routine reindexes are unaffected. Good; the blast radius is limited to explicit activate calls (REST modIndex default case, IndexAjaxAction.activateIndex).
• ESIndexResource.java:620 — switching the log from the raw action query param to the parsed indexAction enum, and adding the sendAdminMessage toast, is a clean improvement and consistent with the existing delete/clear guard handling.
• Test coverage is solid: phases 1/2/3 blocked-when-missing, allowed-when-present, Phase 0 / no-flag / override all covered. The FakeContentletIndexOperations.indexAPI() fix (returning indexExists == true instead of throwing) correctly keeps the existing phase-2/3 activate tests green under the new precondition.

Both findings are 🟡 Medium — non-blocking. The core guard is correct and the "simple first" tradeoff is clearly documented in the PR.

• issue-36360-activate-index-os-guard

[fabrizzio-dotCMS]

Closing — a hard existence guard on activateIndex (block when the OpenSearch counterpart is missing) is fundamentally incompatible with the transparent-mirror / catch-up design.

activate/deactivate are intentionally pointer-store updates keyed by name, not validated against cluster state — a documented, tested contract (ContentletIndexAPIImplMigrationIntegrationTest: "the OS DB pointer reflects the name passed in, regardless of which index the OS cluster actually holds", "deactivateIndex never validates cluster existence … a pointer-store update driven by the name pattern, not by cluster state"). During catch-up the ES/OS names diverge and the OS physical may not exist yet, so activate must mirror the pointer optimistically.

At activation time, two cases are indistinguishable by a point-in-time existence check:

• ✅ the OS copy will exist in a moment (normal dual-write catch-up), vs.
• ❌ the OS copy will never exist (reactivating a pre-migration backup).

Both are "no OS copy right now", so the guard blocks the legitimate catch-up too — breaking normal migration operation. CI confirmed it: 6 ITs in the OpenSearch Upgrade Suite failed for exactly this reason.

The real risk (reactivating an un-mirrored backup → empty search at Phase 3) is a known, accepted characteristic of the optimistic-mirror model; the intended mitigation is the migration-readiness endpoint (#36849) — it detects a MISSING_COUNTERPART and gates the Phase-3 promotion, rather than a hard block on activate.

Finding documented in docs/backend/OPENSEARCH_MIGRATION.md (#36825).