Skip to content

[DEFECT] Confusing empty OS index appears in indices portlet on Phase 1 migration bootstrap #35820

@fabrizzio-dotCMS

Description

@fabrizzio-dotCMS

Problem Statement

When configuring a dotCMS instance to start the ES → OpenSearch migration for the first time and setting the feature flag FEATURE_FLAG_OPEN_SEARCH_PHASE=1, an additional index appears in the admin indices portlet (the page that lists all indices with their count, replicas, status, etc.).

This extra entry is confusing because:

  • It corresponds to an OpenSearch index, but it is not the index where new writes are going.
  • It does not match the index registered in the indicies table (the active OS 3.X working/live index — the one persisted with the os:: tag and managed by VersionedIndicesAPIImpl).
  • It appears completely empty in the portlet: no document count, no number of replicas, no shard info — all fields blank.
  • The index does exist on the OpenSearch console (it is a real cluster resource) but it holds no data.
  • A second OS index (the real shadow index) is what actually receives the dual-writes for contentlets, but the portlet does not distinguish which is which.

The operator can no longer tell at a glance which index is the active OS shadow index, which is empty/stale, or whether something is wrong with the migration bootstrap. This undermines trust in the migration tooling at the exact moment operators need clarity (first activation of Phase 1).

Impact: Operator confusion during the most sensitive moment of the migration (Phase 0 → 1 transition). No data loss; ES still owns reads and the correct OS shadow index still receives writes. Severity bumped above Low because it directly blocks the operator's ability to validate that Phase 1 is healthy before promoting to Phase 2.

Context: See docs/backend/OPENSEARCH_MIGRATION.md — sections "Index Architecture" (os:: tag ownership rule) and "Shadow Index Strategy" for the expected single-shadow-index model per cluster.

Steps to Reproduce

  1. Start with a dotCMS instance running in Phase 0 (FEATURE_FLAG_OPEN_SEARCH_PHASE=0 or unset) with an existing ES index registered in the indicies table — e.g. cluster_<id>.working_20230101.
  2. Bring up an OpenSearch cluster reachable from dotCMS.
  3. Set FEATURE_FLAG_OPEN_SEARCH_PHASE=1 in dotmarketing-config.properties (or via env var).
  4. Restart (or wait for the bootstrap / catchup to run and create the OS shadow index).
  5. Navigate to the admin indices portlet (the page that lists all indices).
  6. Observe the list of indices shown.

Expected: One ES index (working + live) and one OS shadow index (working + live), each clearly labeled with its vendor and with proper document count / replicas / shard info populated. The OS entry shown must be the one registered in the indicies table (the index actually receiving dual-writes).

Actual: An additional OS index appears in the portlet that is not the one registered in indicies and not the one receiving writes. It is rendered with all metrics blank (no count, no replicas). The real shadow index is also present but the operator cannot distinguish which is which without going to the OpenSearch console.

Acceptance Criteria

  • After enabling FEATURE_FLAG_OPEN_SEARCH_PHASE=1 on a fresh-bootstrap instance, the indices portlet shows exactly the OS indices that are registered in the indicies table (no orphan or duplicate OS entries).
  • Every OS entry shown in the portlet has its document count, number of replicas, and shard metadata correctly populated — no blank/empty rows.
  • Each row in the portlet is clearly labeled with its provider (ES vs OS) so the operator can distinguish them at a glance.
  • The OS shadow index shown in the portlet matches the one actually receiving dual-writes (verifiable by comparing the index name in the portlet to the os::-tagged name stored in the indicies table after stripping the tag).
  • If an OS index exists in the cluster but is not registered in the indicies table (orphan), it must not appear in the portlet — or, if shown for diagnostic purposes, it must be visually distinguished as "orphan / not active".
  • Repeating the bootstrap (Phase 0 → 1) on a clean instance never leaves more than one active OS shadow index visible in the portlet.
  • No regression in the ES portlet behavior — existing ES rows continue to render counts, replicas, and shard info as before.

dotCMS Version

Latest from main branch (any build that supports FEATURE_FLAG_OPEN_SEARCH_PHASE). Reproduced specifically on the first activation of Phase 1 (PHASE_1_DUAL_WRITE_ES_READS).

Severity

Medium - Some functionality impacted

Links

NA

Metadata

Metadata

Type

No fields configured for Bug.

Projects

Status

In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions