fix(opensearch): preserve OS index store on phase rollback during in-flight reindex (#36471)

[swicken]

Proposed Changes

Fixes #36471 — rolling FEATURE_FLAG_OPEN_SEARCH_PHASE back to 0 while a full reindex is draining the journal wiped the OS side of the indicies store and stranded a partial .os reindex pair on the cluster; a later boot catchup adopted the partial pair as active, so Phase-2 searches silently returned a fraction of the content (measured 8/24, 0/6 on the demo starter) with zero errors.

Root cause — IndiciesFactory.point() (the legacy ES store) deleted rows by index_type across all index_version values, while only ever reading its own NULL-version rows. Every ES switchover therefore wiped the OS rows (index_version='os-3.x') in the shared table. In Phases 1/2 the wipe was invisible because the OS mirror immediately re-wrote the rows — and only found the values to re-write via the stale VersionedIndicesCache, which point() doesn't flush. In Phase 0 the mirror is skipped, so the rows stayed gone.

• Scope point()'s delete with index_version IS NULL so the legacy store only manages the rows it owns. This also removes the Phase-1/2 dependence on the stale-cache accident.
• Treat OS reindex slots encountered during a Phase-0 switchover or abort as a stranded mid-reindex rollback (abortStrandedOsReindex): clear the slots first (a leftover slot flips isInFullReindex() back to true on a later move to Phase 2, which would switch ES over onto null pointers), then delete the partial physical .os indices so no boot catchup can ever adopt them. Logged at WARN with the deleted index names — no silent path remains.
• New ContentletIndexAPIImplMidReindexRollbackIT in OpenSearchUpgradeSuite covering the point() row-preservation contract plus the Phase-0 switchover and abort paths (3 tests, green).
• OPENSEARCH_MIGRATION.md: documented the mid-reindex rollback behavior and operational rule under "Rollback risk during dual-write phases".

Verification — reproduced live on the single-node-os-migration example (released image: 4→2 rows, partial orphans 201/200 docs, catchup adoption after restart, Phase-2 partial results). Same sequence on the fixed image: .os working/live rows survive, the partial pair is deleted at switchover with a WARN, restart adopts nothing, and Phase-2 counts equal Phase-0 counts exactly (24/24, 6/6, 26/26).

Follow-ups deliberately out of scope: a catchup adoption sanity-check (doc-count vs ES twin) and operator-facing phase-change validation while isInFullReindex() — both defense-in-depth beyond the damage-site fix.

Checklist

• Tests
• Translations (n/a)
• Security implications contemplated (change respects store ownership boundaries; no new input surface)

This PR fixes: #36471

[claude]

Claude finished @swicken's task in 9m 12s —— View job

---

Rollback-Safety Analysis

• Read docs/core/ROLLBACK_UNSAFE_CATEGORIES.md
• Get full PR diff (eb15ac31...5049c6e9)
• Analyze diff against every unsafe category
• Apply label

Verdict: ✅ Safe to Rollback

This PR is pure application-code logic — no runonce DB migration task, no ES/OS mapping change, no schema/DDL, no REST/GraphQL contract change, no OSGi interface change.

• IndiciesFactory.java — scopes the existing DELETE from indicies where index_type=? or index_name=? to ... and index_version is null. This changes only the runtime query predicate, not the indicies table schema. No columns/tables added, renamed, or dropped.
• ContentletIndexAPIImpl.java — adds abortStrandedOsReindex(), called from the existing Phase-0 switchover/abort paths. It only touches rows in indicies/OS versioned-index metadata via existing VersionedIndicesAPI methods (saveIndices, removeVersion) and deletes physical OS cluster indices — no data model, PK, or storage-provider change.
• OPENSEARCH_MIGRATION.md, test additions, suite registration — docs/tests only.

None of C-1…C-4, H-1…H-8, or M-1…M-4 apply: there's no structural storage change, no ES mapping change, no contentlet_as_json version bump, no DROP/RENAME, no PK restructuring, no new field type, no storage-provider switch, no stored-procedure drop, no NOT NULL-without-default column, no VTL/REST/GraphQL/OSGi contract change.

Rolling back to N-1 simply restores the prior (buggy) unscoped-delete/no-abort behavior described in the issue — it does not leave N-1 unable to start, read data, or serve requests. Label AI: Safe To Rollback applied.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request