Add article: upgrading the search domain engine (Elasticsearch 6.8 → 7.10)

[sir-sigurd]

New KB article for deployments whose search domain stayed on Elasticsearch 6.8 while the deployed template declares 7.10 — an engine upgrade fired during a stack update can fail on a transient (typically an automated-snapshot collision), and CloudFormation records the deploy successful without ever retrying, leaving the domain permanently behind.

The article gives the direct CLI path: parameterized commands, read-only until the single upgrade trigger, eligibility-check-then-fire sequencing that launches into the gap between hourly automated snapshots (the passing check doubles as the no-snapshot-running signal), a stall rule with AWS escalation links, and a catalog smoke test. Generic AWS mechanics are linked to AWS docs rather than restated.

Written with a specific support case in hand; the sequence has been rehearsed end-to-end on a scratch staging domain (~50 min on an empty domain; real outputs folded into the article), with the production run next.

🤖 Generated with Claude Code

Greptile Summary

The PR adds a runbook for directly upgrading a Quilt search domain from Elasticsearch 6.8 to 7.10 after CloudFormation leaves its live engine behind.

• Covers compatibility and Auto-Tune checks, asynchronous eligibility polling, and the state-changing upgrade command.
• Adds monitoring, escalation, and catalog smoke-test guidance.
• The initial shell variable block must be corrected before readers can execute the procedure as written.

Confidence Score: 4/5

The article should not merge until its required shell variable assignments are made executable, because the procedure currently fails at the first setup step.

The documented placeholder assignments are parsed as shell redirections rather than variable values, leaving the parameters required by every read-only check and the upgrade trigger unset.

Files Needing Attention: howto-upgrade-search-domain-engine.md

Important Files Changed

Filename	Overview
howto-upgrade-search-domain-engine.md	Adds the complete operational upgrade runbook, but its required DOMAIN and REGION setup lines are invalid shell syntax when copied as shown.

Sequence Diagram

sequenceDiagram
    actor Operator
    participant AWS as AWS Elasticsearch Service
    participant Catalog as Quilt Catalog
    Operator->>AWS: Check compatible versions and Auto-Tune
    Operator->>AWS: Run eligibility check
    loop Until check completes
        Operator->>AWS: Get upgrade status
        AWS-->>Operator: Current check status
    end
    Operator->>AWS: Trigger engine upgrade
    loop Until upgrade completes
        Operator->>AWS: Get upgrade status
        AWS-->>Operator: PRE_UPGRADE_CHECK / SNAPSHOT / UPGRADE
    end
    Operator->>Catalog: Upload file and verify search

Loading

_{Reviews (1): Last reviewed commit: "Add article: upgrading the search domain..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

[greptile-apps]

Invalid shell variable assignments

When a reader copies this setup block before replacing the example text, the shell interprets < and > as redirection operators instead of assigning DOMAIN and REGION, causing the setup and all subsequent AWS commands to fail.

Suggested change

	DOMAIN=<your search domain name>
	REGION=<your stack's region>
	DOMAIN="your-search-domain-name"
	REGION="your-stack-region"

[Copilot]

Pull request overview

Adds a new Knowledge Base runbook describing how to manually upgrade a Quilt deployment’s AWS-managed search domain from Elasticsearch 6.8 to 7.10 when CloudFormation reports a successful stack update but the live domain engine version never advances.

Changes:

• Introduces a step-by-step CLI procedure to verify drift, run a pre-upgrade eligibility check, trigger the engine upgrade, and monitor progress.
• Documents operational guardrails (freeze window, failure modes, stall triage/escalation) and a post-upgrade catalog smoke test.
• Links to AWS reference docs for migration mechanics and troubleshooting.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.