Search Relevance testing infrastructure

[Mpdreamz]

Add Search Integration Tests with Elasticsearch Explain API

Why

Search relevance is critical for documentation discoverability, but debugging why certain documents rank higher than others has been a black box. When search results don't match expectations, we need detailed insights into Elasticsearch's scoring decisions to improve our search queries and boost factors.

Additionally, as we continue to refine our hybrid search implementation (combining lexical and semantic search with RRF), we need automated tests that not only verify correct behavior but also help us understand and improve search ranking over time.

How

This PR introduces a comprehensive search testing infrastructure with two complementary test classes:

1. Infrastructure Changes

ElasticsearchGateway Refactoring (ElasticsearchGateway.cs):

• Extracted query building logic into reusable static methods to eliminate duplication:

  • BuildLexicalQuery() - Encapsulates traditional text search with multiple match types and boost factors
  • BuildSemanticQuery() - Handles semantic search using semantic_text fields
  • NormalizeSearchQuery() - Query normalization (e.g., "dotnet" → "net")

  This extraction serves dual purposes: DRY principle and enabling the explain functionality to use the exact same queries as production searches.

• Implemented Elasticsearch Explain API integration:

  • ExplainDocumentAsync() - Uses Elasticsearch's _explain API to get detailed scoring breakdown for why a document matched (or didn't match) a query
  • ExplainTopResultAndExpectedAsync() - Compares actual top result with expected result, providing side-by-side scoring analysis
  • FormatExplanation() - Recursively formats Elasticsearch's ExplanationDetail tree into human-readable indented output
  • ExplainResult record - Strongly-typed container for explain results (Found, Matched, Score, Explanation)

SearchBootstrapFixture (SearchTestBase.cs):

• Implemented intelligent indexing strategy that checks if remote Elasticsearch already has up-to-date data by comparing semantic channel hashes
• Only triggers indexing when necessary, significantly reducing test execution time
• Shares fixture across multiple test classes using xUnit collection fixtures
• Validates successful indexing by checking exit codes and resource states

2. Test Classes

SearchIntegrationTests - Black-box API testing:

• Tests the public HTTP API endpoint (/docs/_api/v1/search)
• Verifies end-to-end behavior including pagination, error handling, and response formatting
• Ensures the API layer correctly interfaces with ElasticsearchGateway

SearchRelevanceTests - White-box relevance testing with explain output:

• Uses ElasticsearchGateway directly to bypass HTTP layer

• When a test fails (first result doesn't match expected), automatically:

  1. Fetches detailed explain for the actual top result
  2. Fetches detailed explain for the expected result
  3. Outputs formatted scoring breakdowns showing exactly why Elasticsearch ranked them differently
  4. Calculates and displays score differences

• Includes Assert.SkipUnless(searchFixture.Connected) to gracefully handle Elasticsearch unavailability

• Uses the same test cases as SearchIntegrationTests for consistency

3. Configuration & DI

TestParameterProvider:

• Implements IParameterProvider interface for test scenarios
• Bridges the gap between test configuration (user secrets, Aspire config, environment variables) and production parameter providers (AWS Parameter Store, Lambda Extension)
• Follows the same fallback chain pattern used in production

Test Output Example

When a search relevance test fails, developers see:

❌ FIRST RESULT MISMATCH - Fetching detailed explanations...

═══════════════════════════════════════════════════════════════
ACTUAL TOP RESULT: /docs/reference/elasticsearch/clients/python/getting-started
Score: 0.0315
Matched: True
───────────────────────────────────────────────────────────────
Scoring Breakdown:
  0.0315 - max of:
    0.0234 - weight(title:elasticsearch in 1234) [PerFieldSimilarity], result of:
      0.0234 - score(freq=1.0), computed from:
        10.0000 - boost
        2.3400 - idf, computed as log(1 + (N - n + 0.5) / (n + 0.5))
        0.0010 - tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl))
    0.0189 - weight(abstract:elasticsearch in 1234) [PerFieldSimilarity]
      ...

═══════════════════════════════════════════════════════════════
EXPECTED RESULT: /docs/reference/elasticsearch/clients/java/getting-started
Score: 0.0289
Matched: True
───────────────────────────────────────────────────────────────
Scoring Breakdown:
  0.0289 - max of:
    0.0212 - weight(title:elasticsearch in 5678) [PerFieldSimilarity]
      ...

This enables data-driven decisions about boost factors, query types, and field weights.

Current Status

All 15 search integration tests passing:

• 7 SearchIntegrationTests (HTTP API)
• 8 SearchRelevanceTests (direct gateway with explain capability)

The tests currently pass because search results match expectations, but the infrastructure is ready to provide detailed diagnostics the moment search relevance needs improvement.

[github-actions]

🔍 Preview links for changed docs

• aspire/README.md

[reakaleek]

LGTM 🚀

[reakaleek]

Q: I can see this string 4 times across files. Is this just coincidence or should we put this into a constant?

[Mpdreamz]

Its the secret guid defined in aspire.csproj.

Will follow up ensuring this is a constant!