Improve Search Relevance with new query strategy

[Mpdreamz]

Improve Search Relevance with Refined Query Strategy

This PR overhauls the Elasticsearch query implementation to significantly improve search result relevance across the documentation site. The changes focus on better field weighting, improved tokenization, and more targeted query matching.

Strategic Approach: Precision First, Recall Later

This PR temporarily removes RRF (Reciprocal Rank Fusion) and semantic search to establish a new baseline with comprehensive test coverage. The previous implementation combined lexical and semantic search through RRF, making it difficult to debug relevance issues and understand why certain results ranked as they did.

By focusing on lexical search precision first, we can:

1. Build a robust test suite with expected results for common queries
2. Understand and optimize the baseline query behavior
3. Establish clear relevance benchmarks
4. Later reintroduce semantic search + RRF with larger recall once precision is validated

This approach follows the information retrieval principle: optimize precision before expanding recall. Semantic search will be reintroduced after we have confidence in the lexical baseline.

Key Search Relevance Improvements

1. Streamlined Query Structure

The previous implementation used RRF combining 13+ separate query clauses with complex boolean logic. This created unpredictable scoring where short titles (like "ECS") could dominate results due to high term frequency scores, making it nearly impossible to debug why specific documents ranked higher.

The new implementation uses a focused multi-match strategy:

• Primary matching via MultiMatchQuery with bool_prefix type on new search_title completion fields (boost: 2.0)
• Content matching on stripped_body with reduced weight (boost: 0.2) to prevent body text from overwhelming title relevance
• URL matching for single-word queries to ensure direct matches (e.g., "templates") rank appropriately

2. New search_title Field

Introduced a dedicated search_title field that combines the document title with URL components:

• Prevents documents with very short titles from receiving disproportionate relevance scores
• Uses search_as_you_type mapping with n-gram subfields (_2gram, _3gram) for better prefix matching
• Indexed with synonym analysis to handle abbreviations and alternative terms

3. Enhanced Text Analysis Pipeline

Indexing improvements:

• Added kstem filter to the synonyms analyzer for better stemming (e.g., "getting" → "get")
• Expanded synonym dictionary with new mappings:
  • sso ↔ single sign-on
  • querydsl ↔ query dsl

Field-specific optimizations:

• search_title and title now use search_as_you_type for autocomplete-style matching
• Consistent synonyms_analyzer application across searchable fields

4. Boosting and Dampening Strategy

Positive signals:

• Multi-match on completion fields (2.0x boost) - prioritizes title/URL matches
• Single-word URL matches wrapped in constant_score (1.0x) - prevents over-optimization

Negative signals:

• BoostingQuery with negative dampening (0.8x) for generic terms:
  • Documents with "plugin", "client", or "integration" in title/headings/URL receive reduced scores
  • This prevents overly generic integration/plugin docs from dominating specific feature queries

5. Edge Case Handling

Added special handling for ambiguous terms:

• datastream/datastreams/data-stream → automatically expanded to "data streams"
  • Fixes N-of-1 corpus problem where a single page used non-standard terminology

Testing Infrastructure Enhancements

New Search.IntegrationTests Project

• Dedicated integration test suite for search relevance validation
• Comprehensive test cases covering common search patterns:
  • Language-specific queries (c# client, dotnet client)
  • Acronym searches (ecs, sso)
  • Multi-word phrases (elasticsearch getting started)
  • Variant spellings (data-streams, datastream)

Extended TheoryData Support

• Tests now support optional additional expected URLs for first-page verification
• Enables testing that related results appear together (e.g., "runscript" should show both operation docs and security action docs)
• Better validation of result quality beyond just the top result

Elasticsearch Connection Guard

• Tests now skip gracefully when Elasticsearch is unavailable (SkipUnless attribute)
• Improved developer experience for running tests locally

Breaking Changes

• Temporarily removed RRF retriever: Simplified to single query with post-filter for better explainability
• Semantic search disabled: Will be reintroduced after precision baseline is established
• Removed NormalizeSearchQuery: The "dotnet" → "net" transformation is now handled by synonyms

Migration Notes

This change requires reindexing to populate the new search_title field. The field is generated during indexing by combining title and URL components.

Results

Test suite now validates 7+ different search patterns with expected first-page results. The new query strategy produces more intuitive results where:

• Direct title matches rank highest
• Generic integration/plugin docs don't overshadow specific features
• Abbreviations and synonyms work consistently
• Related documentation appears together on the first page

Next Steps

Once the precision baseline is validated in production:

1. Reintroduce semantic search for better natural language understanding
2. Re-enable RRF to combine lexical + semantic signals
3. Expand test coverage based on production search analytics
4. Optimize for recall while maintaining precision standards

[reakaleek]

We need to re-add the changes from #2277 in ElasticsearchGateway.cs

[reakaleek]

We might want to move this logic into a separate class.

If we find additional edge-case handling like this.

[Mpdreamz]

++ I will follow up with making this a new index time synonym list. I just need to make sure that when that list is updated we reindex everything automatically (like we do for all setting/mapping changes).

[Mpdreamz]

@reakaleek #2277 is re-applied after fixing the merge conflict.