Index OpenAPI endpoints for Elasticsearch and Kibana

[Mpdreamz]

Summary

This PR adds support for indexing OpenAPI documentation from Elasticsearch and Kibana OpenAPI specification endpoints, making API reference documentation searchable alongside existing documentation.

This is using our OpenAPI Specification Mirror on S3/Cloudfront to ensure we always have the latest built ready to go.

NOTE this is temporary, once we have an OpenAPI explorer they will flow more naturally through our architecture.

Key Features

1. OpenAPI Document Exporter

• New class: OpenApiDocumentExporter that fetches OpenAPI specs from CloudFront URLs
• Processes both Elasticsearch (elasticsearch-openapi-docs.json) and Kibana (kibana-openapi.json)
• Converts each API operation to a searchable DocumentationDocument
• URL format: /docs/api/doc/{product}/operation/operation-{operationId}

2. Version Filtering

• Reads x-state extension from OpenAPI operations
• Parses "Added in X.Y.Z" version information
• Filters operations to only include those where added version ≤ current product version
• Safe default: includes operations without x-state metadata

3. ApplicableTo Extraction

• Extracts lifecycle information (GA, Beta, Tech Preview, etc.) from x-state
• Parses version information and creates ApplicableTo instances
• Sets Stack applicability on DocumentationDocument for proper version filtering

4. HTML-to-Markdown Transformation

• Detects HTML operation lists in descriptions (e.g., "All methods and paths for this operation:")
• Transforms HTML <div> and <span> elements to clean markdown lists
• Moves the markdown list to the end of the description for better readability
• Format: - **VERB** \path``

5. LLM-Ready Content

• Generates stripped markdown versions for better LLM processing
• Creates abstract summaries from body content and headings
• Optimizes content for semantic search and AI interactions

6. Type-Based Search Result Icons

• Added type field to DocumentationDocument (default: "doc", or "api" for API docs)

NOTE: Will follow up with introducing dynamic styling based on type.

Testing

• Integration test: Validates 300 documents from both Elasticsearch and Kibana
  • Skipped for now, leaving this on to be able to run locally.
• Verifies URLs return 200 OK on elastic.co
• Tests HTML-to-markdown transformation
• Confirms markdown lists appear at the end of descriptions