Add adaptive embedding throughput shaping for Azure 429 limits

[BenjaminMichaelis]

Why

The previous retry-only fix still failed under sustained S0 throttling: large embedding requests kept exhausting retries at the same payload size. We need throughput shaping so rebuilds can continue progressing under rate limits instead of stalling at repeated 429 exhaustion.

What changed

• Added adaptive batch downshifting in embedding rebuilds:
  • starts at configured max batch size
  • on 429/RateLimitReached, splits throttled batches and retries smaller sub-batches
  • reuses the smaller successful size for subsequent requests in the same run
  • fails clearly if batch size 1 still exhausts retries
• Added explicit request pacing controls:
  • AIOptions:EmbeddingRetry:MaxEmbeddingBatchSize (default 2048)
  • AIOptions:EmbeddingRetry:MinInterRequestDelayMs (default 250)
  • embedding requests are serialized and paced between calls to reduce sustained RPM pressure
• Hardened Retry-After parsing:
  • supports retry-after, retry-after-ms, x-ms-retry-after-ms
  • supports extracting retry after N seconds from exception message text
• Added coarse progress logging during rebuilds (not per call):
  • logs start configuration
  • logs progress at 10% milestones when total count is known
  • falls back to every 500 chunks when total count is unknown
  • includes current adaptive batch size in progress messages

Validation

• dotnet build EssentialCSharp.Chat.Shared/EssentialCSharp.Chat.Common.csproj -c Release --nologo
• dotnet test EssentialCSharp.Chat.Tests/EssentialCSharp.Chat.Tests.csproj -c Release --no-restore -v q

Both passed.

[Copilot]

Pull request overview

Improves resilience of Azure OpenAI embedding rebuilds under sustained throttling by introducing adaptive batch downshifting, request pacing, and more robust Retry-After handling so rebuilds can continue progressing instead of repeatedly exhausting retries.

Changes:

• Added adaptive batch splitting/downshifting on 429/RateLimitReached during embedding rebuild uploads.
• Serialized and paced embedding requests with a configurable minimum inter-request delay.
• Hardened Retry-After parsing (more header variants + message parsing) and added coarse rebuild progress logging.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
EssentialCSharp.Web/appsettings.json	Adds default configuration values for max embedding batch size and inter-request pacing delay.
EssentialCSharp.Chat.Shared/Services/EmbeddingService.cs	Implements pacing/serialization, adaptive batch downshifting on throttling, improved Retry-After extraction, and rebuild progress logging.
EssentialCSharp.Chat.Shared/Models/EmbeddingRetryOptions.cs	Introduces new retry/pacing configuration knobs with validation.
EssentialCSharp.Chat.Shared/Extensions/ServiceCollectionExtensions.cs	Clarifies configuration override semantics for the embedding retry options binding.