Add stable MCP discovery continuation tokens

[Widthdom]

Summary

• add stateless, generation-bound continuation tokens and pagination metadata to MCP symbols, files, and validate
• page generation, totals, and rows in one read snapshot with deterministic ordering and typed cursor errors
• bind tokens to committed writes, readiness, and runtime fold state; keep degraded validation totals and summaries explicitly non-authoritative
• cover exact qualified Rust totals, concurrent clients, same-second commit invalidation, readiness transitions, and cursor byte bounds

Validation

• dotnet restore CodeIndex.sln
• dotnet build CodeIndex.sln -c Release -p:UseSharedCompilation=false --no-restore
• dotnet test CodeIndex.sln -c Release -p:UseSharedCompilation=false --no-build --no-restore
  • .NET 8: 10,672 passed, 7 skipped, 0 failed
  • .NET 9: 10,199 passed, 419 skipped, 0 failed
• dotnet test tests/CodeIndex.Tests/CodeIndex.Tests.csproj -c Release -p:UseSharedCompilation=false --no-build --no-restore --filter 'FullyQualifiedName~Issue4853'
• dotnet test tests/CodeIndex.Tests/CodeIndex.Tests.csproj -c Release -p:UseSharedCompilation=false --no-build --no-restore --filter 'FullyQualifiedName~Issue4541'
• dotnet test tests/CodeIndex.Tests/CodeIndex.Tests.csproj -c Release -p:UseSharedCompilation=false --no-build --no-restore --filter 'FullyQualifiedName~McpToolContractTests'
• dotnet format CodeIndex.sln --verify-no-changes --no-restore
• dotnet run --project tools/CodeIndex.Changelog -- check
• codex review --base origin/main

Documentation and changelog

• updated the English and Japanese MCP pagination/testing guidance
• added changelog.d/unreleased/4853.added.md

Follow-up candidates

• none

Fixes #4853