fix: eliminate HuggingFace rate-limit failures in model downloads

[shreeraman96]

Why is this change needed?

Fixes altic-dev/FluidVoice#683. Anonymous HuggingFace clients get 500 API requests per 5-minute fixed window per IP (the api bucket; verified via ratelimit-policy headers). Downloading Parakeet v3 listed one directory per API call (41 calls measured), and any failure wiped the cache and re-ran the burst — AsrManager model loads plus the consumer's retry made a single download gesture cost ~250 listing calls, saturating the window and making every subsequent attempt fail with "Rate limited while listing files" for all engines.

Changes

• Replace the per-directory HuggingFace tree walk in downloadRepo/downloadSubdirectory with a single ?recursive=true listing, with RFC 5988 Link: rel="next" pagination (100-page cap + cycle detection).
• Extract file selection into a pure selectFilesToDownload(tree:patterns:subPath:) that reproduces the old walk's pruning + inclusion semantics exactly.
• Add rate-limit-aware retries for 429/503 on listing, file downloads, and fetchHuggingFaceFile: delay from the ratelimit header's t= (seconds to window reset), else Retry-After, else exponential backoff, clamped to [0.5s, 300s]; CancellationError and URLError.cancelled propagate immediately.
• Gate loadModels' wipe-and-retry: rate-limit, HTML-error-page, network, and cancellation errors rethrow without deleting the cache (partial downloads stay resumable); corruption/compile failures keep the wipe + single retry.
• Fall back to ~/.cache/huggingface/token (HuggingFace CLI login) when the token env vars are absent, and send a User-Agent.
• Add DownloadUtilsRateLimitTests (15 tests) with a URLProtocol stub via an internal sessionOverride seam.

Repro evidence (offline stub HF server via REGISTRY_URL, real Parakeet v3 tree)

Before this change:

Scenario	Result
Healthy server, one downloadRepo	41 listing API calls (1 needed)
All API calls 429	Instant hard failure: "Hugging Face rate limit encountered: Rate limited while listing files" (exact issue FluidInference#683 error)
45-call budget + corrupt model	First pass fits (41 calls), compile failure triggers cache wipe + full re-list → blows budget at call FluidInference#46 → surfaces as a rate-limit error

After this change:

Scenario	Result
Healthy server	Succeeds with 1 listing call; downloaded file set byte-identical to the old walk (23/23 files)
All API calls 429 (t=1)	3 header-paced retries, then a clean rate-limit error; cache untouched
45-call budget + corrupt model	Wipe-and-retry recovery still works (2 listing calls total) and surfaces the honest compile error

Verification

• swift test --filter DownloadUtilsRateLimitTests: 15/15 pass (single-call listing, pagination union, header-aware retry, retry exhaustion message, no-cache-wipe on 429 and on URLError.timedOut, filter-equivalence for subPath/non-subPath/pruned-ancestor cases).
• swift build and swift format lint clean.

Consumer impact

Public API unchanged (internal test seam only), so FluidVoice and other consumers build as-is; FluidVoice picks the fix up via its B/cohere-coreml-asr branch pin. A companion FluidVoice PR gates the app-level cache-clear recovery the same way.