Route geo-restricted downloads through a German egress proxy - #47
Merged
Conversation
…ss proxy Closes the #45 build (Mode A + Mode B): a geo-restricted job (flagged by the detection landed in #46) is fetched through an in-region proxy; unrestricted jobs still go direct. Proven live — the KiKA "Biene Maja" full download, fenced from NZ, now completes through a DE egress. Mode A — bring-your-own (recommended): - Download:ProxyUrl, dialled via a dedicated proxied HttpClient in the raw-MP4 provider (best-first fall-through across candidates) and ffmpeg (-http_proxy). - A geo-restricted job with no egress configured fails fast with a clear message. Mode B — opt-in auto public list: - Proxy table + IProxyRepository (upsert preserves our probe feedback; ranked best-first: known-good → uptime → speed → recency) + EF migration. - IProxyListSource → GeoNodeProxyListSource (typed HTTP client) + RefreshProxyList Action + ProxyRefreshService (BackgroundService, daily by default, mirrors CrawlSchedulerService). Real fetch outcomes feed back into ranking. - Config Download:ProxyList:{Enabled,RefreshInterval,SourceUrl,Country,MaxCandidates}. Shared: - IEgressProxyProvider (Domain) selects candidates: BYO first, then ranked list. Options live in Domain since both App and Infra bind them. - Activity page shows a "DACH" badge on geo-restricted jobs (DownloadJobResponse gains GeoRestricted). CLAUDE.md documents the config. Tests: egress selection, GeoNode parsing, proxy ranking/upsert-preserves-feedback, refresh handler, and the raw provider's fail-fast. App 50 / Arch 8 / Infra 67 green. Live: KRAUTWATCH_TEST_PROXY=<de-proxy> ./build.cmd TestLive does the real KiKA pull. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Completes #45 — the actual geo-aware egress. Building on the detection from #46, the Downloader now routes a geo-restricted job through an in-region (German) proxy, while unrestricted jobs still go direct. Combines the planned Mode A + Mode B into one PR (as requested).
Proven live: the KiKA Biene Maja full download — fenced with a
403from NZ — now completes end-to-end through a DE egress (verified against a Hetzner Falkenstein proxy: real MP4, >5 MB, in ~90s).Mode A — bring-your-own proxy (recommended, default)
Download:ProxyUrl— a proxy the operator controls (their own DE VPS / WireGuard exit).HttpClientinRawMp4DownloadProvider(best-first fall-through across candidates on a failed handshake) and via-http_proxyinFfmpegDownloadProvider.Mode B — opt-in auto public list
Proxytable +IProxyRepository— upsert preserves our probe feedback;GetRankedAsyncorders best-first (known-good → uptime → speed → recency). EF migrationAddProxyTable.IProxyListSource→GeoNodeProxyListSource(typed HTTP client) +RefreshProxyListAction +ProxyRefreshService(BackgroundService, daily by default, mirrorsCrawlSchedulerService). Real fetch outcomes feed back into the ranking.Download:ProxyList:{Enabled,RefreshInterval,SourceUrl,Country,MaxCandidates}— off by default.Shared
IEgressProxyProvider(Domain) selects candidates: BYO first, then the ranked list. Option types live in Domain because both Application and Infrastructure bind them (both may only depend on Domain).DownloadJobResponsegainsGeoRestricted).CLAUDE.mddocuments the config + the live-test knob.Tests
./build.cmd Testgreen — App 50 / Arch 8 / Infra 67 (arch layering still holds).KRAUTWATCH_TEST_PROXY=http://<de-proxy> ./build.cmd TestLivedoes the real KiKA download; without it the KiKA case proves the fail-fast. Ran the KiKA case through a DE proxy: passed.Notes / follow-ups
ftyp+ size check and no credentials transit (public content only).Plan doc:
docs/plans/2026-07-27 - geo-aware-proxy-egress.md. Background: #41.