Skip to content

Route geo-restricted downloads through a German egress proxy - #47

Merged
ChrisonSimtian merged 1 commit into
mainfrom
feat/geo-proxy-egress
Jul 26, 2026
Merged

Route geo-restricted downloads through a German egress proxy#47
ChrisonSimtian merged 1 commit into
mainfrom
feat/geo-proxy-egress

Conversation

@ChrisonSimtian

Copy link
Copy Markdown
Contributor

What

Completes #45 — the actual geo-aware egress. Building on the detection from #46, the Downloader now routes a geo-restricted job through an in-region (German) proxy, while unrestricted jobs still go direct. Combines the planned Mode A + Mode B into one PR (as requested).

Proven live: the KiKA Biene Maja full download — fenced with a 403 from NZ — now completes end-to-end through a DE egress (verified against a Hetzner Falkenstein proxy: real MP4, >5 MB, in ~90s).

Mode A — bring-your-own proxy (recommended, default)

  • Download:ProxyUrl — a proxy the operator controls (their own DE VPS / WireGuard exit).
  • Dialled via a dedicated proxied HttpClient in RawMp4DownloadProvider (best-first fall-through across candidates on a failed handshake) and via -http_proxy in FfmpegDownloadProvider.
  • A geo-restricted job with no egress configured fails fast with an actionable message.

Mode B — opt-in auto public list

  • Proxy table + IProxyRepository — upsert preserves our probe feedback; GetRankedAsync orders best-first (known-good → uptime → speed → recency). EF migration AddProxyTable.
  • IProxyListSourceGeoNodeProxyListSource (typed HTTP client) + RefreshProxyList Action + ProxyRefreshService (BackgroundService, daily by default, mirrors CrawlSchedulerService). Real fetch outcomes feed back into the ranking.
  • Config: Download:ProxyList:{Enabled,RefreshInterval,SourceUrl,Country,MaxCandidates} — off by default.

Shared

  • IEgressProxyProvider (Domain) selects candidates: BYO first, then the ranked list. Option types live in Domain because both Application and Infrastructure bind them (both may only depend on Domain).
  • Activity page shows a 🌍 DACH badge on geo-restricted jobs (DownloadJobResponse gains GeoRestricted).
  • CLAUDE.md documents the config + the live-test knob.

Tests

  • Unit: egress selection (BYO / empty), GeoNode parsing (+ malformed-row skip), proxy ranking + upsert-preserves-feedback + country filter, refresh handler (upsert / empty-noop), and the raw provider's fail-fast (geo + no egress, no network touched).
  • ./build.cmd Test green — App 50 / Arch 8 / Infra 67 (arch layering still holds).
  • Live: KRAUTWATCH_TEST_PROXY=http://<de-proxy> ./build.cmd TestLive does the real KiKA download; without it the KiKA case proves the fail-fast. Ran the KiKA case through a DE proxy: passed.

Notes / follow-ups

  • No editable Settings form — config is env/appsettings-driven; the UI surfaces geo status read-only. An editable settings UI is separate greenfield.
  • ffmpeg egress uses the single best candidate (no fall-through) — geo-restricted content is progressive MP4 in practice (raw path), so this is an edge case.
  • Public-list proxies are best-effort/untrusted; integrity is gated by the existing ftyp + size check and no credentials transit (public content only).

Plan doc: docs/plans/2026-07-27 - geo-aware-proxy-egress.md. Background: #41.

…ss proxy

Closes the #45 build (Mode A + Mode B): a geo-restricted job (flagged by the
detection landed in #46) is fetched through an in-region proxy; unrestricted
jobs still go direct. Proven live — the KiKA "Biene Maja" full download, fenced
from NZ, now completes through a DE egress.

Mode A — bring-your-own (recommended):
- Download:ProxyUrl, dialled via a dedicated proxied HttpClient in the raw-MP4
  provider (best-first fall-through across candidates) and ffmpeg (-http_proxy).
- A geo-restricted job with no egress configured fails fast with a clear message.

Mode B — opt-in auto public list:
- Proxy table + IProxyRepository (upsert preserves our probe feedback; ranked
  best-first: known-good → uptime → speed → recency) + EF migration.
- IProxyListSource → GeoNodeProxyListSource (typed HTTP client) + RefreshProxyList
  Action + ProxyRefreshService (BackgroundService, daily by default, mirrors
  CrawlSchedulerService). Real fetch outcomes feed back into ranking.
- Config Download:ProxyList:{Enabled,RefreshInterval,SourceUrl,Country,MaxCandidates}.

Shared:
- IEgressProxyProvider (Domain) selects candidates: BYO first, then ranked list.
  Options live in Domain since both App and Infra bind them.
- Activity page shows a "DACH" badge on geo-restricted jobs (DownloadJobResponse
  gains GeoRestricted). CLAUDE.md documents the config.

Tests: egress selection, GeoNode parsing, proxy ranking/upsert-preserves-feedback,
refresh handler, and the raw provider's fail-fast. App 50 / Arch 8 / Infra 67 green.
Live: KRAUTWATCH_TEST_PROXY=<de-proxy> ./build.cmd TestLive does the real KiKA pull.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ChrisonSimtian
ChrisonSimtian merged commit 15cbc69 into main Jul 26, 2026
1 check passed
@ChrisonSimtian
ChrisonSimtian deleted the feat/geo-proxy-egress branch July 26, 2026 14:40
@ChrisonSimtian ChrisonSimtian added the enhancement New feature or request label Aug 2, 2026
@ChrisonSimtian ChrisonSimtian changed the title feat(downloads): geo-aware egress proxy for DACH-restricted downloads (#45 · 2/2) Route geo-restricted downloads through a German egress proxy Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant