Detect DACH geo-restricted episodes at crawl time - #46
Merged
Conversation
… model First increment of #45 — surface each asset's broadcaster-declared geo restriction so the Downloader can later route only restricted jobs through a German egress. No behaviour change yet; the flag is recorded, nothing routes. - Read at resolve time: ARD mediaCollection.embedded.isGeoBlocked; ZDF PTMD attributes.geoLocation (anything != "none" = in-region-only). - EpisodeDetail.GeoRestricted → Episode.GeoRestricted (persisted) via EpisodeMapper → snapshot onto DownloadJob.GeoRestricted at enqueue. - EF migration AddGeoRestricted (two boolean NOT NULL DEFAULT false columns). - Tests: mapper carries the flag; AddDownloadByToken snapshots it; live fetch tests assert real data — Biene Maja restricted, Extra 3 / heute-show not. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ChrisonSimtian
added a commit
that referenced
this pull request
Jul 26, 2026
…ss proxy Closes the #45 build (Mode A + Mode B): a geo-restricted job (flagged by the detection landed in #46) is fetched through an in-region proxy; unrestricted jobs still go direct. Proven live — the KiKA "Biene Maja" full download, fenced from NZ, now completes through a DE egress. Mode A — bring-your-own (recommended): - Download:ProxyUrl, dialled via a dedicated proxied HttpClient in the raw-MP4 provider (best-first fall-through across candidates) and ffmpeg (-http_proxy). - A geo-restricted job with no egress configured fails fast with a clear message. Mode B — opt-in auto public list: - Proxy table + IProxyRepository (upsert preserves our probe feedback; ranked best-first: known-good → uptime → speed → recency) + EF migration. - IProxyListSource → GeoNodeProxyListSource (typed HTTP client) + RefreshProxyList Action + ProxyRefreshService (BackgroundService, daily by default, mirrors CrawlSchedulerService). Real fetch outcomes feed back into ranking. - Config Download:ProxyList:{Enabled,RefreshInterval,SourceUrl,Country,MaxCandidates}. Shared: - IEgressProxyProvider (Domain) selects candidates: BYO first, then ranked list. Options live in Domain since both App and Infra bind them. - Activity page shows a "DACH" badge on geo-restricted jobs (DownloadJobResponse gains GeoRestricted). CLAUDE.md documents the config. Tests: egress selection, GeoNode parsing, proxy ranking/upsert-preserves-feedback, refresh handler, and the raw provider's fail-fast. App 50 / Arch 8 / Infra 67 green. Live: KRAUTWATCH_TEST_PROXY=<de-proxy> ./build.cmd TestLive does the real KiKA pull. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
5 tasks
ChrisonSimtian
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
First of three increments for #45 (geo-aware proxy egress). Surfaces each asset's broadcaster-declared geo restriction through the model so the Downloader can later route only restricted jobs through a German egress.
No behaviour change yet — the flag is detected and persisted, but nothing routes on it. That lands in PR 2 (Mode A:
Download:ProxyUrl+ geo-aware routing).How
mediaCollection.embedded.isGeoBlocked(bool)attributes.geoLocation.value— restricted when!= "none"(e.g."dach","de")EpisodeDetail.GeoRestricted→Episode.GeoRestricted(persisted) viaEpisodeMapper→ snapshot ontoDownloadJob.GeoRestrictedat enqueue (rides the same rails as the existingStreamUrl/Qualitysnapshot).AddGeoRestricted— twoboolean NOT NULL DEFAULT falsecolumns (existing rows default to false; they get re-crawled anyway).Tests
EpisodeMappercarries the flag;AddDownloadByTokensnapshots it onto the job../build.cmd TestLive): the three fetch tests now assert the flag against real API data — Biene Maja (KiKA)= true, Extra 3 (ARD) / heute-show (ZDF)= false. Verified locally:Passed: 3../build.cmd Test(deterministic) green.Background
Why this is the right signal, and the proven bypass, are documented in #41 (investigation) and the #45 thread. Plan doc:
docs/plans/2026-07-27 - geo-aware-proxy-egress.md.