Add ARD and ZDF crawler clients - #34
Merged
Merged
Conversation
First real crawler slice — proves search + fetch against the live ARD Mediathek API.
- Infrastructure/Crawling/Ard/ArdCatalogClient: resolves shows the way the site does
(ARD's search-system returns sparse, non-resolving refs). Walks the A-Z catalog
(letter widget id = base64("<Brand>.<letter>")) → filters by title → the editorial
show page → the "Ganze Folgen" gridlist. Parameterised by catalog scope: `ard` and
`kika` (KiKA runs on the same platform, DR-010/#10). Light transient retry.
- tests/Live.Tests: real-network tests tagged [Trait("Category","Live")]:
- ARD "Extra 3": search finds it; fetch full episodes with sane metadata.
- KiKA "Biene Maja": search finds it (kika scope); fetch episodes.
- Download: [Skip] (long-running + storage; needs stream-resolve + ffmpeg — next).
Runs sequentially (no self-throttling).
- Fallout: the CI Test target now filters out Category=Live (external APIs drift), and
a new TestLive target runs only the live tests: ./build.cmd TestLive.
Default ./build.cmd Test = 62 deterministic tests green (live excluded); ./build.cmd
TestLive = 4 live pass + 1 skip.
Next: ZDF "Heute Show" (GraphQL + token + PTMD), the download slice, and wiring the
crawl Actions into the agents.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the ZDF broadcaster (the third acceptance show) and a genuine download.
- Infrastructure/Crawling/Zdf/ZdfCatalogClient: search via the REST /search/documents
endpoint (episodes come back directly), then resolve a stream episode-doc →
ptmd-template ({playerId} expansion) → PTMD priorityList → best progressive MP4.
Carries the static Api-Auth Bearer key (rotates; documented). Transient retry.
- tests/Live.Tests ZdfLiveTests [Live]:
- "Heute Show": search finds episodes; resolve a progressive MP4;
- Downloads_a_HeuteShow_episode — REAL download of the resolved MP4, bounded to ~5 MB
(fast + small), verified by size + the MP4 'ftyp' signature, then cleaned up.
Full live suite (./build.cmd TestLive): 7 pass + 1 skip (the ARD download placeholder,
kept until ARD stream-resolve lands). CI Test run unchanged — live still excluded.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ChrisonSimtian
changed the title
Contributor
Author
|
Added ZDF Heute Show + a real download. Full live suite now (
CI |
Not just listing/streaming — actually fetch one real full episode per broadcaster and return the complete normalized record. - Infrastructure/Crawling/EpisodeDetail: shared normalized episode (title, show, broadcaster, air date, duration, synopsis, progressive-MP4 stream, webvtt subtitle). - ArdCatalogClient.FetchEpisodeDetailAsync(ArdEpisode): item page → metadata + mcV6 streams[].media[] (best video/mp4 by resolution) + subtitles (webvtt). Covers ARD (Extra 3) and KiKA (Biene Maja). - ZdfCatalogClient.FetchEpisodeDetailAsync(ZdfEpisode): episode doc → title/brand/ editorialDate/duration/leadParagraph + resolved MP4. - tests/Live.Tests/FullEpisodeFetchTests: fetch ONE full episode per show and assert the complete data — Extra 3 (NDR, >20 min, synopsis, stream, subtitle), Biene Maja (KiKA, >5 min, stream), Heute Show (ZDF, >20 min, synopsis, stream). TestLive: 10 pass + 1 skip. CI Test unchanged (live excluded). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Un-skips the ARD download now that FetchEpisodeDetail yields the progressive MP4. - tests/Live.Tests/Download.VerifyRawMp4Async: shared helper — downloads the stream RAW (a straight byte copy of the exact MP4 the Mediathek serves; no transcoding, no ffmpeg — any HLS remux is a later orchestration step), bounded to ~5 MB, verified by the MP4 'ftyp' box. - ARD "Extra 3" download is now a real download (was [Skip]), via the resolved progressive MP4 from the episode detail. TestLive: all live green (ARD + ZDF both do a real raw download). CI Test unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ChrisonSimtian
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first real crawler slice — proves search + fetch against the live ARD Mediathek API, with your shows as the acceptance cases.
What's here
Infrastructure/Crawling/Ard/ArdCatalogClient— resolves shows the way the site actually does it. ARD'ssearch-systemreturns sparse refs that don't resolve, so it walks the A-Z catalog (letter widget id =base64("<Brand>.<letter>")) → filters by title → the editorial show page → the "Ganze Folgen" gridlist. Parameterised by catalog scope:ardandkika(KiKA runs on the same platform — DR-010 / KiKA scoping within the ARD client #10). Light transient retry.tests/Live.Tests— real-network tests,[Trait("Category","Live")]:extra 3 vom …full-episode with >20 min runtime).kikascope; fetch episodes.[Skip](long-running + storage; needs stream-resolve + ffmpeg — next slice), exactly as you asked.Testtarget now filters outCategory=Live(external APIs drift/rate-limit → not your bug), and a newTestLivetarget runs only them:./build.cmd TestLive.Verified
./build.cmd Test→ 62 deterministic tests green, live excluded (this is what CI runs)../build.cmd TestLive→ 4 live pass + 1 skip against the real ARD/KiKA API.Next
ZDF "Heute Show" (GraphQL + token + PTMD — the complex one), the download slice (stream-resolve → ffmpeg), and wiring the crawl Actions into the agents.