Extraction Recipes

Extraction recipes

Use an extraction recipe when automatic scraping does not understand a source. Recipes describe repeating records and the fields containing a channel name, AceStream ID, group, logo or EPG ID. They do not execute website JavaScript.

Two ways to build a recipe

Inside the app: open Scraper, then a source's More actions → Configure extraction. For a new source, choose Extraction builder from the Scraper page's actions menu. Enter its URL and source type. Load source URL fetches a sample using this installation; no public access to your installation is needed.

On GitHub Pages: open Extraction builder in the tools and docs hub, alongside Docker setup and Docs. Paste or upload an HTML, text or JSON sample. The builder works in your browser and never connects to your scraper. Export a recipe file or copy the recipe JSON, then import/paste it in your app's extraction builder.

Switching tools keeps both builders' drafts in the current page. Reloading the page clears the extraction draft, so export your recipe before leaving. The Docs section links to the existing wiki; the docs have not moved yet.

The Pages helper is published through the existing Jenkins docs flow after this feature is integrated; a local feature branch does not publish that URL.

Provide a sample

Try the source URL in the app first. If the page is generated by JavaScript, open it in your browser, open developer tools, select the channel container in Elements, and use Copy → Copy outerHTML. No console command is necessary. Alternatively, copy a JSON response from the Network panel or upload a saved .html, .txt or .json file. Maximum sample size is 32 MiB.

Pasted HTML is a snapshot. A successful sample test does not prove that a scheduled HTTP fetch will contain the same data. In the app, Fetch and test source URL verifies the response the scraper actually receives. If it contains only a loading screen or verification challenge, use the underlying JSON endpoint when available. Recipes cannot solve login, JavaScript rendering or blocked fetches.

Select records and fields

• HTML elements: choose Visual picker and assign a repeating record. Click a channel's heading and Select parent until the selected element is its whole card/row. Then assign the channel name and ID inside that row. Field selectors are relative to each row. Link fields read href; copy fields can read value or an AceStream data attribute. Review and edit selectors when needed.
• Raw source: select one complete record and use it as a record template. Select its name and ID to generalize the generated regex. The generated pattern is a starting point: review the surrounding text and test all records.
• JSON response: select the array of channel objects, then select the fields from the first object. Paths use dot-separated keys and numeric array indices, such as data.channels, title and stream.id. Leave the records path empty for a root array. An ID array produces multiple streams with one shared name.
• Text / regex: each match of the record regex defines one record. Field regexes run inside that record. The first capture is the value, or the whole match when there is no capture. Named groups, backreferences and language- specific regex extensions are not supported. Flags: i, m, s.

The visual preview preserves HTML structure and inline styles but blocks original scripts, navigation, forms and external assets. It may look different without remote styles, fonts and images. Raw source is always available. Text selection and visual selection are different authoring methods; inspect the generated rules before switching methods.

Review and save

Tests show record counts, extracted channels, invalid records and duplicate IDs. Each record requires exactly one nonempty name of at most 200 characters. IDs must be 40 hexadecimal characters, optionally prefixed by acestream://. Missing fields are not paired with fields from another row. At most 1,000 records/channel IDs are accepted; narrow larger sources. Only absolute HTTP(S) logo URLs are kept.

Walkthrough: HTML channel cards

Start with HTML channel cards in Recipe catalogue, then choose Load template and sample. This example uses synthetic IDs for learning; they are not playable channels.

1. Select Visual picker. Set Assign selection to to Repeating record.
2. Click Example News, then Select parent until the whole channel card is selected. The repeating row selector should be section.channel.
3. Choose Channel name in Assign selection to, then click the heading. Its selector is h3, relative to each card; leave its attribute empty to read text.
4. Choose AceStream ID, then click Watch. Its selector is a and its attribute is href, which contains the AceStream link.
5. Choose Test sample on the public helper or Test with scraper engine in the app. Expect two records, two channels and zero invalid records.
6. Replace the sample with your source's response and repeat the selection. Check a second card and inspect every extracted name/ID pair before saving.

For a table, the repeating record is usually a row (tr); field selectors pick its cells or links. A record must contain both the name and ID. Choosing the whole page as one record often produces an ambiguous name; choosing only the heading leaves its neighboring link outside the record.

If an attribute contains extra text, use a field regex such as ([0-9a-fA-F]{40}) to capture just the ID. This operates inside the chosen field, not across unrelated cards elsewhere on the page.

Walkthrough: JSON from an API

For a response shaped like this:

{"data":{"channels":[
  {"title":"News","stream":{"id":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"}},
  {"title":"Sports","stream":{"id":"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"}}
]}}

1. Paste/upload the response, or use Load source URL in the app.
2. Set Source format to JSON response and open JSON fields.
3. Assign Repeating record to the data.channels array. You can also type data.channels into Records array path.
4. Assign Channel name to title and AceStream ID to stream.id. These paths start inside each channel object; do not repeat data.channels.
5. Test and review the two pairs. Leave field regexes blank when values are already correct. An ID array is supported; its IDs share that record's name.

For a top-level array, leave Records array path blank. Paths support ordinary dot-separated keys and numeric array indices. They do not support JSONPath wildcards, filters, recursive searches, or keys containing a literal dot. If the endpoint returns multiple pages, the builder sees only the response you provided.

Walkthrough: raw text and regex

Use this fallback when a visual preview cannot show the relevant fields. In Raw source, select one complete channel record and choose Use selected text as record template. Select its name, choose Channel name, and assign the selection. Do the same for the ID. Select the 40-character ID itself; leave a surrounding acestream:// prefix outside the selection.

The helper generalizes the selected fields while retaining surrounding text. Test against at least two different records. Generated rules may still contain changing IDs, dates, classes or whitespace that you did not select; edit those parts as needed. Selections must stay inside the chosen record and must not overlap. Editing the sample clears the selection template.

For a line format News | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa, these manual rules provide a starting point:

Setting	Value
Source format	Text / regex
Repeating record regex	[^\r\n]+|\s*[0-9a-fA-F]{40}
Channel name regex	^\s*(.*?)\s*|
AceStream ID regex	|\s*([0-9a-fA-F]{40})

Use plain capture groups (…): the first capture becomes the field value. With no capture, the whole match becomes the value. i ignores case, m makes ^/$ line-based, and s lets . include newlines. Named groups, backreferences, lookbehind and language-specific escapes are rejected. A browser preview and Python extraction can still differ on unusual regex/HTML edge cases; the installed app's result is authoritative.

Limits and unsupported workflows

Area	Current behavior
Source/sample size	Up to 32 MiB; fetched data is measured after HTTP decompression
Visual/field picker	2 MiB; HTML also allows up to 20,000 markup delimiters. Use raw selection, manual JSON paths or a smaller example when exceeded.
Preview upload	Decoded sample: 32 MiB. JSON request envelope: 64 MiB + 128 KiB, including escaping; uploads time out after 30 seconds.
Recipe file	Up to 512 KiB; export includes rules only
Output	Up to 1,000 records and 1,000 distinct channel IDs
Matching time	Up to 15 seconds per preview; the Linux worker also has a 10-second CPU limit and 1 GiB memory limit
Concurrent backend work	Two fetches and two extraction workers per app process
Fetching	GET only, 20 seconds per request, at most six requests including redirects
Authentication	No custom request headers, cookies, login flows or POST bodies in recipes
Dynamic sites	No website JavaScript execution, scrolling, pagination or automatic multi-request joins
Preview fidelity	External images/styles/fonts and executable content are blocked; use raw source when needed
Coverage	Tests cover supplied/fetched records, not unloaded rows or the entire website

Larger DOM trees can still exceed processing limits even when their source is under 32 MiB. Copying the channel container or using a smaller JSON response often makes authoring easier. Extraction confirms format and pairing; it does not verify that a channel is online or that media will play.

Troubleshooting

Symptom	What to check
Sample works, source URL fails	Compare the fetched response with the browser DOM. JavaScript, login or anti-bot pages often produce different content. Look for the underlying JSON response in the browser's Network panel.
Zero records	Check the repeating selector, record regex or array path first. Fields cannot match until records are found.
Missing/ambiguous channel name	Select exactly one name inside each record; avoid a selector that also picks navigation or several headings.
Invalid ID	Read the link/data attribute rather than its “Watch” label; use a field regex to capture exactly 40 hexadecimal characters.
Wrong name paired with an ID	Narrow the record to one card/row containing both fields. Never build separate whole-page name and ID lists.
Timeout or worker failure	Simplify broad/nested regex repetition or narrow the input. Try selectors/JSON paths instead of matching the entire page with regex.
Save is disabled	Test again after changing the recipe, sample or URL. Saving needs at least one channel and zero invalid records.
Source fetch is blocked	Check the URL and the installation's private-source policy. Metadata addresses are always blocked.
Stale channels remain after scraping	Custom recipes preserve old source channels. Remove stale entries through inventory management.

Save and run the source

Inside the app, saving requires a passing test with channels and no invalid records. Editing rules, samples or the source address invalidates the previous test for saving. Tests themselves never import/delete channels or save a recipe. After saving, launch a normal source scrape when ready.

Custom recipes preserve existing source channels even when a new response no longer contains them. Remove stale channels through normal inventory management. A failed recipe or invalid record preserves the catalogue and records a source error. There is no silent fallback to automatic extraction. Use automatic extraction clears the recipe. The older bare-ID option applies to automatic extraction only. Existing sources remain automatic after upgrading.

Catalogue and sharing

Both helpers include the same versioned recipe catalogue. The initial entries are starter templates with synthetic examples, not claims of support for particular websites. Loading a template replaces the current draft and sample. Installed recipes are copies with an explicit version, so catalogue changes do not silently change a source. Load/import a newer version and test it before saving.

Exported recipe files contain rules and their name/version only, not the sample, URL or credentials. To share one, use the contribution link and attach the recipe plus a small sanitized example. Review the files before submitting; do not include cookies, authorization headers, private page data or token-bearing URLs. Publication requires review and passing fixture tests. File import also works offline.