PureHTML is a declarative, language-agnostic specification and toolkit for extracting structured JSON data from HTML documents using simple YAML or JSON schemas.
Instead of writing repetitive, error-prone DOM querying and scraping logic in application code, PureHTML allows you to describe what data to extract and how to transform it declaratively. The schema can be authored, validated, and shared across different services and programming languages.
Explore the documentation and interactive playground at https://purescraps.github.io/purehtml/.
This monorepo houses official implementations of the PureHTML specification across multiple languages, all validated against a shared compliance test suite:
| Language | Directory | Package / Module | Badges |
|---|---|---|---|
| TypeScript / JavaScript | typescript/ |
@purescraps/purehtml |
|
| Go | go/ |
github.com/purescraps/purehtml/go |
|
| Python | python/ |
purehtml-purescraps |
|
| Java | java/ |
com.purescraps.purehtml:purehtml |
|
| Ruby | ruby/ |
purehtml |
PureHTML separates data extraction into declarative schemas, an HTML document input, and an extraction engine.
selector: .product-card
type: array
items:
properties:
title:
selector: h2.title
transform: trim
price:
selector: .price
transform: [trim, number]
link:
selector: a.buy-btn
transform:
- attr(href)
- resolve(https://example.com)```typescript
import { ConfigFactory, extract } from '@purescraps/purehtml';
const config = ConfigFactory.fromYAML(yamlString);
const result = extract(htmlString, config, 'https://example.com');
```
<!-- slide -->
```go
import "github.com/purescraps/purehtml/go"
factory := purehtml.NewConfigFactory()
config, _ := factory.FromYAML(yamlString)
result, _ := purehtml.ExtractFromString(purehtml.DefaultBackend, htmlString, config, "https://example.com")
```
<!-- slide -->
```python
from purehtml import ConfigFactory, extract
config = ConfigFactory.from_yaml(yamlString)
result = extract(config, htmlString, "https://example.com")
```
<!-- slide -->
```java
import com.purescraps.purehtml.configs.ConfigFactory;
import static com.purescraps.purehtml.PureHTML.extract;
var config = ConfigFactory.fromYAML(yamlString);
var result = extract(config, htmlString, "https://example.com");
```
<!-- slide -->
```ruby
require 'purehtml'
config = PureHTML::ConfigFactory.from_yaml(yamlString)
result = PureHTML.extract(config, htmlString, "https://example.com")
```
- Declarative Schemas: YAML-based definitions for extracting primitives, nested objects, arrays, constants, unions, and default fallbacks.
- Fallback Selectors: Supply an array of selectors (
selector: ['.sale-price', '.price']) or selector objects withsampleHTMLsfor resilient scraping when websites A/B test or change their DOM structure. - Chainable Transformers: 20+ built-in transformers including
trim,number,boolean,attr,html,text,replace,split,join,json,jsonpath,resolve,urlQueryParam, and more with support for quoted arguments and regex patterns. - Cross-Language Compliance: All ports are continuously tested against shared test fixtures in
specs/to guarantee identical parsing and transformation behavior. - Zero Runtime Overhead Schemas: Pre-compile schemas once and execute them efficiently over millions of documents.
Websites often vary their DOM structure across A/B test variants or responsive views. selector can be configured as an array of fallback selectors evaluated in order:
selector:
- .flash-deal-price
- .sale-price
- .regular-price
transform: trimThe schema also supports selector objects with sampleHTMLs, allowing you to document and reference HTML fixtures alongside each selector variant:
selector:
- selector: .promo-price
sampleHTMLs:
- tests/fixtures/promo.html
- selector: .standard-price
sampleHTMLs:
- tests/fixtures/standard.htmlTransformers support parameter lists, but arguments containing spaces, commas, colons, or regex escape sequences must follow quoting rules:
- Bare (unquoted) arguments: Simple alphanumeric identifiers matching
[\w-]+can be written without quotes (e.g.attr(href),trim,split(-)). - Quoted arguments (
"..."or'...'): Required when arguments contain spaces, commas, colons, or regex metacharacters:replace("\s+", " ")or'replace(",", "")'split(", ")or'split("::")'join(" - ")orjoin(", ")urlQueryParam("utm_source")orurlQueryParam('ref')
- Escaping inside quotes: Backslashes escape matching quote characters (
\"or\') or literal backslashes (\\). Regex escape sequences like\s,\d, and\wpass through untouched to regex engines. - YAML tip: In YAML schemas, wrap transform expressions in outer quotes (e.g.
transform: 'replace("\s+", " ")') or use YAML array syntax (- replace("\s+", " ")) to prevent YAML parsing conflicts.
Modern frameworks (Next.js, Nuxt, Remix) and SEO-rich web pages embed structured JSON data directly inside <script> tags (script#__NEXT_DATA__, script#__NUXT_DATA__, or script[type="application/ld+json"]). Extracting directly from these hydration scripts is vastly faster and more durable than querying volatile presentation CSS classes.
Combine json with jsonpath to pluck properties directly:
properties:
productId:
selector: script#__NEXT_DATA__
transform:
- json
- 'jsonpath("$.props.pageProps.product.id")'
productName:
selector: script#__NEXT_DATA__
transform:
- json
- 'jsonpath("$.props.pageProps.product.title")'
price:
selector: script#__NEXT_DATA__
transform:
- json
- 'jsonpath("$.props.pageProps.product.price")'
jsonLdPrice:
selector: 'script[type="application/ld+json"]'
transform:
- json
- 'jsonpath("$.offers.price")'
- numberScrapers and crawlers frequently encounter relative links in pagination navigation (<a class="next" href="/catalog/page-3">Next</a>) or listing item anchors (<a href="details/401">).
Combine attr(href) with resolve to automatically resolve relative URLs against the base URL provided to extract():
properties:
currentPage:
selector: .pagination .current
transform:
- 'replace("Page ", "")'
- trim
- number
nextPageUrl:
selector: .pagination a.next
transform:
- attr(href)
- resolve
items:
selector: ul.items li a
type: array
items:
properties:
name:
transform: trim
url:
transform:
- attr(href)
- resolveReal-world pages frequently exhibit variable markup due to A/B testing, promo badges, or optional fields (e.g., discounted sale prices vs. standard prices, out-of-stock items missing prices, or promotional items marked "FREE").
Combine union with a fallback constant value at the end to gracefully handle variant structures with reliable defaults:
selector: .products .product
type: array
items:
properties:
name:
selector: .name
transform: trim
price:
union:
- selector: .sale-price
transform:
- 'replace("\$", "")'
- trim
- number
- selector: .price
transform:
- 'replace("\$", "")'
- trim
- number
- selector: .free-text
transform:
- 'replace("FREE", "0")'
- number
- constant: 0
status:
union:
- selector: .badge
transform: trim
- constant: "Standard"See docs/src/examples/advanced.hackernews-listing.ts for a production-ready Hacker News listing scraper that extracts post IDs, titles, URLs, timestamps, scores, and comment links:
selector: tr.athing.submission
items:
properties:
id: { transform: attr(id) }
title: { selector: span.titleline }
url: { selector: 'span.titleline a:first-child', transform: [attr(href), resolve] }
date: { selector: '+tr span.age', transform: attr(title) }
score: { selector: '+tr span.score' }
comments: { selector: '+tr span.age a', transform: [attr(href), resolve] }Explore and test this example interactively in the Playground.
The PureHTML schema definition is located at spec.schema.yaml and typescript/src/config-schema.json.
Shared cross-language test suites are maintained in specs/. Each language implementation runs these specifications to ensure complete compatibility:
- Go:
cd go && go test -v - Python:
cd python && python3 run_tests.py - Java:
cd java && mvn test - TypeScript:
cd typescript && pnpm test - Ruby:
cd ruby && rake test
You can use yaml-language-server for schema validation and autocompletions in YAML files.
Add this modeline to the top of your YAML schema:
# yaml-language-server: $schema=typescript/src/config-schema.jsonFor VS Code, install the Red Hat YAML extension.