Skip to content

feat: structured-data extraction (JSON-LD, Microdata, OpenGraph, RDFa) #273

Description

@gaborbernat

Gap

turbohtml extracts no structured data. Scrapers want JSON-LD, Microdata, OpenGraph and Twitter meta tags, and RDFa, and reach for extruct, metadata_parser, or hand-rolled walks. This is the highest-value capability turbohtml lacks for the scraping audience.

Proposed API

data = document.structured_data()
# {"json_ld": [...], "microdata": [...], "opengraph": {...}, "microformats": [...], "rdfa": [...]}

Pull each format from the tree:

  • JSON-LD: every <script type="application/ld+json">, parsed with stdlib json.
  • Microdata: itemscope / itemprop / itemtype walked into nested dicts.
  • OpenGraph and Twitter: <meta property="og:*"> and <meta name="twitter:*">.
  • RDFa and microformats2 in a later phase.

The tree walk runs in C; JSON parsing stays in the Python facade. Per-format helpers (document.json_ld(), document.opengraph()) sit beside the combined call.

Notes

Unlocks an extruct migration section. JSON-LD and OpenGraph cover most real use, Microdata next, RDFa and microformats2 last. Top of the new-feature list. Severity: high.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions