Hext — Extract Data from HTML
Hext is a domain-specific language for extracting structured data from HTML documents.
The Hext project is released under the terms of the Apache License v2.0.
Suppose you want to extract all hyperlinks from a web page. Hyperlinks have an
anchor tag <a>, an attribute called href and a text that visitors can
click. The following Hext snippet will produce a dictionary for every matched
element. Each dictionary will contain the keys
title which refer
to the href attribute and the text content of the matched <a>.
# Extract links and their text <a href:link @text:title />
Quick Install via Pip
You can install the htmlext command-line utility and the python bindings through pip:
pip install hext htmlext --version
Hext for Node via NPM
Hext for Node is available on npm:
npm install hext node -e 'require("hext")' && echo "hext loaded successfully"
The npm package is compatible with:
- Node v8, v10, v11, v12
- Linux (GLIBC ≥2.14, basically any distribution built after the year 2012)
- Mac OS X (10.11 El Capitan or later)
- x86_64 only
Building Hext from source
Dependencies: g++ ≥7.3 or clang ≥6.0, cmake ≥3.8, libboost-dev ≥1.55, libboost-regex-dev ≥1.55, libboost-program-options-dev ≥1.55, libgumbo-dev ≥0.10.1 and rapidjson-dev ≥1.1.0
wget https://github.com/html-extract/hext/archive/v0.8.0.tar.gz tar xf *.tar.gz cd hext*/build cmake -DBUILD_SHARED_LIBS=On .. && make -j 2 ./htmlext --help
See Hext's build instructions for more.
Components of this Project
htmlext: Command line utility that applies Hext snippets to an HTML document and produces JSON.
libhext: C++ library that contains a Hext parser but also allows for customization.
libhext-test: Unit tests for libhext.
Hext bindings: Bindings for scripting languages. There are extensions for Node.js, Python, Ruby and PHP that are able to parse Hext and extract values from HTML.
├── build Build directory for htmlext ├── cmake CMake modules used by the project ├── htmlext Source for the htmlext command line tool ├── libhext The libhext project │ ├── bindings Hext bindings for scripting languages │ ├── build Build directory for libhext │ ├── doc Doxygen documentation for libhext │ ├── examples Examples making use of libhext │ ├── include Public libhext API │ ├── ragel Ragel input files │ ├── scripts Helper scripts for libhext │ ├── src libhext implementation files │ └── test The libhext-test project │ ├── build Build directory for libhext-test │ └── src Source for libhext-test ├── man Htmlext man page ├── scripts Scripts for building and testing releases ├── syntaxhl Syntax highlighters for Vim and ACE └── test Blackbox tests for htmlext
Dependencies for development
- Ragel generates the state machine that is used to parse Hext
- The unit tests for libhext are written with Google Test
- libhext's public API documentation is generated by Doxygen
- libhext's scripting language bindings are generated by Swig
There are unit tests for libhext and blackbox tests for Hext as a language,
whose main purpose is to detect unwanted change in syntax or behavior.
The libhext-test project is located in
/libhext/test and depends on Google
Test. Nothing fancy, just build the project and run the executable
libhext-test. How to write test cases with Google Test is described
The blackbox tests are located in
/test. There you'll find a shell script
blackbox.sh. This script applies Hext snippets to HTML documents and
compares the result to a third file that contains the expected output. For
example, there is a test case
icase-quoted-regex that consists of three files:
icase-quoted-regex.expected. To run this test case you would do the following:
$ ./blackbox.sh case/icase-quoted-regex.hext
blackbox.sh will then look for the corresponding
of the same name in the directory of
icase-quoted-regex.hext. Then it will
htmlext with the given Hext snippet and HTML document and compare the
icase-quoted-regex.expected. To run all blackbox tests in
$ ./blackbox.sh case/*.hext
blackbox.sh will look for the
htmlext binary in
that, it looks for the binary in the default build directory. You can tell
blackbox.sh which command to use by setting HTMLEXT. For example, to run all
tests through valgrind you'd run the following:
$ HTMLEXT="valgrind -q ../build/htmlext" ./blackbox.sh case/*.hext
— An HTML5 parsing library in pure C99
Gumbo is used as the HTML parser behind
hext::Html. It's fast, easy to integrate and even fixes invalid HTML.
— Ragel State Machine Compiler
The state machine that is used to parse Hext snippets is generated by Ragel. You can find the definition of this machine in
— A fast JSON parser/generator for C++
RapidJSON powers the JSON output of the
htmlextcommand line utility.
— A lightweight and flexible command-line JSON processor
An indispensable tool when dealing with JSON in the shell. Piping the output of
jqlets you do all sorts of crazy things.
- Ace — A Code Editor for the Web
Used as the code editor in the "Try Hext in your Browser!" section and as a highlighter for all code examples. The highlighting rules for Hext are included in this project in
/syntaxhl/ace. Also, there's a script in
/libhext/scripts/syntax-hl-acethat uses Ace to transform a code snippet into highlighted HTML.
— HTTP and WebSocket built on Boost.Asio in C++11
The Websocket server behind the "Try Hext in your Browser!" section is built with Beast. See github.com/html-extract/hext-on-websockets for more.