Standards-compliant library for parsing and serializing HTML documents and fragments in Python
Python Other
Latest commit 17499b9 Feb 18, 2017 @moben moben committed with gsnedders Avoid DeprecationWarnings on Python 3.6 (#318)
Python 3.6 produces warnings on invalid escape sequences in
strings, such as "\s", and they will be syntax errors in a future
version of Python.

See <https://docs.python.org/3/reference/lexical_analysis.html#strings>.
Permalink
Failed to load latest commit information.
doc Fix typo in docs. (#300) Oct 27, 2016
html5lib Avoid DeprecationWarnings on Python 3.6 (#318) Feb 18, 2017
utils Fix all the files outside of html5lib to flake8 cleanly May 20, 2016
.coveragerc Add codecov. Apr 25, 2016
.gitignore Move to far more complete .gitignore from GH's repo. Apr 28, 2015
.gitmodules Move testdata submodule to using HTTPS. May 16, 2013
.prospector.yaml Add prospector/pylint config for the sake of Landscape. May 20, 2016
.pylintrc Add prospector/pylint config for the sake of Landscape. May 20, 2016
.pytest.expect Update tests to tip of html5lib-tests Jul 6, 2016
.travis.yml Fix Travis (#319) Feb 18, 2017
AUTHORS.rst Make self closing col start tags not cause a parse error (#244) Jun 27, 2016
CHANGES.rst Fix attribute order to the treebuilder to be document order Jul 15, 2016
CONTRIBUTING.rst separate AUTHORS.rst May 9, 2013
LICENSE Readd LICENSE Apr 10, 2013
MANIFEST.in Add AUTHORS.rst and test files to manifest. Dec 3, 2015
README.rst Fix #120: introduce keyword arguments for encodings by source Jul 11, 2016
debug-info.py drop usage of charade now chardet is maintained again May 22, 2016
flake8-run.sh Fix all the files outside of html5lib to flake8 cleanly May 20, 2016
parse.py Move a whole bunch of private modules to be underscore prefixed Jul 14, 2016
pytest.ini Change py.test config. May 10, 2016
requirements-install.sh Fix Travis (#319) Feb 18, 2017
requirements-optional.txt Preserve attribute order when parsing May 22, 2016
requirements-test.txt Require flake8 to be < 3.0 for Python 2.6 support (#291) Jul 26, 2016
requirements.txt Declare explicit dependency on Six 1.9 (#301) Oct 27, 2016
setup.cfg add pep8/flake8 config to get something useful happening with them May 20, 2016
setup.py Declare explicit dependency on Six 1.9 (#301) Oct 27, 2016
tox.ini Require flake8 to be < 3.0 for Python 2.6 support (#291) Jul 26, 2016

README.rst

html5lib

https://travis-ci.org/html5lib/html5lib-python.png?branch=master

html5lib is a pure-python library for parsing HTML. It is designed to conform to the WHATWG HTML specification, as is implemented by all major web browsers.

Usage

Simple usage follows this pattern:

import html5lib
with open("mydocument.html", "rb") as f:
    document = html5lib.parse(f)

or:

import html5lib
document = html5lib.parse("<p>Hello World!")

By default, the document will be an xml.etree element instance. Whenever possible, html5lib chooses the accelerated ElementTree implementation (i.e. xml.etree.cElementTree on Python 2.x).

Two other tree types are supported: xml.dom.minidom and lxml.etree. To use an alternative format, specify the name of a treebuilder:

import html5lib
with open("mydocument.html", "rb") as f:
    lxml_etree_document = html5lib.parse(f, treebuilder="lxml")

When using with urllib2 (Python 2), the charset from HTTP should be pass into html5lib as follows:

from contextlib import closing
from urllib2 import urlopen
import html5lib

with closing(urlopen("http://example.com/")) as f:
    document = html5lib.parse(f, transport_encoding=f.info().getparam("charset"))

When using with urllib.request (Python 3), the charset from HTTP should be pass into html5lib as follows:

from urllib.request import urlopen
import html5lib

with urlopen("http://example.com/") as f:
    document = html5lib.parse(f, transport_encoding=f.info().get_content_charset())

To have more control over the parser, create a parser object explicitly. For instance, to make the parser raise exceptions on parse errors, use:

import html5lib
with open("mydocument.html", "rb") as f:
    parser = html5lib.HTMLParser(strict=True)
    document = parser.parse(f)

When you're instantiating parser objects explicitly, pass a treebuilder class as the tree keyword argument to use an alternative document format:

import html5lib
parser = html5lib.HTMLParser(tree=html5lib.getTreeBuilder("dom"))
minidom_document = parser.parse("<p>Hello World!")

More documentation is available at https://html5lib.readthedocs.io/.

Installation

html5lib works on CPython 2.6+, CPython 3.3+ and PyPy. To install it, use:

$ pip install html5lib

Optional Dependencies

The following third-party libraries may be used for additional functionality:

  • datrie can be used under CPython to improve parsing performance (though in almost all cases the improvement is marginal);
  • lxml is supported as a tree format (for both building and walking) under CPython (but not PyPy where it is known to cause segfaults);
  • genshi has a treewalker (but not builder); and
  • chardet can be used as a fallback when character encoding cannot be determined.

Bugs

Please report any bugs on the issue tracker.

Tests

Unit tests require the pytest and mock libraries and can be run using the py.test command in the root directory; ordereddict is required under Python 2.6. All should pass.

Test data are contained in a separate html5lib-tests repository and included as a submodule, thus for git checkouts they must be initialized:

$ git submodule init
$ git submodule update

If you have all compatible Python implementations available on your system, you can run tests on all of them using the tox utility, which can be found on PyPI.

Questions?

There's a mailing list available for support on Google Groups, html5lib-discuss, though you may get a quicker response asking on IRC in #whatwg on irc.freenode.net.