Add experimental HTML5 writer #3464

shibukawa · 2017-02-26T03:50:18Z

Subject:

Feature or Bugfix

Feature

Purpose

Provides HTML5 builder
Creates EPUB file that passes EPUB validator.

Detail

if html_experimental_html5_builder option is specified, use docutils' html5_polyglot Writer instead of html4css1 Writer.

Relates

tk0miya

It looks great! LGTM with nits

tk0miya · 2017-02-26T05:05:44Z

doc/conf.py

@@ -5,6 +5,7 @@
 import re
 import sphinx

+html_experimental_html5_builder = True


Could you move this close to html_* entries?

tk0miya · 2017-02-26T05:06:57Z

doc/config.rst

+
+   Output is processed with HTML5 writer.  This feature needs docutils 0.13.1 or newer.  Default is ``False``.
+
+   .. versionadded:: 1.5


This will be released as 1.6.

tk0miya · 2017-02-26T05:10:21Z

doc/config.rst

@@ -1088,6 +1088,11 @@ that use Sphinx's HTMLWriter class.

   Output file base name for HTML help builder.  Default is ``'pydoc'``.

+.. confval:: html_experimental_html5_builder
+
+   Output is processed with HTML5 writer.  This feature needs docutils 0.13.1 or newer.  Default is ``False``.


The name and descriptions are different. which is correct; html5_builder or HTML5 Writer?

tk0miya · 2017-02-26T05:13:28Z

sphinx/builders/html.py

@@ -1320,6 +1334,7 @@ def setup(app):
    app.add_config_value('html_search_options', {}, 'html')
    app.add_config_value('html_search_scorer', '', None)
    app.add_config_value('html_scaled_image_link', True, 'html')
+    app.add_config_value('html_experimental_html5_builder', False, 'html')


We need to warn if users enable HTML5 builder without docutils-0.13.

tk0miya · 2017-02-26T05:13:52Z

sphinx/util/__init__.py

+
+def html5_writer_available():
+    import docutils
+    return list(map(int, docutils.__version__.split('.'))) > [0, 13, 0]


You can use sphinx.util.docutils.__version_info__ instead.

tk0miya · 2017-02-26T05:15:52Z

sphinx/util/__init__.py

@@ -612,3 +612,8 @@ def status_iterator(iterable, summary, color="darkgreen", length=0, verbosity=0,
        yield item
    if l > 0:
        logger.info('')
+
+
+def html5_writer_available():


Personally, I'd like to name the functions which return a boolean value to is_*(). What do you think?

In addition, please add type annotation.
For this function, it's better to annotate like following:

def html5_writer_available(): # type: () -> bool

tk0miya · 2017-02-26T05:36:15Z

sphinx/writers/html5.py

@@ -0,0 +1,934 @@
+# -*- coding: utf-8 -*-
+"""
+    sphinx.writers.html5


Could you tell me how did you make this? This writer is too large to review whole of code. So I'd like to know the difference with html4 writer.
As a quick look, it is almost same as html4 writer. So I guess you were copied the html4 writer and rewrote the parent class. is this right?

tk0miya · 2017-02-26T05:39:16Z

It seems some testcases are broken. Could you check them please?

BTW, it seems there are no testcases for HTML5 writer.
At least, we have to invoke it once.

shibukawa · 2017-03-01T01:58:19Z

This is my code memo:

    This class is an almost copy of HTMLTranslator because:

    - docutils still uses old style class for Python 2.x. In this case,
      HTMLTranslator and HTML5Translator share logic with different parent
      class. But mixing old and new styles with multiple inheritance
      would bring potential difficult issue.
    - In future (Sphinx 2000?), Sphinx would drop HTML4 support.
      At that time, dropping task becomes easy if they are independent.

    Current different points with ``sphinx.writers.html`` are:

    - HTMLWriter is completely same. So this file omit it.
    - ``visit_table()`` adds ``docutils`` class to ``<table>`` tag
      like HTML4CSS1 writer of docutils.
      But in future it should be removed (now I keep it to keep
      compatibility with existing themes.
    - Omit quick fix to remove named HTML entity (#3452).
      ``visit_option_group()``, ``visit_entry()``, ``visit_field_name()``
      are removed.

    HTML5 results have changes from HTML4 ones:

    - ``admonition`` nodes doesn't add ``first`` and ``last`` classes
      to their children. CSS3 pseudo-classes are enought to replace
      them in theme.
    - Many ``<p>`` nodes (for example in ``<li>`` tags)
    - Doc field and field list doesn't use ``<table>``, but ``<dt>``
      and ``<dd>``.

tk0miya · 2017-02-26T16:20:11Z

sphinx/application.py

@@ -625,6 +626,8 @@ def add_node(self, node, **kwds):
            translator = self._translators.get(key)
            if translator is not None:
                pass
+            elif key == 'html' and is_html5_writer_available():


html_experimental_html5_writer is not referred here.
If I remember correctly, the config object is instantiated until this phase. so we should refer it to determine which writer is used.

tk0miya · 2017-03-01T05:11:37Z

CHANGES

@@ -74,6 +74,8 @@ Features added
 * Make ``'extraclassoptions'`` key of ``latex_elements`` public (refs #3480)
 * #3463: Add warning messages for required EPUB3 metadata. Add default value to
  ``epub_description`` to avoid warning like other settings.
+* HTML buildre uses experimental HTML5 writer if ``html_experimental_html5_builder`` is True
+  and docutils 0.13.1 and newer is installed.


docutils-0.13.1 is a bug fix release. so this would work with 0.13.

tk0miya · 2017-03-01T05:12:15Z

doc/config.rst

@@ -1088,6 +1088,11 @@ that use Sphinx's HTMLWriter class.

   Output file base name for HTML help builder.  Default is ``'pydoc'``.

+.. confval:: html_experimental_html5_writer
+
+   Output is processed with HTML5 writer.  This feature needs docutils 0.13.1 or newer.  Default is ``False``.


This should also be docutils-0.13.

tk0miya · 2017-03-01T05:12:30Z

doc/config.rst

+
+   Output is processed with HTML5 writer.  This feature needs docutils 0.13.1 or newer.  Default is ``False``.
+
+   .. versionadded:: 1.5


It is released as 1.6. Please update this.

tk0miya · 2017-03-01T05:19:24Z

sphinx/builders/html.py

+            self.app.warn(' '.join((
+                'html_experimental_html5_writer is set, but current version is old.',
+                'Docutils\' version should be or newer than 0.13.1, but %s.',
+            )) % __version_info__)


Could you check is this work? __version_info__ is a tuple containing version info (cf. (0, 13, 1)).
So python formatting might cause an error.

tk0miya · 2017-03-01T05:24:20Z

tests/test_build_html5.py

+
+from sphinx import __display_version__
+from util import remove_unicode_literals, strip_escseq
+from etree13 import ElementTree


Unfortunately, etree13 is removed at #3483. Could you merge the HEAD of master (I just merged the change into master now)?

tk0miya · 2017-03-01T05:27:30Z

tests/test_build_html5.py

+from etree13 import ElementTree
+from html5lib import getTreeBuilder, HTMLParser
+import pytest
+from test_build_html import flat_dict, tail_check, check_xpath


Now import statement is ordered by following:

python standard libraries

3rd party libraries (pytest, six and so on)

sphinx-core

files under tests/

Could you reorder them please?

tk0miya · 2017-03-01T05:39:53Z

sphinx/writers/html5.py

+
+    Experimental docutils writers for HTML5 handling Sphinx' custom nodes.
+
+    This class is an almost copy of HTMLTranslator because:


Is this comment needed? Certainly, I requested the difference of writers. But it is only used for reviewing.
I worried about this comment will not be maintained in future version.

This comment describes differences with HTML4 writer mainly. But the old writer will be deprecated in nearly future. I think it is not a description of this codes, and it is used temporarily.
Therefore it would be better to use github-comments or other temporarily space, not in source code.

shibukawa · 2017-03-01T17:04:59Z

updated

tk0miya · 2017-03-02T16:32:08Z

Merged, Thank you for great work.
To be HTML5 writer official, we should confirm the behavior and improve it step by step.
I can't say the roadmap of that now, but I'd like to be it official at 1.7 release.

Anyway, thank you always. I'm very appreciate you.

tk0miya requested changes Feb 26, 2017

View reviewed changes

tk0miya added type:enhancement enhance or introduce a new feature builder:html labels Feb 26, 2017

tk0miya added this to the 1.6 milestone Feb 26, 2017

shibukawa force-pushed the feature/experimental_html5_writer branch 3 times, most recently from 4980d3a to 25c5e3c Compare March 1, 2017 01:56

shibukawa force-pushed the feature/experimental_html5_writer branch from 25c5e3c to 780f280 Compare March 1, 2017 01:59

tk0miya requested changes Mar 1, 2017

View reviewed changes

shibukawa mentioned this pull request Mar 1, 2017

Please add a MOBI builder #739

Closed

shibukawa force-pushed the feature/experimental_html5_writer branch from 780f280 to da5996e Compare March 1, 2017 17:04

shibukawa force-pushed the feature/experimental_html5_writer branch from da5996e to 7aa22b3 Compare March 1, 2017 17:12

Add experimental HTML5 writer

0ef9ac5

shibukawa force-pushed the feature/experimental_html5_writer branch from 7aa22b3 to 0ef9ac5 Compare March 2, 2017 00:57

tk0miya approved these changes Mar 2, 2017

View reviewed changes

tk0miya merged commit 7332023 into sphinx-doc:master Mar 2, 2017

shibukawa deleted the feature/experimental_html5_writer branch March 2, 2017 16:44

This was referenced Mar 2, 2017

Variant of Basic Theme that is Valid HTML5 #1562

Closed

Invalid XHTML generated #1733

Closed

Sphinx does not produce HTML5 output #2376

Closed

The table in exported EPUB doesn't pass EpubCheck validation #3287

Closed

shibukawa mentioned this pull request Mar 4, 2017

Validation error with epubcheck #1376

Closed

tk0miya mentioned this pull request Mar 9, 2017

Captions for markup should be rendered using the <caption> and <figcaption> tags #3526

Open

StrikerRUS mentioned this pull request Oct 3, 2017

[docs] documentation build test microsoft/LightGBM#958

Merged

github-actions bot locked as resolved and limited conversation to collaborators Sep 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add experimental HTML5 writer #3464

Add experimental HTML5 writer #3464

shibukawa commented Feb 26, 2017

tk0miya left a comment

tk0miya Feb 26, 2017

tk0miya Feb 26, 2017

tk0miya Feb 26, 2017

tk0miya Feb 26, 2017

tk0miya Feb 26, 2017

tk0miya Feb 26, 2017

tk0miya Feb 26, 2017

tk0miya commented Feb 26, 2017

shibukawa commented Mar 1, 2017 •

edited

tk0miya Feb 26, 2017

tk0miya Mar 1, 2017

tk0miya Mar 1, 2017

tk0miya Mar 1, 2017

tk0miya Mar 1, 2017

tk0miya Mar 1, 2017

tk0miya Mar 1, 2017

tk0miya Mar 1, 2017

shibukawa commented Mar 1, 2017

tk0miya commented Mar 2, 2017


		Output is processed with HTML5 writer. This feature needs docutils 0.13.1 or newer. Default is ``False``.

		.. versionadded:: 1.5


		Experimental docutils writers for HTML5 handling Sphinx' custom nodes.

		This class is an almost copy of HTMLTranslator because:

Add experimental HTML5 writer #3464

Add experimental HTML5 writer #3464

Conversation

shibukawa commented Feb 26, 2017

Feature or Bugfix

Purpose

Detail

Relates

tk0miya left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tk0miya commented Feb 26, 2017

shibukawa commented Mar 1, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shibukawa commented Mar 1, 2017

tk0miya commented Mar 2, 2017

shibukawa commented Mar 1, 2017 •

edited