Allow incomplete Python code to be highlighted

[shimizukawa]

Currently, the try_parse check in PygmentsBridge.highlight_block prevents incomplete code which Pygments is able to highlight from being highlighted in Sphinx.

For instance, Pygments proper highlights this code snippet:

#!python

{
  'foo' : bar,
  ...
}

(note the ellipsis) but Sphinx can't.

I propose to remove the check (at least when the language is explictly stated in the markup) and fall back to unhighlighted if Pygments fails to highlight the code snippet.

---

• Bitbucket: https://bitbucket.org/birkenfeld/sphinx/issue/695
• Originally reported by: Anonymous
• Originally created at: 2011-05-10T21:59:52.312

[shimizukawa]

From Georg Brandl on 2011-05-15 09:39:28+00:00

+1 for forcing highlighting if the language is explicitly given.

This needs a change to the code to differentiate between "language set globally" and "language set locally".

[shimizukawa]

From Anonymous on 2011-05-15 16:37:43+00:00

\o/

Could you explain why you try to parse the code in Sphinx at all? Why not let Pygments do all the parse/guess work?

[shimizukawa]

From Anonymous on 2011-07-18 15:59:53+00:00

bump

[shimizukawa]

From Anonymous on 2011-08-23 08:37:03+00:00

Can we get this done soon? I'm willing to write a patch but I'm not familiar with the Sphinx codebase.

[shimizukawa]

From Georg Brandl on 2011-09-21 16:39:16+00:00

The explanation is that the default highlighting language is Python, so all literal blocks you write would be highlighted as Python. The parsing is usually a convenient way to decide which block is actually Python, without you having to mark non-Python blocks as such.

[shimizukawa]

From Anonymous on 2011-10-06 19:09:59+00:00

I still don't understand what you get from doing two parses.

If you do only one parse (in Pygments) and the code is valid, it gets highlighted. If it isn't valid, it's catched in {{{highlighting.py:209 (except ErrorToken:)}}} and is rendered unhighlighted.

Now if you do a manual parse in Sphinx first and the code is valid, it is passed to Pygments which highlights the code. If it's invalid, it's rendered unhighlighted -- the //exact// same thing that happens if you don't pre-parse it at all.

[shimizukawa]

From Anonymous on 2011-10-06 20:06:59+00:00

Here's a patch that removes {{{try_parse}}} entirely and always uses the Pygments lexer. The test suite changes were made because now the Pygments Python lexer is always used and it seems to be a bit less strict than the Python grammar, so things like {{{"some code"}}} are highlighted as valid Python. Besides that, the patch does not caues any test failures.

[shimizukawa]

From Georg Brandl on 2011-10-07 09:10:28+00:00

There are no "two parses": Pygments lexers are only lexers (although some of them have a bit more intelligence than you would put in an ordinary lexer, in order to recognize function names and such).

There is a large amount of text that Pygments will highlight happily as Python code, while it is not: that's why the parser pass is valuable.

That said, I agree that it may be unfortunate in places, so we'll add some customizability in the future.

[shimizukawa]

From Anonymous on 2011-10-07 14:16:42+00:00

Here's a another patch that uses a different approach - it only removes the {{{try_parse}}} call in case the highlighting language was stated explicitly.

[shimizukawa]

From Georg Brandl on 2011-11-01 07:47:34+00:00

Issue #806 was marked as a duplicate of this issue.

[shimizukawa]

From Anonymous on 2012-01-03 12:05:40+00:00

Can we get this patch applied soon? It's been 8 months.

[shimizukawa]

From Georg Brandl on 2012-01-29 09:21:07+00:00

Closes #695: When the highlight language "python" is specified explicitly, do not try to parse the code to recognize non-Python snippets.

Thanks to Jonas Haag for the patch.

→ <>

[shimizukawa]

From Anonymous on 2012-03-19 21:17:58+00:00

Finally! Thanks!