docs syntax highlighting missing #591

flying-sheep · 2015-10-25T14:26:01Z

starting with 0.18.0, the syntax highlighting for all but the first code block on any page vanished.

it seems some change on readthedocs, docutils or sphinx made .. highlight:: python not work properly.

and index.rst doesn’t have that directive, so the magic for autodetecting language or whatever made it work before is also gone.

The text was updated successfully, but these errors were encountered:

asvetlov · 2015-10-25T14:39:37Z

The issue should be solved by Pygments library. Nothing to do in aiohttp itself.

flying-sheep · 2015-10-25T14:45:25Z

great. what is the issue?

asvetlov · 2015-10-25T14:47:59Z

async/await support I mean.

flying-sheep · 2015-10-25T21:54:45Z

ah, so it simply fails at parsing and therefore doesn’t display stuff?

makes sense, although most other highlighters are more fault tolerant

asvetlov · 2015-10-25T23:20:30Z

ah, so it simply fails at parsing and therefore doesn’t display stuff?

True

We are tightly coupled with sphinx for making docs.
I'm happy with the tool and prefer waiting for fix instead of switching to something another one.

flying-sheep · 2015-10-26T06:36:52Z

of course!

rutsky · 2016-02-05T23:41:37Z

Highlighting issue can be easily fixed by explicitly specifying code-block type to python, e.g.:

.. code-block:: python

   import asyncio
   from aiohttp import web

   async def hello(request):
       return web.Response(body=b"Hello, world")

will be properly highlighted.

Also you can see, that some code blocks with async are already highlighted, e.g. first block in this section:
https://aiohttp.readthedocs.org/en/stable/web.html#custom-routing-criteria

I made Proof of Concept change for fixing one block in #769.

@asvetlov, I think this issue should be reopened and marked as "easy to fix", if my approach in #769 is suitable for us (if anyone want to fix this issue completely - PR is welcome).

asvetlov · 2016-02-06T08:49:14Z

I prefer a new issue.
The proposed solution is a bit verbose but I can live with that.

flying-sheep · 2016-02-06T09:45:36Z

uh, isn’t .. highlight:: python at the top of each file sufficient?

asvetlov · 2016-02-06T09:58:49Z

I used to believe highlight_language = 'python' in conf.py should be enough -- but we definitely see blocks marked up incorrectly.
I don't know why and how explicit .. code-block:: python fixes the issue.

flying-sheep · 2016-02-06T10:10:00Z

oh, yeah, if the conf.py isn’t sufficient and somehow being explicit per code block means something else, that’s weird.

kxepal · 2016-02-06T10:18:22Z

This looks as RTD specific issue. Anyone can reproduce it with local build? I don't.

rutsky · 2016-02-06T11:47:04Z

@kxepal make html in docs with deps installed from dev-requirements.txt on my machine generates not-highlighted code blocks, so the issue is reproduced for me locally too.
Is all blocks are highlighted when you build docs locally, @kxepal?

@asvetlov what is the reason for a new issue? I think discussion in this issue is important, no artifact were produced when the issue was closed (so nothing to invalidate), and there is no formal process that stops issue from being reopened.

kxepal · 2016-02-06T11:52:45Z

@rutsky Did the same. At least discussed web.rst case highlighted correctly as like as the main page. Also tried to install exactly the same deps versions that RTD uses (sphinx-1.3.4, pygments-2.0.2 in contrast to sphinx-1.3.5 and pygments-2.1) - all highlighted fine.

Also noticed that generated HTML on RTD contains right highlight hits about the language, but the python code itself wasn't tagged with the highlight markup. No idea why.

asvetlov · 2016-02-06T12:36:20Z

Reopen the issue.

Maybe raising minimal versions for sphinx and pygments in requirement files will help?

kxepal · 2016-02-06T12:51:34Z

If only find which exact versions to pin...

Btw, on RTD there is a build issue:

/home/docs/checkouts/readthedocs.org/user_builds/aiohttp/checkouts/latest/docs/api.rst:29: WARNING: autodoc: failed to import method 'AsyncStreamReaderMixin.iter_chunked' from module 'aiohttp.streams'; the following exception was raised:
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/aiohttp/envs/latest/lib/python3.4/site-packages/sphinx/util/inspect.py", line 109, in safe_getattr
    return getattr(obj, name, *defargs)
AttributeError: type object 'AsyncStreamReaderMixin' has no attribute 'iter_chunked'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/aiohttp/envs/latest/lib/python3.4/site-packages/sphinx/ext/autodoc.py", line 392, in import_object
    obj = self.get_attr(obj, part)
  File "/home/docs/checkouts/readthedocs.org/user_builds/aiohttp/envs/latest/lib/python3.4/site-packages/sphinx/ext/autodoc.py", line 288, in get_attr
    return safe_getattr(obj, name, *defargs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/aiohttp/envs/latest/lib/python3.4/site-packages/sphinx/util/inspect.py", line 115, in safe_getattr
    raise AttributeError(name)
AttributeError: iter_chunked
/home/docs/checkouts/readthedocs.org/user_builds/aiohttp/checkouts/latest/docs/api.rst:30: WARNING: autodoc: failed to import method 'AsyncStreamReaderMixin.iter_any' from module 'aiohttp.streams'; the following exception was raised:
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/aiohttp/envs/latest/lib/python3.4/site-packages/sphinx/util/inspect.py", line 109, in safe_getattr
    return getattr(obj, name, *defargs)
AttributeError: type object 'AsyncStreamReaderMixin' has no attribute 'iter_any'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/aiohttp/envs/latest/lib/python3.4/site-packages/sphinx/ext/autodoc.py", line 392, in import_object
    obj = self.get_attr(obj, part)
  File "/home/docs/checkouts/readthedocs.org/user_builds/aiohttp/envs/latest/lib/python3.4/site-packages/sphinx/ext/autodoc.py", line 288, in get_attr
    return safe_getattr(obj, name, *defargs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/aiohttp/envs/latest/lib/python3.4/site-packages/sphinx/util/inspect.py", line 115, in safe_getattr
    raise AttributeError(name)
AttributeError: iter_any

Which I don't observe locally even with the very verbose mode on ¯_(ツ)_/¯.

rutsky · 2016-02-06T14:31:17Z

@kxepal that build issue probably because docs are built on Python 3.4 in which AsyncStreamReaderMixin doesn't have iter_any: https://github.com/KeepSafe/aiohttp/blob/master/aiohttp/streams.py#L46
I see same error locally on Python 3.4, and don't see that error on Python 3.5.

kxepal · 2016-02-06T17:18:49Z

Ah, that's it! I used 3.5 and there everything is fine. Build with 3.4 indeed has no code highlight.

rutsky · 2016-02-06T17:32:07Z

@kxepal, indeed, docs built with Python 3.5 has proper syntax highlighting in my environment too!

So is it possible to make RTD use Python 3.5?

kxepal · 2016-02-06T17:38:14Z

Suddenly, they only allows to choose from generic CPython 2.x / CPython 3.x, not exact version /:

rutsky · 2016-02-06T17:47:51Z

Here is related issue on RTD: readthedocs/readthedocs.org#1865

As I understand @aaugustin, this issue will be fixed with next Pygments release (probably even on Python 3.4).

kxepal · 2016-02-06T17:52:53Z

Nice! However, what I don't understand: if next Pygment release pretends to fix the issue, why .. code-block:: python trick works today? o.O
Sounds like there are two ways to highlight the code in sphinx and one of them works almost right.

jashandeep-sohi · 2016-02-06T23:12:56Z

@asvetlov why was highlight_language = 'python3' changed to highlight_language = 'python' in conf.py ? I thought #650 fixed this.

kxepal · 2016-02-07T04:28:52Z

Indeed, highlight_language = 'python3' helps. However, it generates more warnings for sphinx-1.3.5:

/home/kxepal/projects/aio-libs/aiohttp/docs/client_reference.rst:446: WARNING: Could not parse literal_block as "python3". highlighting skipped.                                                                                
/home/kxepal/projects/aio-libs/aiohttp/docs/contributing.rst:46: WARNING: Could not parse literal_block as "python3". highlighting skipped.                                                                                     
/home/kxepal/projects/aio-libs/aiohttp/docs/contributing.rst:51: WARNING: Could not parse literal_block as "python3". highlighting skipped.
/home/kxepal/projects/aio-libs/aiohttp/docs/contributing.rst:56: WARNING: Could not parse literal_block as "python3". highlighting skipped.
/home/kxepal/projects/aio-libs/aiohttp/docs/contributing.rst:64: WARNING: Could not parse literal_block as "python3". highlighting skipped.
/home/kxepal/projects/aio-libs/aiohttp/docs/contributing.rst:68: WARNING: Could not parse literal_block as "python3". highlighting skipped.
/home/kxepal/projects/aio-libs/aiohttp/docs/contributing.rst:79: WARNING: Could not parse literal_block as "python3". highlighting skipped.
/home/kxepal/projects/aio-libs/aiohttp/docs/contributing.rst:98: WARNING: Could not parse literal_block as "python3". highlighting skipped.
/home/kxepal/projects/aio-libs/aiohttp/docs/contributing.rst:115: WARNING: Could not parse literal_block as "python3". highlighting skipped.
/home/kxepal/projects/aio-libs/aiohttp/docs/index.rst:29: WARNING: Could not parse literal_block as "python3". highlighting skipped.                                                                                            
/home/kxepal/projects/aio-libs/aiohttp/docs/index.rst:34: WARNING: Could not parse literal_block as "python3". highlighting skipped.
/home/kxepal/projects/aio-libs/aiohttp/docs/index.rst:112: WARNING: Could not parse literal_block as "python3". highlighting skipped.
/home/kxepal/projects/aio-libs/aiohttp/docs/multipart.rst:219: WARNING: Could not parse literal_block as "python3". highlighting skipped.                                                                                       
/home/kxepal/projects/aio-libs/aiohttp/docs/web.rst:57: WARNING: Could not parse literal_block as "python3". highlighting skipped.                                                                                              
/home/kxepal/projects/aio-libs/aiohttp/docs/web.rst:950: WARNING: Could not parse literal_block as "python3". highlighting skipped.

jashandeep-sohi · 2016-02-07T04:55:21Z

@kxepal Yeah okay, warnings were added in sphinx-doc/sphinx#1565.

But it's still highlighting the python parts for me. What I don't understand is why doesn't highlight = python raise these warning for the same code blocks. They are still invalid python. Maybe the python lexer is more forgiving.

And just to be clear the warning are coming from blocks which are not python. i.e. the command-line examples.

kxepal · 2016-02-07T05:04:36Z

@jashandeep-sohi I can only quote myself

¯_(ツ)_/¯

So what would be the fix? We seems have three ways to go:

highlight_language=python3 and explicitly set code blocks where warnings raises;
Explicitly set .. code-block:: python for async/await examples as @rutsky proposed;
Do nothing and wait for 3.5 || new Pygment on RTD;

Imho, first one is good one: we have highlighting fixed and explicit warnings about where we have a problems. With code-block it would be easy to miss something since noone warns about broken highlight. I fear we'll wait for too long to have third way work.

jashandeep-sohi · 2016-02-07T05:30:31Z

re the 1st & 2nd point: IMO, highlight language should be python3 no matter what. To the best of my knowledge, python is the lexer for Python 2.x & python3 is the lexer for Python 3.x.

re 3rd point: Doesn't RTD use the pygments version specified in requirements-ci.txt?
Pygments 2.1 was release w/ async/await support. So, what would happen if you set pygments>=2.1 in there?

kxepal · 2016-02-07T05:37:21Z

@jashandeep-sohi No highlight using Pygments==2.1 with Python 3.4 for me.

jashandeep-sohi · 2016-02-07T05:53:54Z

Arrrhhh...this is annoying. Can't wait for Python 3.4 to die ;-D

kxepal · 2016-02-07T06:05:45Z

@jashandeep-sohi 👍

jashandeep-sohi · 2016-02-07T06:28:52Z

@kxepal
I have syntax highlighting (with async/await) working on RTD. http://3f2e08e98e2f74684b61aa4ac7d5a037.readthedocs.org/en/latest/

Only changes (from master) are

highlight_language = 'python3' in docs/conf.py
pygments >= 2.1 in requirements-ci.txt

kxepal · 2016-02-07T08:10:41Z

@jashandeep-sohi yeap, the first one. Would you like to submit a patch? Related build warnings need to be fixed as well I think.

asvetlov · 2016-02-07T08:31:02Z

@jashandeep-sohi if highlight_language = 'python3' works I'll revert it back.
Removed because I scared warning messages mentioned in the thread

jashandeep-sohi · 2016-02-07T08:34:05Z

@asvetlov Yes, please revert. Also, you might want to require pygments >= 2.1 to have async/await highlighted.

The only errors/warnings I'm seeing ATM are related to AsyncStreamReaderMixin. This is a separate issue that should probably be fixed by removing the conditional method definitions.

asvetlov · 2016-02-07T08:55:32Z

@jashandeep-sohi fixed by f1b7c9c
ReadTheDocs will be updated on aiohttp 0.21.1 publishing.

jashandeep-sohi · 2016-02-07T09:00:35Z

👀 👍 http://aiohttp.readthedocs.org/en/latest/

asvetlov · 2016-02-07T09:05:06Z

Closing the issue again.

This fixes the syntax highlighting on readthedocs. See issue #591 of aiohttp for more information: aio-libs/aiohttp#591

lock · 2019-10-29T13:01:50Z

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.

flying-sheep closed this as completed Oct 25, 2015

rutsky added a commit to rutsky/aiohttp that referenced this issue Feb 5, 2016

POC of syntax highlighting fixing (aio-libs#591)

d670a65

asvetlov reopened this Feb 6, 2016

asvetlov closed this as completed Feb 7, 2016

rutsky mentioned this issue Feb 7, 2016

POC of syntax highlighting fixing (#591) #769

Closed

vxgmichel pushed a commit to asyncio-docs/asyncio-doc that referenced this issue Jul 26, 2016

Add python3 as default highlighting language

6e5b44d

This fixes the syntax highlighting on readthedocs. See issue #591 of aiohttp for more information: aio-libs/aiohttp#591

aio-libs-bot mentioned this issue Jan 11, 2019

Invalid syntax #3521

Closed

lock bot added the outdated label Oct 29, 2019

lock bot locked as resolved and limited conversation to collaborators Oct 29, 2019

docs syntax highlighting missing #591

docs syntax highlighting missing #591

Comments

flying-sheep commented Oct 25, 2015

asvetlov commented Oct 25, 2015

flying-sheep commented Oct 25, 2015

asvetlov commented Oct 25, 2015

flying-sheep commented Oct 25, 2015

asvetlov commented Oct 25, 2015

flying-sheep commented Oct 26, 2015

rutsky commented Feb 5, 2016

asvetlov commented Feb 6, 2016

flying-sheep commented Feb 6, 2016

asvetlov commented Feb 6, 2016

flying-sheep commented Feb 6, 2016

kxepal commented Feb 6, 2016

rutsky commented Feb 6, 2016

kxepal commented Feb 6, 2016

asvetlov commented Feb 6, 2016

kxepal commented Feb 6, 2016

rutsky commented Feb 6, 2016

kxepal commented Feb 6, 2016

rutsky commented Feb 6, 2016

kxepal commented Feb 6, 2016

rutsky commented Feb 6, 2016

kxepal commented Feb 6, 2016

jashandeep-sohi commented Feb 6, 2016

kxepal commented Feb 7, 2016

jashandeep-sohi commented Feb 7, 2016

kxepal commented Feb 7, 2016

jashandeep-sohi commented Feb 7, 2016

kxepal commented Feb 7, 2016

jashandeep-sohi commented Feb 7, 2016

kxepal commented Feb 7, 2016

jashandeep-sohi commented Feb 7, 2016

kxepal commented Feb 7, 2016

asvetlov commented Feb 7, 2016

jashandeep-sohi commented Feb 7, 2016

asvetlov commented Feb 7, 2016

jashandeep-sohi commented Feb 7, 2016

asvetlov commented Feb 7, 2016

lock bot commented Oct 29, 2019