Build PEPs using Sphinx

[brettcannon]

Might make things easier to manage, style, etc.

[Carreau]

Does that have any impacts on https://www.python.org/dev/peps/* ?(which already renders the peps well)

[birkenfeld]

Indeed, since the main purpose is to be rendered on www.python.org I don't think this would change much. The added markup also isn't really used for PEPs.

[brettcannon]

So I envision this tying into #3 so that we can host things on Read the Docs and have a sub-domain for the PEPs.

[birkenfeld]

Making that match the design and style currently used on python.org would be tedious, though.

[Carreau]

So I envision this tying into #3 so that we can host things on Read the Docs and have a sub-domain for the PEPs.

So would that mean pepe would no longer be displayed on Python.org ?

Also from other project i'm contributing too, multiple domains for different section of a website are painful, (especially on readthedocs) because the navigation does not work well across site.

It would though most likely be feasible to build this on travis and cross-push on the python/pythondotorg repository to trigger a rebuild of the main website. Or of you don't like that, push a separate branch and auto submit a PR.

[brettcannon]

I don't care if the style changes. I just think having the PEPs buried in www.python.org and having such a separate build process is odd, especially when the contents have no direct connection to what's on www.python.org (who directly links to the PEPs from www.python.org anyway beyond navigation links?)

As for setting up some way to have pythondotorg automatically re-build the PEPs when a commit occurs, I'm quite happy to see that happen.

[Carreau]

I don't care if the style changes. I just think having the PEPs buried in www.python.org and having such a separate build process is odd, especially when the contents have no direct connection to what's on www.python.org (who directly links to the PEPs from www.python.org anyway beyond navigation links?)

As for setting up some way to have pythondotorg automatically re-build the PEPs when a commit occurs, I'm quite happy to see that happen.

Thanks, that's a much simpler problem if we have less backward compatibility.

[brettcannon]

I'm combining #3 with this issue as there's no point in making the PEPs build with Sphinx if we don't create a subdomain of peps.python.org that is hosted on Read the Docs.

[foonsun]

thanks for your effort

[flying-sheep]

for fun i have created a PEP server that can be used to iterate on PEPs while seeing them in their final style.

it could be useful to get something close to the way PEPs look on python.org (since i already did the work to create a HTML template managing that does that):

https://gist.github.com/flying-sheep/c296482b785662d505f06c8bfe32f16e

it includes a server that serves HTML renders of PEPs without writing files. the pages contain live.js for auto-reloading and link the python.org stylesheet, so you can just edit away.

should i do a PR with it?

if yes, i’m interested in the scope for the provided scripts: are they all just for development? why the RSS stuff then?

[brettcannon]

The scripts in the project are actually used to generate the final files served at python.org/dev/peps.

As for a PR, feel free to propose it but I don't know what to think about it. I personally have never been bothered by the lack of styling when viewing generated PEPs (but then again I don't even bother with that anymore since we moved to GitHub when making edits to an existing PEP). If you want to discover this further, please open a new issue since this doesn't directly tie into using Sphinx for building.

[flying-sheep]

I just learned that using Sphinx would also fix #165 since Sphinx converts typewriter quotes to typographically correct ones, very nice 😄

[ncoghlan]

Noting what I'd personally consider the main advantage of switching to Sphinx for PEP generation: we'd be able to set up intersphinx inventory support referencing the development version of the CPython documentation, so we'd gain access to all our usual :term: glossary entries and other cross references to modules and API entries.

Of course, the downside of that is that changes to the dev docs could break references from old PEPs.

[brettcannon]

Switching to Sphinx also gets us Pygments support for code examples.

As for broken links, that's called the internet. 😉 I would assume the text in the PEP would be unambiguous enough to make it obvious what the PEP was referring to.

[ncoghlan]

#701 is another problem that would be resolved by switching to Sphinx: it turns out SVGs don't currently work because the HTML 4.1 docutils writer that we're using emits them as an object tag instead of an img tag (for IE6 compatibility!), and the rendering code on the python.org side of things doesn't fix up the link correctly the way it does for image tags.

[AA-Turner]

No worries @matrixise - @pablogsal has reviewed several of them, the only one left for now is #1933

A

[brettcannon]

I guess either the SC, Documentation WG, or PSF infrastructure would probably make the call as to whether moving to Read the Docs was the thing to do.

[warsaw]

I would think the Doc WG would at least make a recommendation to the SC.

[AA-Turner]

Hi Barry,

Sorry for the naive question - what's the Doc WG? I didn't realise that this work had to be submitted to someone for approval, sorry - I might need some help on writing a proposal or whatever is required.

Thanks,
Adam

[warsaw]

I'm not sure that the Doc WG is off the ground yet. @willingc and @Mariatta can probably provide insight on that.

[AA-Turner]

Fair enough, do let me know what will be required (if anything) in terms of formal proposals and I will have a stab at it :)

For now I'll focus on the sphinx-integration side, I realised that there was one issue I somehow missed (code/literal markup in titles, for examle PEP 604) - the current rendering on python dot org just strips the grave accents, but I'd like to try rendering it within the header tags.

A

[willingc]

Thanks @AA-Turner. Sorry for the late response.

I absolutely think that the PEPs should be built using Sphinx for better readability and rendering. One of the big benefits that I see with RTD is the ability to provide a current view of PEPs that are still in draft PR status since it is straightforward to seeing the rendered review version without having to pull locally and render.

As a first step, let's create a Sphinx doc hosted on RTD from the content in this repo (similar to how we do the dev guide).
We could host initially on RTD so folks can see a more nicely rendered page.

Things beyond this such as peps on python dot org or integration into other places can be treated as separate issues.

[AA-Turner]

Thanks for the note @willingc :)

To sort some things out in my thinking, I tried to draw up the pros of each as I saw them (attempting objectivity!). Personal commentary is at the very end.

Pro GH-Pages:

• Easy/integrated observability of builds (for debugging, if nothing else)
• Direct integration with GH (this could also be seen as a con)
• Local build and CI builds are identical (RTD injects additional config https://github.com/readthedocs/readthedocs.org/blob/master/readthedocs/doc_builder/templates/doc_builder/conf.py.tmpl)

Pro Read the Docs:

• Already used by the Python org (devguide)
• Has workflow convenience elements (e.g. PR builds)
• Can keep multiple versions (though likely less relevant for PEPs)

Pro Read the Docs (that I don't think are relevant):

• Automatic ePub/PDF/etc -- As I've overridden 'native' sphinx so much, additional work is needed to render ePubs nicely (see Create ePub versions of PEPs #1994 / Sphinx - epub #1995). Note, I am speculating here that the main body of what RtD does is run a parallel build with the ePub 3 builder from Sphinx -- if that isn't the case more investigative work is needed.
• Integrated analytics -- probably not relevant for PEPs!

Shared Pros:

• Custom domains
• CI integration
• Free (ish) hosting (RtD is advertising-supported)

(please excuse me sorry if I have missed any pros on either side, this was from a quick-ish read through of the features bits of both products etc)

(Personal) summary:

I have done the Sphinx support with GH-pages in mind, partly as of the simplicity of the product - it doesn't do anything that I don't want it to. I don't know what the impacts of the additional config code that RtD adds are (although I could probably guess by reading through fully), and it's hard to reproduce locally (except by adding in all of the additional config, of course).

So I would probably favour GH-pages, but I do see the value of automatic previews of content, which is partly what #1996 was trying to solve.

In the spirit of sportsmanlike conduct though I am going to set up a RtD environment, as I don't want to be a blocker to my own work, and I appreciate that you are a founding member (with @Mariatta) of the body that decides this work's fate, so to speak!

Thanks,
Adam

[AA-Turner]

Set up a test RtD site (more accurately, found one I was using from last year) and tried to build - config file here (https://github.com/AA-Turner/peps/blob/sphinx-read-the-docs-test/readthedocs.yaml)

Got a failure (https://readthedocs.org/projects/peps-aaturner/builds/14124926/) -- unless I'm doing something wrong, it appears that read the docs does not support Python 3.9.

I also couldn't find a way to use a custom build script (currently build.py) -- this would be mildly annoying as there is some consequential stuff in there, but I believe it could go into __init__.py as a pre/post build hook.

Don't really know how to proceed from here (I guess I could rewrite the entire package using Python 3.8 syntax but that is less than ideal)

Sorry,
Adam

[willingc]

Let's do this @AA-Turner (Thanks for the detailed response too.). I've opened an issue on the repo for the docs workgroup (which we are still getting ready for a virtual meeting) python/docs-community#10. My recommendation to get us moving forward is to deploy to both RTD and using GH-pages initially.

I have a couple of days of vacation later this week and can look closer at the existing PRs or folks that have privileges please feel free to merge those that are approved and don't break any other project's workflow.

On the issue of Python 3.9 I'm pretty sure that I have used that on some Jupyter RTD projects but my memory is fuzzy. We tend to set up stuff using a RTD yaml file instead of the RTD GUI.

Let's keep the existing PRs moving and add RTD support as well. Thanks @AA-Turner

[warsaw]

See this ticket about Python 3.9.

Since GHP doesn't support Sphinx out of the box (only Jekyll for Markdown AFAIK), are you planning on using Actions to call out to Sphinx if you go that route?

[AA-Turner]

are you planning on using Actions to call out to Sphinx

The build script is already committed! https://github.com/python/peps/blob/master/.github/workflows/deploy-gh-pages.yaml

(GHA calls make which calls build.py which orchastrates the process)

A

[AA-Turner]

My recommendation to get us moving forward is to deploy to both RTD and using GH-pages initially.

Sounds good, thanks Carol!

[ericholscher]

We can definitely get 3.9 running on RTD. I will ping @astrojuanlu to look into this soon. We're also happy to remove ads for the PEPs repo, if that's an issue.

Another benefit of using RTD is that we're working on a number of additional features that will be useful for this. In particular, content embedding -- as well as generally having a more powerful search experience than vanilla Sphinx. The RTD team is also committed to helping work to make this a success, which is probably another Pro :)

[AA-Turner]

Thanks Eric - I'm going to have a go on the testing image which supports 3.9, but part of me is pretty nervous deploying something explicitly labelled as testing!

Will look through the rest in the morning

A

[ericholscher]

@AA-Turner Ha, no worries on testing. It's basically just a way for us to provide it while we work on a better plan for upgrading our docker images. It should be as stable as any other build.

[astrojuanlu]

Thanks for the ping @ericholscher, I am now up to date with the whole conversation. Sent a small PR to save @AA-Turner some minutes of searching through our docs :) happy to help with anything else AA-Turner#3

[willingc]

This is coming along nicely: https://python.github.io/peps/

Thanks to all for working on this 🍰

[AA-Turner]

#2023 looks to add support for read the docs (or any other external documentation builder that supports Sphinx). Rendered preview at https://peps-aaturner.readthedocs.io/

A

[astrojuanlu]

xref: conversation continues at https://discuss.python.org/t/request-for-comment-making-pep-rendering-self-contained/10774?u=astrojuanlu

[AA-Turner]

small update: PEP 676 has been announced to python-dev at https://mail.python.org/archives/list/python-dev@python.org/thread/C675PLAF535FSUL7KX4FS2NK6ZPPQ3HB/

A

[AA-Turner]

Update: PEP 676 has been accepted!

I'll leave this issue open until the implementation work is fully finished.

A