In ./Doc, "make html" and "make build" should depend on "make venv"

[jdevries3133]

BPO	44756
Nosy	@ned-deily, @encukou, @ambv, @hroncok, @pablogsal, @miss-islington, @jdevries3133
PRs	bpo-44756: in ./Doc, make build depends on make html #27403 [3.10] bpo-44756: in ./Doc, make build depends on make html (GH-27403) #27410 [3.9] bpo-44756: in ./Doc, make build depends on make html (GH-27403) #27411 bpo-44756: [docs] revert automated virtual environment creation on make html #27635 [3.9] bpo-44756: [docs] revert automated virtual environment creation on make html (GH-27635) #27636 [3.10] bpo-44756: [docs] revert automated virtual environment creation on make html (GH-27635) #27637 bpo-44756: Remove misleading NEWS entries of a change that was reverted before release #28075

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-08-06.19:53:19.845>
created_at = <Date 2021-07-28.04:47:25.985>
labels = ['3.9', '3.10', '3.11', 'type-feature', 'release-blocker', 'docs']
title = 'In ./Doc, "make html" and "make build" should depend on "make venv"'
updated_at = <Date 2021-08-30.21:56:40.474>
user = 'https://github.com/jdevries3133'

bugs.python.org fields:

activity = <Date 2021-08-30.21:56:40.474>
actor = 'lukasz.langa'
assignee = 'docs@python'
closed = True
closed_date = <Date 2021-08-06.19:53:19.845>
closer = 'lukasz.langa'
components = ['Demos and Tools', 'Documentation']
creation = <Date 2021-07-28.04:47:25.985>
creator = 'jack__d'
dependencies = []
files = []
hgrepos = []
issue_num = 44756
keywords = ['patch']
message_count = 19.0
messages = ['398344', '398373', '398381', '398383', '398384', '398827', '398840', '398848', '398868', '398874', '398875', '398884', '399101', '399116', '399124', '399130', '399135', '400672', '400673']
nosy_count = 8.0
nosy_names = ['ned.deily', 'petr.viktorin', 'docs@python', 'lukasz.langa', 'hroncok', 'pablogsal', 'miss-islington', 'jack__d']
pr_nums = ['27403', '27410', '27411', '27635', '27636', '27637', '28075']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue44756'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[jdevries3133]

In Doc/Makefile, all of the build rules should be dependent on the existence of a virtual environment. I could see this being controversial, because folks who have these tools installed elsewhere might prefer not to have a venv made for them, but my personal workflow is to strive to keep my system site-packages folder as empty as possible and to use virtual environments frequently. In any case, I'm interested to hear what we think.

Momentarily, I will attach a PR where I went ahead and made these changes. Here is a summary of the changes:

• venv rule is now conditional, and only does anything if $VENVDIR does
  not exist
• add rule "clean-venv". This can be removed – what do you all think?
• build rule is dependent on venv
• as a result, rules dependent on build are dependent on venv
  • html
  • latex
  • etc.
• update Doc/README.rst appropriately. Now users only need to type
  make html – nice!

Let me know what you think. I may have a blind spot to others' workflows! Also, I'm not a Windows user, so I have no insight as to whether make.bat needs to be updated as well.

[ambv]

New changeset d22c876 by Jack DeVries in branch 'main':
bpo-44756: in ./Doc, make build depends on make html (bpo-27403)
d22c876

[ambv]

New changeset f113195 by Miss Islington (bot) in branch '3.10':
bpo-44756: in ./Doc, make build depends on make html (GH-27403) (GH-27410)
f113195

[ambv]

New changeset fd2c246 by Miss Islington (bot) in branch '3.9':
bpo-44756: in ./Doc, make build depends on make html (GH-27403) (GH-27411)
fd2c246

[ambv]

Thanks, Jack! ✨ 🍰 ✨

[encukou]

I'm one of those who disagree with "make html" suddenly downloading code from the internet and running it, but I guess I'm in a minority. I'll switch to using sphinx-build directly.

Still, I'd have appreciated a heads-up notice.

[jdevries3133]

@petr.viktorin a whatsnew entry was added, what more notice could have been provided?

I have an idea for an alternative that might be better. What if make clean deletes and restores the venv only if it already existed in the first place?

[jdevries3133]

Actually, I tested out that idea (main...jdevries3133:bpo-44756-doc-make), and I don't think its as nice of a solution. I think it is valuable for new contributors to be able to type "make html" and have it work. Look at how much the change simplifies the README for new contributors to setup their documentation build toolchain. Then, look at how many docs@python open issues there are. We need documentation contributors, and there is value in simplifying the process for them.

Additionally, this is the extent of the "downloading code from the internet and running it" that occurs::

pip install -U pip setuptools
pip install sphynx blurb python-docs-theme

If running that is ever unsafe, we have big problems!

[hroncok]

The 3.10.0rc1 source tarballs contain the Docs/venv directory populated with pablogsal's venv: bpo-44824

[encukou]

The issue this (or lack of communication about it) caused in rc1 is tracked in https://bugs.python.org/issue44823

@petr.viktorin a whatsnew entry was added, what more notice could have been provided?

Ideally, the python-dev mailing list (or Discourse).

pip install sphinx blurb python-docs-theme
If running that is ever unsafe, we have big problems!

Who is "we"?
We do have big problems. Anyone who can upload wheels for sphinx blurb python-docs-theme or any of their dependencies (or anyone who has their credentials) can now easily put code on machines of CPython developers.

For example, PyPI doesn't guarantee that wheels correspond to sources. "Markupsafe" is particularly dangerous because the wheels are platform-specific and have compiled code, so tampering is nearly undetectable. (But if another dependency starts using platform-specific wheels, I don't think anyone would notice.)

[encukou]

I've sent a mail to python-dev: https://mail.python.org/archives/list/python-dev@python.org/thread/MGPNI7OSA7UXNOTVDVW2I2GUMXV25FRS/

[pablogsal]

Note this issue has impacted negatively the release process of 3.10.0rc1:

https://bugs.python.org/issue44823

[ned-deily]

There was a previous attempt at adding a dependency on venv that ended up being reverted (bpo-30487 and 122fc136b34e11906466851e77bb6959946467ee0). Over the years we have had a number of iterations of tweaking Doc/Makefile to balance ease of use (by providing the venv) with those of more advanced users who build docs as part of their automated processeses. I think what we had prior to this most recent change worked reasonably well for all. The previous process is clearly documented, for example in the Dev Guide (https://devguide.python.org/documenting/?highlight=venv#building-the-documentation) and does not strike me as very onerous. I appreciate the motivation behind this change but I really think there isn't a problem here that needs to be solved and I would support reverting the change for 3.9 and 3.10 and *perhaps* reconsider something for 3.11.

[ambv]

New changeset 55fa87b by Łukasz Langa in branch 'main':
bpo-44756: [docs] revert automated virtual environment creation on make html (GH-27635)
55fa87b

[ambv]

New changeset 91f6d38 by Miss Islington (bot) in branch '3.9':
bpo-44756: [docs] revert automated virtual environment creation on make html (GH-27635) (GH-27636)
91f6d38

[miss-islington]

New changeset 1e9c9ca by Miss Islington (bot) in branch '3.10':
bpo-44756: [docs] revert automated virtual environment creation on make html (GH-27635)
1e9c9ca

[ambv]

Revert done. Re-closing!

[ambv]

New changeset 5246dbc by Łukasz Langa in branch 'main':
bpo-44756: Remove misleading NEWS entries of a change that was reverted before release (GH-28075)
5246dbc

[ambv]

New changeset 9ef1843 by Łukasz Langa in branch '3.9':
bpo-44756: Remove misleading NEWS entries of a change that was reverted before release (GH-28075)
9ef1843