Doc: thoughts on documentation overhaul?

[alvyjudy]

Hi,

I feel currently the setuptools documentation is quite cluttered and it makes it hard to improve the documentation, as one had to think really hard where to put the new documentation pieces (e.g. the 8 open issues that has stalled).

Take the index for example, the most commonly used features are all clustered into one section, while the relatively trivial ones take up like 4/5 of the top entries. Also, the main section still mentions a lot about python eggs, which I suppose is considered deprecated? The order of each section is less than ideal and the actual content seems to assume a big deal of knowledge from distutils. It would be nicer if they could be organized in such fasion: 1) quick start, 2) user guide, 3) reference, 4) deprecated/discouraged.

Especially now with PEP 517 being pushed forward, it might be a good idea to overhaul the documentation to 1) encourage the adaptation of this new standard and 2) inform developers of the new role setuptools will be playing in this new landscape and most importantly 3) make it more reader friendly.

I would like to take on this challenge and improve the documentation if that's desired:)

[jaraco]

@alvyjudy I'd be delighted if you could help with that. Thanks for volunteering!

In order to not waste your time, I'd like to provide feedback incrementally as you work on it. I'd hate for you to overhaul the documentation only to have to revert half of it based on an understanding that I can't convey quickly. To that end, I recommend one of two approaches. Either,

• Break down the effort into small, incremental steps, editing only a few sections at a time, or
• Migrate most/all of the existing documentation into a "legacy" section and start from scratch in the index with a link to the legacy docs.

Regardless how you approach it, rest assured that we're eager to improve the documentation and your desire to help there warms my heart. I'll do everything I can to help keep you from being stalled on the effort. Feel free to ping me directly if you need attention on something.

[alvyjudy]

Hi @jaraco , thanks so much for your support I really appreciate it!

Regarding the approach, I would prefer to update the documentation incrementally considering that most of the content remain relevant, given how ubiquitous setuptools is and how many libraries still depend on it. Re-writing it from scratch and push everything to a legacy section might actually upset readers.

I think the first step is to reorganize the content order and index, with minimal editing, to make it more reader-friendly so that we can find a better angle to paint the PEP 517 landscape down the road.

I will open a new branch and PR and start editing the docs slowly. This whole process might take a while so for the feedback please take your time. I'll also adjust my own pace depending on how fast the review is taking place.

[jaraco]

In #1765, I've merged another large-scale doc change. Please be aware of that.

[alvyjudy]

I realize I haven't made the structure of the new doc clear in previous PRs so I will present it here for others interested too

setuptools/doc/
    index.txt
    userguide/
        quickstart.txt
        *.txt
        *.txt
    references/
        *.txt
        *.txt

@jaraco I have addressed other changes made to the doc in the latest PR (2146). Please let me know your thoughts on that (and general suggestions if there is any:)

[jaraco]

Note, I merged #2151. I attempted to merge master into this branch, but there were unrelated conflicts, so I cherry-picked it as 01a8762 instead.

[jaraco]

In 8cca69b, I've merged master into the docs/2093-docs-revamp branch. Unfortunately, the branch is emitting lots of warnings when building the docs:

docs develop-inst-noop: /Users/jaraco/code/public/pypa/setuptools
docs installed: alabaster==0.7.12,Babel==2.8.0,certifi==2020.6.20,chardet==3.0.4,docutils==0.16,idna==2.10,imagesize==1.2.0,jaraco.packaging==8.1.0,Jinja2==2.11.2,MarkupSafe==1.1.1,packaging==20.4,pip==20.2.3,Pygments==2.7.1,pygments-github-lexers==0.0.5,pyparsing==2.4.7,python-dateutil==2.8.1,pytz==2020.1,requests==2.24.0,rst.linker==2.0.0,-e git+gh://pypa/setuptools@8cca69b8ffd372ce5e3089cebbdbf92c071f3761#egg=setuptools,six==1.15.0,snowballstemmer==2.0.0,Sphinx==3.2.1,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==1.0.3,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.4,urllib3==1.25.10,wheel==0.34.2
docs run-test-pre: PYTHONHASHSEED='157388818'
docs run-test: commands[0] | python -m sphinx . /Users/jaraco/code/public/pypa/setuptools/build/html
Running Sphinx v3.2.1
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 0 source files that are out of date
updating environment: [config changed ('project')] 31 added, 0 changed, 0 removed
reading sources... [  3%] build_meta
reading sources... [  6%] deprecated/distutils-legacy
reading sources... [  9%] deprecated/easy_install
reading sources... [ 12%] deprecated/functionalities
reading sources... [ 16%] deprecated/index
reading sources... [ 19%] deprecated/python3
reading sources... [ 22%] deprecated/python_eggs
reading sources... [ 25%] developer-guide
reading sources... [ 29%] development
reading sources... [ 32%] history
reading sources... [ 35%] index
reading sources... [ 38%] pkg_resources
reading sources... [ 41%] python 2 sunset
reading sources... [ 45%] references/keywords
reading sources... [ 48%] releases
reading sources... [ 51%] roadmap
reading sources... [ 54%] setuptools
reading sources... [ 58%] userguide/commands
reading sources... [ 61%] userguide/datafiles
reading sources... [ 64%] userguide/declarative_config
reading sources... [ 67%] userguide/dependency_management
reading sources... [ 70%] userguide/development_mode
reading sources... [ 74%] userguide/distribution
reading sources... [ 77%] userguide/entry_point
reading sources... [ 80%] userguide/extension
reading sources... [ 83%] userguide/functionalities_rewrite
reading sources... [ 87%] userguide/index
reading sources... [ 90%] userguide/keywords
reading sources... [ 93%] userguide/miscellaneous
reading sources... [ 96%] userguide/package_discovery
reading sources... [100%] userguide/quickstart

/Users/jaraco/code/public/pypa/setuptools/docs/build_meta.txt:59: WARNING: Unknown target name: "declarative config".
/Users/jaraco/code/public/pypa/setuptools/docs/development.txt:30: WARNING: toctree contains reference to nonexisting document 'formats'
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/commands.txt:275: WARNING: Unknown target name: "adding new egg-info files".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/commands.txt:315: WARNING: Unknown target name: "specifying your project's version".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/datafiles.txt:20: WARNING: Unknown target name: "adding support for revision control systems".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/dependency_management.txt:280: WARNING: Unexpected indentation.
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/development_mode.txt:51: WARNING: Unknown target name: "develop".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/distribution.txt:23: WARNING: Unknown target name: "egg_info".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/distribution.txt:34: WARNING: Unknown target name: "rotate".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/distribution.txt:47: WARNING: Unknown target name: "alias".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/distribution.txt:59: WARNING: Unknown target name: "adding support for revision control systems".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/distribution.txt:116: WARNING: Unknown target name: "egg_info".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/distribution.txt:116: WARNING: Unknown target name: "alias".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/distribution.txt:240: WARNING: Unknown target name: "egg_info".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/entry_point.txt:38: WARNING: Unexpected indentation.
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/entry_point.txt:58: WARNING: Unexpected indentation.
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/entry_point.txt:149: WARNING: Unexpected indentation.
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/extension.txt:9: WARNING: Unknown target name: "dynamic discovery of services and plugins".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/keywords.txt:9: WARNING: Unknown target name: "including data files".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/keywords.txt:15: WARNING: Unknown target name: "including data files".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/keywords.txt:21: WARNING: Unknown target name: "including data files".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/keywords.txt:36: WARNING: Unknown target name: "declaring dependencies".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/keywords.txt:41: WARNING: Unknown target name: "dynamic discovery of services and plugins".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/keywords.txt:41: WARNING: Unknown target name: "automatic script creation".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/keywords.txt:49: WARNING: Unknown target name: "declaring dependencies".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/keywords.txt:81: WARNING: Unknown target name: "namespace packages".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/keywords.txt:101: WARNING: Unknown target name: "test".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/keywords.txt:101: WARNING: Unknown target name: "test".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/keywords.txt:154: WARNING: Unknown target name: "automatic resource extraction".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/miscellaneous.txt:41: WARNING: Unknown target name: "creating distutils extensions".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/miscellaneous.txt:41: WARNING: Unknown target name: "adding new egg-info files".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/miscellaneous.txt:69: WARNING: Unknown target name: "accessing data files at runtime".
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/quickstart.txt:82: WARNING: Inline literal start-string without end-string.
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/quickstart.txt:82: WARNING: Inline literal start-string without end-string.
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/quickstart.txt:82: WARNING: Inline literal start-string without end-string.
looking for now-outdated files... none found
pickling environment... done
checking consistency... /Users/jaraco/code/public/pypa/setuptools/docs/build_meta.txt: WARNING: document isn't included in any toctree
/Users/jaraco/code/public/pypa/setuptools/docs/deprecated/functionalities.txt: WARNING: document isn't included in any toctree
/Users/jaraco/code/public/pypa/setuptools/docs/history.txt: WARNING: document isn't included in any toctree
/Users/jaraco/code/public/pypa/setuptools/docs/pkg_resources.txt: WARNING: document isn't included in any toctree
/Users/jaraco/code/public/pypa/setuptools/docs/references/keywords.txt: WARNING: document isn't included in any toctree
/Users/jaraco/code/public/pypa/setuptools/docs/roadmap.txt: WARNING: document isn't included in any toctree
/Users/jaraco/code/public/pypa/setuptools/docs/setuptools.txt: WARNING: document isn't included in any toctree
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/functionalities_rewrite.txt: WARNING: document isn't included in any toctree
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/miscellaneous.txt: WARNING: document isn't included in any toctree
done
preparing documents... done
writing output... [  3%] build_meta
writing output... [  6%] deprecated/distutils-legacy
writing output... [  9%] deprecated/easy_install
writing output... [ 12%] deprecated/functionalities
writing output... [ 16%] deprecated/index
writing output... [ 19%] deprecated/python3
writing output... [ 22%] deprecated/python_eggs
writing output... [ 25%] developer-guide
writing output... [ 29%] development
writing output... [ 32%] history
writing output... [ 35%] index
writing output... [ 38%] pkg_resources
writing output... [ 41%] python 2 sunset
writing output... [ 45%] references/keywords
writing output... [ 48%] releases
writing output... [ 51%] roadmap
writing output... [ 54%] setuptools
writing output... [ 58%] userguide/commands
writing output... [ 61%] userguide/datafiles
writing output... [ 64%] userguide/declarative_config
writing output... [ 67%] userguide/dependency_management
writing output... [ 70%] userguide/development_mode
writing output... [ 74%] userguide/distribution
writing output... [ 77%] userguide/entry_point
writing output... [ 80%] userguide/extension
writing output... [ 83%] userguide/functionalities_rewrite
writing output... [ 87%] userguide/index
writing output... [ 90%] userguide/keywords
writing output... [ 93%] userguide/miscellaneous
writing output... [ 96%] userguide/package_discovery
writing output... [100%] userguide/quickstart

/Users/jaraco/code/public/pypa/setuptools/docs/pkg_resources.txt:145: WARNING: undefined label: namespace packages (if the link has no caption the label must precede a section header)
/Users/jaraco/code/public/pypa/setuptools/docs/references/keywords.txt:155: WARNING: undefined label: including data files (if the link has no caption the label must precede a section header)
/Users/jaraco/code/public/pypa/setuptools/docs/references/keywords.txt:161: WARNING: undefined label: including data files (if the link has no caption the label must precede a section header)
/Users/jaraco/code/public/pypa/setuptools/docs/references/keywords.txt:167: WARNING: undefined label: including data files (if the link has no caption the label must precede a section header)
/Users/jaraco/code/public/pypa/setuptools/docs/references/keywords.txt:182: WARNING: undefined label: declaring dependencies (if the link has no caption the label must precede a section header)
/Users/jaraco/code/public/pypa/setuptools/docs/references/keywords.txt:187: WARNING: undefined label: dynamic discovery of services and plugins (if the link has no caption the label must precede a section header)
/Users/jaraco/code/public/pypa/setuptools/docs/references/keywords.txt:187: WARNING: undefined label: automatic script creation (if the link has no caption the label must precede a section header)
/Users/jaraco/code/public/pypa/setuptools/docs/references/keywords.txt:195: WARNING: undefined label: declaring dependencies (if the link has no caption the label must precede a section header)
/Users/jaraco/code/public/pypa/setuptools/docs/references/keywords.txt:238: WARNING: undefined label: namespace packages (if the link has no caption the label must precede a section header)
/Users/jaraco/code/public/pypa/setuptools/docs/references/keywords.txt:312: WARNING: undefined label: automatic resource extraction (if the link has no caption the label must precede a section header)
/Users/jaraco/code/public/pypa/setuptools/docs/references/keywords.txt:319: WARNING: unknown document: python3
/Users/jaraco/code/public/pypa/setuptools/docs/references/keywords.txt:323: WARNING: unknown document: python3
/Users/jaraco/code/public/pypa/setuptools/docs/references/keywords.txt:327: WARNING: unknown document: python3
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/dependency_management.txt:21: WARNING: undefined label: quickstart (if the link has no caption the label must precede a section header)
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/dependency_management.txt:37: WARNING: undefined label: guide on deprecated practice (wip) (if the link has no caption the label must precede a section header)
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/entry_point.txt:137: WARNING: undefined label: dependency_management (if the link has no caption the label must precede a section header)
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/keywords.txt:161: WARNING: unknown document: python3
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/keywords.txt:165: WARNING: unknown document: python3
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/keywords.txt:169: WARNING: unknown document: python3
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/package_discovery.txt:8: WARNING: undefined label: keywords_ref (if the link has no caption the label must precede a section header)
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/package_discovery.txt:12: WARNING: undefined label: quickstart (if the link has no caption the label must precede a section header)
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/quickstart.txt:122: WARNING: undefined label: entry_point (if the link has no caption the label must precede a section header)
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/quickstart.txt:143: WARNING: undefined label: dependency_management (if the link has no caption the label must precede a section header)
/Users/jaraco/code/public/pypa/setuptools/docs/userguide/quickstart.txt:165: WARNING: undefined label: datafiles (if the link has no caption the label must precede a section header)
generating indices... done
writing additional pages...  searchdone
copying static files... ... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 68 warnings.

The HTML pages are in ../build/html.
___________________________________ summary ____________________________________
  docs: commands succeeded
  congratulations :)

Additionally, there are other problems, like 'keywords' exists in both references and userguide. I share the blame on that, as I approved the changes.

In retrospect, we probably should not have attempted to develop these changes in a branch but should instead have made incremental changes that were stable.

I was hoping to at least merge these changes into master and stop the divergence, but given the issues above, I'm reluctant to do that.

On the other hand, most of these changes were reviewed and the issues can probably be dealt with subsequently. Perhaps we just accept the warnings and push forward.