Emit major version based canonical URLs for docs

[ncoghlan]

BPO	26355
Nosy	@birkenfeld, @ncoghlan, @ezio-melotti, @berkerpeksag, @Carreau, @wheerd
PRs	[Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
Files	cannonical-doc-for-3.4plus.patch cannonical-doc-for-2.7.patch cannonical-doc-for-2.6.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ncoghlan'
closed_at = <Date 2017-02-09.16:07:27.661>
created_at = <Date 2016-02-13.08:52:19.337>
labels = ['type-feature', '3.7', 'docs']
title = 'Emit major version based canonical URLs for docs'
updated_at = <Date 2017-03-31.16:36:30.069>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2017-03-31.16:36:30.069>
actor = 'dstufft'
assignee = 'ncoghlan'
closed = True
closed_date = <Date 2017-02-09.16:07:27.661>
closer = 'ncoghlan'
components = ['Documentation']
creation = <Date 2016-02-13.08:52:19.337>
creator = 'ncoghlan'
dependencies = []
files = ['46563', '46564', '46565']
hgrepos = []
issue_num = 26355
keywords = ['patch']
message_count = 22.0
messages = ['260227', '260237', '286319', '286336', '286339', '286342', '286344', '287139', '287191', '287203', '287226', '287249', '287419', '287425', '287430', '287431', '287432', '287433', '287437', '287441', '287445', '287448']
nosy_count = 8.0
nosy_names = ['georg.brandl', 'ncoghlan', 'ezio.melotti', 'docs@python', 'python-dev', 'berker.peksag', 'mbussonn', 'Wheerd']
pr_nums = ['552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue26355'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']

[ncoghlan]

Based on a recent comment on one of the mailing lists, I spent a bit of time looking into canonical URLs and their implications for how search engines treat versioned documentation.

The most relevant resource for that in relation to the CPython docs appears to be this page on ReadTheDocs: http://docs.readthedocs.org/en/latest/canonical.html

I sometimes encounter direct links to particular 3.x versions in search results, so I'm wondering if it might be desirable to set up the Python 3 docs to report their canonical URL as "/3/<path>", and the Python 2 docs as "/2/<path>".

[birkenfeld]

So we'd have to add a <link> tag? Should be easy.

[ncoghlan]

Belatedly following up on this, yeah, the RTFD page indicates that the header link should look like:

<link rel="canonical" href="http://docs.python.org/3/<path>">

[Carreau]

Does this have to be implemented on the doc build of EOL pythons versions (like 2.6), or can it be a script which is ran once on these old docs ?

One of the issues I had trying to implement that on other projects was that you don't know in advance what the future pages will be or if it will be gone. So you can go this route[1], in which case it's a 10 line fix that I'm happy to contribute also here.

Typically https://docs.python.org/2/c-api/string.html has no cannonical on 3 whouch you can't know while building with sphinx.

Or I though about injecting an html comment with a html comment that you search and replace once you publish the "new" version and only if the target page exists on the new version.

So :

• is that ok to have non existing cannonical ?
• is that ok if adding <rel>s on docs is (or requires) a post-sphinx script ?

1: https://github.com/xonsh/xonsh/pull/1914/files

Thanks

[Carreau]

Here is a patch that add a canonical link to the current documentation that should apply cleanly to 3.4 and above. I can do similar for older versions.

[wheerd]

I create the last diff without creating a commit, so maybe this one works better.

[wheerd]

Sorry, I accidentally replied to the worng issue -.-

[ncoghlan]

Thanks Matthias!

Regarding 2v3, the layout differences aren't a problem, since the canonical URLs are separate (/2/* vs /3/). That's one of the benefits I actually hope for with this change - due to PEP-430, deep links still go to the Python 2 documentation by default, and once this change is made in the Python 2.7 branch it should teach search engines that those should start being presented in results as "/2/" qualified links.

We also don't tend to make wholesale changes to the URL layouts in the docs in X.Y releases, so I think the assumption of "the relative path of this page won't change" is fine.

Georg, Berker - any further thoughts before we make this change to 3.4+ and the 2.7 docs?

[Carreau]

Thanks Nick,

I can work on the similar patch for Python 2.7.
I'll wait for the migration to GitHub which IIRC should be soon.

Side question, is there some metrics (like google analytics) to know how much traffic there is on older Python docs and if it's worth patching ?

[berkerpeksag]

Patch looks good to me too, thanks! Just left a super minor comment on Rietveld.

You might want to commit this to 3.5+ because we don't daily build 3.4 docs anymore: https://github.com/python/docsbuild-scripts/blob/master/build_docs.py#L30

I didn't see any links from 2.6 or older Python 2 versions in search results before so I don't have a strong opinion on backporting the patch to 2.7 (I probably wouldn't bother with Python 2 docs anymore :))

[ncoghlan]

OK, I've marked 2.7 and 3.5+ as the minimal set of versions to get the change via the CPython source repo.

However, looking at the results of https://www.google.com/search?q=python+httplib and https://www.google.com/search?q=python+http+client I think it's going to be worth backfilling the old branches as well - having multiple different versions of the 3.x documentation showing up in search results isn't helpful to anyone.

[Carreau]

Updated patch to take comments into account (added space before /> for consistency). I'm still unfamiliar with hg so let me know if I did anything wrong.

[ncoghlan]

Looks good to me, so I'll apply these right now :)

[python-dev]

New changeset c63b09833141 by Nick Coghlan in branch '3.5':
Issue bpo-26355: Specify canonical URLs in docs pages
https://hg.python.org/cpython/rev/c63b09833141

New changeset 80970cf56048 by Nick Coghlan in branch '3.6':
Merge issue bpo-26355 fix from Python 3.5
https://hg.python.org/cpython/rev/80970cf56048

New changeset 26af402c291f by Nick Coghlan in branch 'default':
Merge issue bpo-26355 fix from 3.6
https://hg.python.org/cpython/rev/26af402c291f

[python-dev]

New changeset a47e20b636d2a5559e5831c6805df3cba1ddb2a1 by Nick Coghlan in branch 'master':
Issue bpo-26355: Specify canonical URLs in docs pages
a47e20b

New changeset 37150972faf660571a3ae7076a623087c06b8791 by Nick Coghlan in branch 'master':
Merge issue bpo-26355 fix from Python 3.5
3715097

New changeset d267bc695eaf9422668daedb9e44442696e01fe7 by Nick Coghlan in branch 'master':
Merge issue bpo-26355 fix from 3.6
d267bc6

[python-dev]

New changeset a47e20b636d2a5559e5831c6805df3cba1ddb2a1 by Nick Coghlan in branch '3.5':
Issue bpo-26355: Specify canonical URLs in docs pages
a47e20b

[python-dev]

New changeset b07d454e45a2 by Nick Coghlan in branch '2.7':
Issue bpo-26355: Specify canonical URLs in docs pages
https://hg.python.org/cpython/rev/b07d454e45a2

[ncoghlan]

OK, I'm marking this as closed, since it's as resolved as we can make it through a *CPython* change.

Since the old branches aren't autobuilt anymore, adding a canonical URL reference to them would presumably be a matter of running a script over the built docs.

[python-dev]

New changeset e8455e7137b9bd63d4c0183558161dba6ea00d32 by Nick Coghlan in branch '2.7':
Issue bpo-26355: Specify canonical URLs in docs pages
e8455e7

[Carreau]

OK, I'm marking this as closed, since it's as resolved as we can make it through a *CPython* change.

Thanks you Nick, I appreciate the time you took to do that. ANd thank you Berker for the review.

[ncoghlan]

Matthias - thanks for figuring out how to turn my "we should do this" idea into a change we've actually made :)

[Carreau]

turn my "we should do this" idea into a change we've actually made :)

Looking forward to being able to do this more on GitHub as I am more familiar with git.

Good luck for the transition, I'm pretty sure there will be some hard time in the next few weeks.