Avoid Python 2 documentation to appear in Web search results

[bittner]

BPO	40012
Nosy	@terryjreedy, @bittner, @JulienPalard, @ammaraskar, @tirkarthi

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-04-02.16:16:52.734>
created_at = <Date 2020-03-19.09:35:58.986>
labels = ['type-feature']
title = 'Avoid Python 2 documentation to appear in Web search results'
updated_at = <Date 2021-04-02.16:16:52.733>
user = 'https://github.com/bittner'

bugs.python.org fields:

activity = <Date 2021-04-02.16:16:52.733>
actor = 'ammar2'
assignee = 'none'
closed = True
closed_date = <Date 2021-04-02.16:16:52.734>
closer = 'ammar2'
components = []
creation = <Date 2020-03-19.09:35:58.986>
creator = 'bittner'
dependencies = []
files = []
hgrepos = []
issue_num = 40012
keywords = []
message_count = 7.0
messages = ['364597', '364721', '364725', '364868', '365050', '365110', '390078']
nosy_count = 6.0
nosy_names = ['terry.reedy', 'bittner', 'mdk', 'ammar2', 'xtreak', 'Terry Davis']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue40012'
versions = ['Python 2.7']

[bittner]

Currently, when you do a Web search (e.g. using Google, Bing, Yahoo!, DuckDuckGo, et al.) for a Python module or function call you'll find a link to the related Python 2 documentation first.

How to reproduce:

1. Search for simply "os.environ" in your favorite search engine.
2. Find a link to the Python documentation in the first 3 results. Typically, this will point to the Python 2 docs first.

(Side note: Google seems to now actively manipulate the results ranking Python 3 results higher. Apparently, this is the only popular search engine behaving like that.)

Expected result:

• When searching for Python modules, functions, builtins, etc. on the Web, no search results for Python 2 should pop up at all if the same content exists for Python 3

Possible implementation:

• Add a "noindex" meta tag to the header of the generated HTML documentation
• see https://support.google.com/webmasters/answer/93710

[terryjreedy]

I completely agree with the goal. But I think that this is duplicate of at one previous issue either here or on the website tracker.

[tirkarthi]

See also the approach to use robots.txt https://github.com/python/pythondotorg/issues/1030

[TerryDavis]

It seems like using "noindex" to tell search engines not to look at Python 2 docs at all would have the side-effect of breaking Python 2 documentation searches.
I have gotten into the habit of searching for "python 3" explicitly, which I think is a reasonable work-around. Especially since python 2 and 3 could be considered different languages.

[ammaraskar]

Instead of noindex maybe the 3.x documentation can be marked as the canonical one: https://support.google.com/webmasters/answer/139066

This should still allow the old docs to be crawled but emphasize the latest docs on the website:

Google Search result usually points to the canonical page, unless one of the duplicates is explicitly better suited for a user: for example, the search result will probably point to the mobile page if the user is on a mobile device, even if the desktop page is marked as canonical.

Presumably the "better suited" means that if you search for "python2 timeit" you'd still find the py2 docs.

Terry Reedy, could you link the earlier issue so this information can be posted there, or are you referring to the issue Karthik linked?

[terryjreedy]

Sorry, the 'previous issue' is a vague memory which I cannot find. For some reason, searching this tracker for 'Google' return 1500+ hits.

[ammaraskar]

This has been fixed as part of python/docsbuild-scripts#99

Thank you to Julien, our Google results are much cleaner now :)