[DOCS] Keep archived doc versions out of search results - #3125
Merged
Conversation
Old, versioned documentation was being indexed by search engines and ranking above the current docs. For example, the 1.4.1 Discord invite page ranked as the top result for "sedona apache discord"; it was removed from current docs, so its rel=canonical points to a /latest/ URL that now 404s and search engines ignore the canonical and index the stale page. robots.txt Disallow does not fix this: a disallowed page can no longer be crawled, so search engines never observe a removal signal and may keep the stale URL indexed. Instead, tag every page of the archived versions and the development snapshot with a crawlable <meta name="robots" content="noindex"> so they can be crawled and dropped, while /latest/ stays indexable. Archived versions are frozen once released and never rebuilt, so the tag is applied by a post-deploy step (tools/noindex_archived_docs.py) that runs over the published website branch. It is idempotent and skips the current stable version, which latest symlinks to. The step backfills existing versions on the next deploy and tags each version as it is superseded by a new release.
- Only skip a page when its robots meta already contains noindex. If it carries other directives (e.g. "index, follow" or "noarchive"), merge noindex in and drop a conflicting "index" instead of leaving the page indexable. - Fail (exit non-zero) when "latest" is missing, is not a symlink, or points to a non-existent directory, instead of tagging every version including the current stable one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Did you read the Contributor Guide?
Is this PR related to a ticket?
[DOCS] my subjectWhat changes were proposed in this PR?
Old, versioned documentation is being indexed by search engines and ranking above the current docs. For example, searching "sedona apache discord" returns the outdated
1.4.1Discord invite page as the top result.Root cause, verified on the live site:
https://sedona.apache.org/1.4.1/community/discord-invite-form.htmlrel=canonical→https://sedona.apache.org/latest/community/discord-invite-form/robotsmeta tag on the pagemikealready setscanonical_version: latest, so archived pages carry a canonical to their/latest/equivalent — but when that equivalent has been removed, the canonical points at a 404 and search engines ignore it and index the stale archived page.The intuitive fix — a
robots.txtDisallowfor the archived paths — does not work here: a disallowed page can no longer be crawled, so search engines never observe a removal signal and may keep the stale URL in the index (Google explicitly advises against robots.txt for removal, and Bing similarly needs the page crawlable to see anoindex). The reliable signal on static hosting (no control over response headers / redirects / 410s) is a crawlable<meta name="robots" content="noindex">.This PR adds
tools/noindex_archived_docs.pyand a post-deploy step indocs.ymlthat:<meta name="robots" content="noindex, follow">to every page of the archived versions and the development snapshot on the publishedwebsitebranch, leaving them crawlable so search engines can drop them.latestis a symlink to that version's directory, so tagging it would de-index the live docs; its duplicate versioned URL (e.g./1.9.0/) is already consolidated onto/latest/by the existing canonical tag.<head>(the Discord invite pages in1.4.1/1.5.0— exactly the reported case) by prepending the tag, which the HTML parser hoists into the head.robots.txt(advertises/latest/sitemap.xml, blocks nothing) so thenoindexstays crawlable.Supersedes #3123 (the earlier robots.txt-only approach, which had this exact removal limitation).
How was this patch tested?
websitebranch: 19,671 archived pages tagged across all numbered versions +latest-snapshot; the current stable1.9.0(and thelatestsymlink) left untouched; 0 archived HTML pages left without a robots meta, including both headless Discord fragment pages.tools/noindex_archived_docs.pypassesblack,pyupgrade, and the license-header hook;docs.ymlpassesactionlintandyamllint(pre-commit).Did this PR include necessary documentation updates?