x/website: go.dev tends not to show up on Google search results, but tip.golang.org does

[cespare]

Go version

n/a

Output of go env in your module/workspace:

n/a

What did you do?

I performed Google searches for various documents that I know are hosted on go.dev. Example search queries:

• golang doc comments
• golang 1.23 release notes

What did you see happen?

The copy of the document on tip.golang.org was the first Google search result each time.

Here are the two examples:

(I noticed this on my personal Google account, but I took those two screenshots from a Chrome incognito window.)

What did you expect to see?

I expect the canonical version of these documents to be the first search result, or at least near the top.

• golang doc comments:
  • https://tip.golang.org/doc/comment is the first result
  • https://go.dev/doc/comment isn't anywhere in the first two pages of search results
• golang 1.23 release notes:
  • https://tip.golang.org/doc/go1.23 is the first result
  • https://go.dev/doc/go1.23 isn't anywhere in the first two pages of search results
  • Weirdly, the second search result is a go.dev URL, the "Go 1.23 is released" blog post at https://go.dev/blog/go1.23.

I guess for most projects I probably wouldn't file an SEO ticket, but you folks are mostly paid by Google, so I figure you ought to be able to sort it out :)

[dr2chase]

@golang/tools-team

[findleyr]

I'm guessing this is because golang.org had historical momentum, and tip.golang.org doesn't redirect in the same way that golang.org does.

Perhaps we should just introduce tip.go.dev, and redirect tip.golang.org. I imagine then both would be downranked, but perhaps eventually go.dev links would make their way to the top.

(I know we work at Google, but I for one have very little understanding of how this works :))

[cespare]

Perhaps I shouldn't have focused so much on tip.golang.org. My main point with filing this issue is that there seems to be some SEO issue preventing the canonical versions of important go.dev pages from appearing anywhere near the top of Google search results in my recent tests.

[kevinpeno]

I was going to post on #69785, but I figured it might be better to contain the conversation since this one had more traction. Sorry if this is the wrong place.

---

Since it looks like most of your content is dynamic (I'm not seeing much beyond basic route configs in the code) you might be better off making sure you have redirects qualified correctly from the legacy sites (and maybe tip, see below) and adding a link[rel="canonical"] on each page rather than generating a sitemap (link[rel="canonical"] is a hard canonical vs sitemaps soft).

After that, ensure you have a robot.txt file pointing to your core directories/roots and then submit it to Google Search Console (this will trigger the initial "re-spidering"). The current one doesn't seem sufficient enough, but it might be okay if you meta+no-index appropriately.

Finally, you could probably clean up by adding link tags for language variants (link[hreflang]).

It also appears that tip.golang.org is duplicating go.dev content, which will cause search console issues and result in poorer results if they both try to canonize (like those found in OP). I don't currently understand the purpose of tip.golang.org (is it meant to redirect to the "latest" on go.dev?), so I'll defer to your judgment there.

I am happy to provide further guidance, but I cannot say much else without access to the search console and a clearer understanding of how the content is stored on these dynamic routes.