x/website: go.dev tends not to show up on Google search results, but tip.golang.org does

[cespare]

Go version

n/a

Output of go env in your module/workspace:

n/a

What did you do?

I performed Google searches for various documents that I know are hosted on go.dev. Example search queries:

• golang doc comments
• golang 1.23 release notes

What did you see happen?

The copy of the document on tip.golang.org was the first Google search result each time.

Here are the two examples:

(I noticed this on my personal Google account, but I took those two screenshots from a Chrome incognito window.)

What did you expect to see?

I expect the canonical version of these documents to be the first search result, or at least near the top.

• golang doc comments:
  • https://tip.golang.org/doc/comment is the first result
  • https://go.dev/doc/comment isn't anywhere in the first two pages of search results
• golang 1.23 release notes:
  • https://tip.golang.org/doc/go1.23 is the first result
  • https://go.dev/doc/go1.23 isn't anywhere in the first two pages of search results
  • Weirdly, the second search result is a go.dev URL, the "Go 1.23 is released" blog post at https://go.dev/blog/go1.23.

I guess for most projects I probably wouldn't file an SEO ticket, but you folks are mostly paid by Google, so I figure you ought to be able to sort it out :)

[dr2chase]

@golang/tools-team

[findleyr]

I'm guessing this is because golang.org had historical momentum, and tip.golang.org doesn't redirect in the same way that golang.org does.

Perhaps we should just introduce tip.go.dev, and redirect tip.golang.org. I imagine then both would be downranked, but perhaps eventually go.dev links would make their way to the top.

(I know we work at Google, but I for one have very little understanding of how this works :))

[cespare]

Perhaps I shouldn't have focused so much on tip.golang.org. My main point with filing this issue is that there seems to be some SEO issue preventing the canonical versions of important go.dev pages from appearing anywhere near the top of Google search results in my recent tests.

[kevinpeno]

I was going to post on #69785, but I figured it might be better to contain the conversation since this one had more traction. Sorry if this is the wrong place.

---

Since it looks like most of your content is dynamic (I'm not seeing much beyond basic route configs in the code) you might be better off making sure you have redirects qualified correctly from the legacy sites (and maybe tip, see below) and adding a link[rel="canonical"] on each page rather than generating a sitemap (link[rel="canonical"] is a hard canonical vs sitemaps soft).

After that, ensure you have a robot.txt file pointing to your core directories/roots and then submit it to Google Search Console (this will trigger the initial "re-spidering"). The current one doesn't seem sufficient enough, but it might be okay if you meta+no-index appropriately.

Finally, you could probably clean up by adding link tags for language variants (link[hreflang]).

It also appears that tip.golang.org is duplicating go.dev content, which will cause search console issues and result in poorer results if they both try to canonize (like those found in OP). I don't currently understand the purpose of tip.golang.org (is it meant to redirect to the "latest" on go.dev?), so I'll defer to your judgment there.

I am happy to provide further guidance, but I cannot say much else without access to the search console and a clearer understanding of how the content is stored on these dynamic routes.

[cespare]

Once again today I searched for "golang doc comments" and found that https://go.dev/doc/comment wasn't anywhere to be seen.

Based on what @kevinpeno's comment says and some more background reading and poking around on my own, I suspect that the issue is mostly that the content is duplicated between the sites. It appears that for each page, only one of two copies (the one on tip.golang.org or the one on go.dev) is highly ranked. I found some examples where it is go.dev: the Release History page seems to show the go.dev version rather than the tip.golang.org version.

Probably the "correct" fix is to stop using tip.golang.org and use some other URL scheme instead (e.g., https://go.dev/doc/comment?version=tip). But I do think that we can probably improve the situation by saying that go.dev is canonical. We would attach tags to all the tip.golang.org pages:

<link rel="canonical" href="https://go.dev/...">

@golang/pkgsite WDYT? I imagine this is a simple enough change.

[adonovan]

Thanks for poking around. Let's give the canonical link a try.

I guess for most projects I probably wouldn't file an SEO ticket, but you folks are mostly paid by Google, so I figure you ought to be able to sort it out :)

Yeah, SEO issues are generally handled by the division of East Coast Television and Microwave Oven Programming, whereas for tax reasons the Go team is actually a subsidiary of the Sheinhardt Wig Company.

[gopherbot]

Change https://go.dev/cl/712040 mentions this issue: _content/site.tmpl: add canonical link