x/website: redirect golang.org/pkg and golang.org/cmd to pkg.go.dev

[rsc]

Thanks to significant work by @julieqiu and team on pkg.go.dev, godoc.org now redirects to pkg.go.dev, and we've now reduced the number of documentation servers run by the Go team from three to two. If pkg.go.dev is ready now to serve godoc.org's traffic, then it's also ready to serve golang.org's documentation traffic. We should do that, reducing the number of documentation servers from two to one (the ideal number!).

One concern we had was whether there are request parameters that pkg.go.dev does not support. I analyzed the logs, and the answer is no, there aren't. The most important ones are GOOS= and GOARCH=, and those work on pkg.go.dev for the standard library (other modules in progress).

The only other parameter with any frequency is ?m=all, which displays unexported API. We do not plan to add that feature to pkg.go.dev - it would expand the index significantly without creating much value, if any, for users. But most of the ?m=all requests are coming from search engine results that have inadvertently indexed that form of the page. Those would be served just as well (or better) by only showing exported API.

There is about one request per day using m=all to link to unexported functions, or about 0.0007% of traffic. That one request per day will break with the redirect, and that seems an acceptable cost for cutting our doc server count by 50%.

Details below.

Query parameters overall.

Downloaded 10,000 request URLs from App Engine logs matching "golang.org/pkg", spanning 96 minutes (1.7 QPS).
808 were from bots (user agent matching /bot|crawler/).

Only 39 of the 9,192 non-bot requests (0.42%) had query parameters.

21 used ?m=all to see unexported data; all were coming from www.google.com results
(which probably did not intend and are not well served by ?m=all).

9 fetched /pkg/syscall/?GOOS=windows.
Most hid the referer but one was from https://golang.org/doc/go1.16.
There were also
2 /pkg/syscall/?GOOS=darwin
2 /pkg/syscall/?GOOS=linux
1 /pkg/syscall/js/?GOOS=js&GOARCH=wasm (found via a search for [golang js.FuncOf])
1 /pkg/os/exec/?GOOS=windows

3 used ?source=post_page---------------------------, which does nothing.
They all came from duckduckgo.

m= parameter

Downloaded 500 request URLs matching "/pkg.*m=", spanning 41 hours (0.0034 QPS).
75 were from bots (user agent matching /bot|crawler/).
Of the 425 remaining, 414 used m=all, and 353 of those had Referer lines.
Of the 353 referers for m=all, there were:

• 331 from search engines (google, duckduckgo, yandex)
• 9 from m=all pages on golang.org (maybe using Referer = URL)
• 5 from pkg.go.dev/golang.org/x/tools/cmd/godoc, which mentions https://golang.org/pkg/math/big/?m=all
  (this would need to be updated)
• 5 from keyanp.com/cgo-segfault.html, which links to
  https://golang.org/pkg/runtime/?m=all#SetCgoTraceback
  (the m=all is unnecessary in that URL)
• 1 from API: Fix global URL when external address has no port prometheus/prometheus#8359, which links to
  https://golang.org/pkg/net/?m=all#JoinHostPort
  (the m=all is unnecessary in that URL)
• 1 from net/http: don't block RoundTrip when the Transport hits MaxConcurrentStreams #27044 (2018), which links to
  https://golang.org/pkg/net/http/?m=all#http2ClientConn.awaitOpenSlotForRequest
  (this would break if we removed m=all)
• 1 from https://syslog.ravelin.com/byte-vs-string-in-go-d645b67ca7ff (2016), which links to
  https://golang.org/pkg/runtime/?m=all#stringtoslicebyte
  https://golang.org/pkg/runtime/?m=all#slicebytetostring
  https://golang.org/pkg/runtime/?m=all#slicebytetostringtmp
  https://golang.org/pkg/runtime/?m=all#stringtoslicebytetmp
  (this would break if we removed m=all)

Of the 11 non-bot, non-m=all requests:

• 10 are false positives (like /src/pkg/ or ...testdata/src/pkg/ with m=text, and one from=index)
• 1 is https://golang.org/pkg/flag/?m=src
  (this would break if we removed m=src)

GOOS= usage

Downloaded 500 request URLs matching /GOOS/, spanning 53 hours (0.0026 QPS).
79 were from bots (user agent matching /bot|crawler/).
Of the 421 remaining, 110 were false positives (lower-case, or GOOS in search query),
and another 14 were attempts to exploit bad parameter handling (GOOS=/etc/passwd etc).
The remaining 297 break down as:

204 golang.org/pkg/syscall/?GOOS=windows
14 golang.org/pkg/syscall/js/?GOOS=js&GOARCH=wasm
13 golang.org/pkg/syscall/?GOOS=windows&GOARCH=amd64
11 golang.org/pkg/os/exec/?GOOS=windows
10 golang.org/pkg/os/?GOOS=windows
8 golang.org/pkg/syscall/?GOOS=linux&GOARCH=amd64
7 golang.org/pkg/syscall/?GOOS=darwin
5 golang.org/pkg/syscall/?GOARCH=amd64&GOOS=linux
5 golang.org/pkg/syscall/?GOARCH=amd64&GOOS=windows
4 golang.org/pkg/syscall/?GOOS=linux
3 golang.org/pkg/syscall/?GOOS=linux&GOARCH=mips64le
2 golang.org/pkg/syscall/js/?GOARCH=wasm&GOOS=js
2 golang.org/pkg/syscall/?GOOS=openbsd&GOARCH=386
2 golang.org/pkg/syscall/?GOOS=plan9
1 golang.org/pkg/net/?GOOS=js
1 golang.org/pkg/net/?GOOS=wasm
1 golang.org/pkg/net/?GOOS=windows
1 golang.org/pkg/net?GOOS=wasm
1 golang.org/pkg/syscall/?GOOS=nacl
1 golang.org/pkg/syscall?GOOS=windows
1 golang.org/x/syscall/?GOOS=windows

[tmthrgd]

How would pkg.go.dev work for access tip.golang.org docs? As far as I can tell there’s currently no way to do that.

Also I’m clearly very much in the minority here, but I happen to use ?m=all quite often and would be really sad to see it go. I find it very useful when working on my own code, and also when poking around the standard library to understand why something works or doesn’t, or how things are implemented.

[slrz]

How would pkg.go.dev work for access tip.golang.org docs? As far as I can tell there’s currently no way to do that.

I guess that's #44390.

[magical]

I would be very sorry to see golang.org/pkg disappear. It is my main reference for the package documentation, and I prefer its presentation to pkg.go.dev.

[gopherbot]

Change https://golang.org/cl/293493 mentions this issue: cmd/golangorg: redirect /cmd and /pkg to pkg.go.dev

[rsc]

@tmthrgd https://pkg.go.dev/math@master is working now to show what tip.golang.org does.

@magical I'd be interested to hear what you don't like about pkg.go.dev, but it would be off-topic for this issue. Maybe mail me at rsc@golang.org?

[rsc]

This hasn't happened yet because I realized that it would break visitors using golang.google.cn, because pkg.go.dev is not visible in China.

The new plan is to redirect by default but allow ?m=old to access the old docs. And requests from China will get ?m=old automatically.

tip.golang.org will probably not stick around.
Edit: What I meant by that is that tip.golang.org will probably just redirect to pkg.go.dev/math@master rather than stick around as a separate code base. The URLs will keep working.

[fzipp]

I find it somewhat odd that the new entry page for the standard library https://pkg.go.dev/std will show internal packages, the README and the cmd subdirectory (again full of internal packages).

[rsc]

@fzipp agreed, thanks. Will make sure it gets fixed.