Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/website: redirect golang.org/pkg and golang.org/cmd to pkg.go.dev #44356

Open
rsc opened this issue Feb 18, 2021 · 13 comments
Open

x/website: redirect golang.org/pkg and golang.org/cmd to pkg.go.dev #44356

rsc opened this issue Feb 18, 2021 · 13 comments
Assignees
Labels
Milestone

Comments

@rsc
Copy link
Contributor

@rsc rsc commented Feb 18, 2021

Thanks to significant work by @julieqiu and team on pkg.go.dev, godoc.org now redirects to pkg.go.dev, and we've now reduced the number of documentation servers run by the Go team from three to two. If pkg.go.dev is ready now to serve godoc.org's traffic, then it's also ready to serve golang.org's documentation traffic. We should do that, reducing the number of documentation servers from two to one (the ideal number!).

One concern we had was whether there are request parameters that pkg.go.dev does not support. I analyzed the logs, and the answer is no, there aren't. The most important ones are GOOS= and GOARCH=, and those work on pkg.go.dev for the standard library (other modules in progress).

The only other parameter with any frequency is ?m=all, which displays unexported API. We do not plan to add that feature to pkg.go.dev - it would expand the index significantly without creating much value, if any, for users. But most of the ?m=all requests are coming from search engine results that have inadvertently indexed that form of the page. Those would be served just as well (or better) by only showing exported API.

There is about one request per day using m=all to link to unexported functions, or about 0.0007% of traffic. That one request per day will break with the redirect, and that seems an acceptable cost for cutting our doc server count by 50%.

Details below.

Query parameters overall.

Downloaded 10,000 request URLs from App Engine logs matching "golang.org/pkg", spanning 96 minutes (1.7 QPS).
808 were from bots (user agent matching /bot|crawler/).

Only 39 of the 9,192 non-bot requests (0.42%) had query parameters.

21 used ?m=all to see unexported data; all were coming from www.google.com results
(which probably did not intend and are not well served by ?m=all).

9 fetched /pkg/syscall/?GOOS=windows.
Most hid the referer but one was from https://golang.org/doc/go1.16.
There were also
2 /pkg/syscall/?GOOS=darwin
2 /pkg/syscall/?GOOS=linux
1 /pkg/syscall/js/?GOOS=js&GOARCH=wasm (found via a search for [golang js.FuncOf])
1 /pkg/os/exec/?GOOS=windows

3 used ?source=post_page---------------------------, which does nothing.
They all came from duckduckgo.

m= parameter

Downloaded 500 request URLs matching "/pkg.*m=", spanning 41 hours (0.0034 QPS).
75 were from bots (user agent matching /bot|crawler/).
Of the 425 remaining, 414 used m=all, and 353 of those had Referer lines.
Of the 353 referers for m=all, there were:

Of the 11 non-bot, non-m=all requests:

  • 10 are false positives (like /src/pkg/ or ...testdata/src/pkg/ with m=text, and one from=index)
  • 1 is https://golang.org/pkg/flag/?m=src
    (this would break if we removed m=src)

GOOS= usage

Downloaded 500 request URLs matching /GOOS/, spanning 53 hours (0.0026 QPS).
79 were from bots (user agent matching /bot|crawler/).
Of the 421 remaining, 110 were false positives (lower-case, or GOOS in search query),
and another 14 were attempts to exploit bad parameter handling (GOOS=/etc/passwd etc).
The remaining 297 break down as:

204 golang.org/pkg/syscall/?GOOS=windows
14 golang.org/pkg/syscall/js/?GOOS=js&GOARCH=wasm
13 golang.org/pkg/syscall/?GOOS=windows&GOARCH=amd64
11 golang.org/pkg/os/exec/?GOOS=windows
10 golang.org/pkg/os/?GOOS=windows
8 golang.org/pkg/syscall/?GOOS=linux&GOARCH=amd64
7 golang.org/pkg/syscall/?GOOS=darwin
5 golang.org/pkg/syscall/?GOARCH=amd64&GOOS=linux
5 golang.org/pkg/syscall/?GOARCH=amd64&GOOS=windows
4 golang.org/pkg/syscall/?GOOS=linux
3 golang.org/pkg/syscall/?GOOS=linux&GOARCH=mips64le
2 golang.org/pkg/syscall/js/?GOARCH=wasm&GOOS=js
2 golang.org/pkg/syscall/?GOOS=openbsd&GOARCH=386
2 golang.org/pkg/syscall/?GOOS=plan9
1 golang.org/pkg/net/?GOOS=js
1 golang.org/pkg/net/?GOOS=wasm
1 golang.org/pkg/net/?GOOS=windows
1 golang.org/pkg/net?GOOS=wasm
1 golang.org/pkg/syscall/?GOOS=nacl
1 golang.org/pkg/syscall?GOOS=windows
1 golang.org/x/syscall/?GOOS=windows

@gopherbot gopherbot added this to the Unreleased milestone Feb 18, 2021
@rsc rsc self-assigned this Feb 18, 2021
@rsc rsc added the NeedsFix label Feb 18, 2021
@tmthrgd
Copy link
Contributor

@tmthrgd tmthrgd commented Feb 18, 2021

How would pkg.go.dev work for access tip.golang.org docs? As far as I can tell there’s currently no way to do that.

Also I’m clearly very much in the minority here, but I happen to use ?m=all quite often and would be really sad to see it go. I find it very useful when working on my own code, and also when poking around the standard library to understand why something works or doesn’t, or how things are implemented.

@slrz
Copy link

@slrz slrz commented Feb 18, 2021

How would pkg.go.dev work for access tip.golang.org docs? As far as I can tell there’s currently no way to do that.

I guess that's #44390.

@magical
Copy link
Contributor

@magical magical commented Feb 20, 2021

I would be very sorry to see golang.org/pkg disappear. It is my main reference for the package documentation, and I prefer its presentation to pkg.go.dev.

@gopherbot
Copy link

@gopherbot gopherbot commented Feb 23, 2021

Change https://golang.org/cl/293493 mentions this issue: cmd/golangorg: redirect /cmd and /pkg to pkg.go.dev

@rsc
Copy link
Contributor Author

@rsc rsc commented Feb 25, 2021

@tmthrgd https://pkg.go.dev/math@master is working now to show what tip.golang.org does.

@magical I'd be interested to hear what you don't like about pkg.go.dev, but it would be off-topic for this issue. Maybe mail me at rsc@golang.org?

@rsc
Copy link
Contributor Author

@rsc rsc commented Feb 25, 2021

This hasn't happened yet because I realized that it would break visitors using golang.google.cn, because pkg.go.dev is not visible in China.

The new plan is to redirect by default but allow ?m=old to access the old docs. And requests from China will get ?m=old automatically.

tip.golang.org will probably not stick around.
Edit: What I meant by that is that tip.golang.org will probably just redirect to pkg.go.dev/math@master rather than stick around as a separate code base. The URLs will keep working.

@fzipp
Copy link
Contributor

@fzipp fzipp commented Feb 25, 2021

I find it somewhat odd that the new entry page for the standard library https://pkg.go.dev/std will show internal packages, the README and the cmd subdirectory (again full of internal packages).

@rsc
Copy link
Contributor Author

@rsc rsc commented Feb 26, 2021

@fzipp agreed, thanks. Will make sure it gets fixed.

gopherbot pushed a commit to golang/pkgsite that referenced this issue Mar 1, 2021
The README is no longer displayed at pkg.go.dev/std.

It is also no longer stored in the database when a std module is
fetched.

For golang/go#44356

Change-Id: I6b1605389bc3c97694b6ec2b06e28006a38be999
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/296950
Trust: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Mar 1, 2021
Packages in cmd/ are now omitted from the directory listing on
pkg.go.dev/std, as in the case of golang.org/pkg.

For golang/go#44356

Change-Id: I160965d1b914a11863a815db3184d8b9823e8ae1
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/296951
Trust: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Mar 1, 2021

Change https://golang.org/cl/296951 mentions this issue: internal/frontend: omit cmd packages from /std

@gopherbot
Copy link

@gopherbot gopherbot commented Mar 1, 2021

Change https://golang.org/cl/296950 mentions this issue: internal: do not display README at /std

gopherbot pushed a commit to golang/pkgsite that referenced this issue Mar 1, 2021
The following is refactored:

* The UnitPage fields Subdirectories and NestedModules are removed,
  since they are not needed. UnitPage.Directories is used to determine
  if the directories section should be shown.

* The directories section now displays the internal directory at the
  bottom of the page.

For golang/go#44356

Change-Id: I3e6f7e3cacf27e129d6d7ea78d7a89e6e58e4be7
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/296953
Trust: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Mar 1, 2021
Internal sudirectories that are not at the top level of the unit are no
longer displayed in directories section to reduce noise in this section.

For golang/go#44356

Change-Id: Id4c141f215e4fdab61e57361e5d265594eb21bd6
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/297109
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Mar 1, 2021

Change https://golang.org/cl/296953 mentions this issue: content,internal/frontend: refactor directories section

@gopherbot
Copy link

@gopherbot gopherbot commented Mar 1, 2021

Change https://golang.org/cl/297109 mentions this issue: internal/frontend: skip nested internal directories

@gopherbot
Copy link

@gopherbot gopherbot commented Jun 15, 2021

Change https://golang.org/cl/327849 mentions this issue: internal/web: redirect golang.org/pkg/... to pkg.go.dev/...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants