x/pkgsite: slow to download std when running locally

[adonovan]

Steps to reproduce:

• $ go run golang.org/x/pkgsite/cmd/pkgsite@master &
• $ open http://localhost:8080
• Enter "fmt", hit Return, see search results page.
• Click first link for package fmt
• Wait 17s for download of std@latest.
• See documentation for fmt.

Ideally we would eliminate step 5. Is it necessary to select std@latest when there's a recent version of std already already installed?

Perhaps std@installedversion could also be offered on the "Or browse local modules:" menu, since just as it is desirable for the docs to reflect the local edits to modules, so too it is desirable to show the version of Go that is installed locally.

Loosely related:

• x/pkgsite: large module pages are very slow to render #51564
• x/pkgsite: slower loads than godoc.org #38892

[seankhliao]

There's the -gorepo flag for use with a checkout of the go repo
Though i still don't know how to get pkgsite to show "contents of my current CL" for std

related #57742

[adonovan]

There's the -gorepo flag for use with a checkout of the go repo

Thanks. I tried it twice, once with -gorepo=$(dirname $(dirname $(go list -f {{.Dir}} fmt))) to be consistent with the installed go command on the $PATH, and once with -goroot=$HOME/w/goroot which is a git client. In neither case did it reduce the latency of loading std. I haven't dug in to investigate.

[thepudds]

Hi @qiulaidongfeng, I think you originally reported this slow download issue in #59056.

Does that still happen for you in Go 1.21?

I just tried to reproduce this in Go 1.21 with a fresh installation of cmd/pkgsite, and it seemed very fast to go to the fmt std library page, for example, as well as to see the list of std packages in http://localhost:8080/std.

(The reason I ended up here was I had just re-read the comment on cmd/pkgsite documentation that says "Although standard library packages will work by default, the docs can take a while to appear the first time because the Go repo must be cloned and processed", and I was thinking about filing an issue asking if that git clone could be avoided based on the new Go 1.21 features around downloading and caching a copy of the Go toolchain as a module zip file... but then it seemed fast when I tried it just now, so maybe it is already faster than it was when first reported... or maybe I just didn't reproduce correctly, including I am not on an international link or maybe I wasn't fast enough with my checks, and hence the question to you. I haven't dug very deep here).

[qiulaidongfeng]

Does that still happen for you in Go 1.21?

go version -m D:\file\gofile\bin\pkgsite.exe
D:\file\gofile\bin\pkgsite.exe: devel go1.22-b2dbfbfc Fri Jan 12 00:56:20 2024 +0000
        path    golang.org/x/pkgsite/cmd/pkgsite
        mod     golang.org/x/pkgsite    v0.0.0-20240118234023-24c36df908d0      h1:VmjPAeY68SV22BUg9DTWAj20M65yXvWjbyqG5PZWTpo=
        dep     github.com/google/licensecheck  v0.3.1  h1:QoxgoDkaeC4nFrtGN1jV7IPmDCHFNIVh54e5hSt6sPs=
        dep     github.com/google/safehtml      v0.0.3-0.20211026203422-d6f0e11a5516    h1:pSEdbeokt55L2hwtWo6A2k7u5SG08rmw0LhWEyrdWgk=
        dep     golang.org/x/mod        v0.14.0 h1:dGoOF9QVLYng8IHTm7BAyWqCqSheQ5pYWGhzW00YJr0=
        dep     golang.org/x/net        v0.20.0 h1:aCL9BSgETF1k+blQaYUBx9hJ9LOGP3gAVemcZlf1Kpo=
        dep     golang.org/x/sync       v0.6.0  h1:5BMeUDZ7vkXGfEr1x9B4bRcTH4lpkTkpdh0T/J+qjbQ=
        dep     golang.org/x/text       v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
        dep     golang.org/x/tools      v0.17.0 h1:FvmRgNOcs3kOa+T20R1uhfP9F6HgG2mfxDv1vrx1Htc=
        dep     rsc.io/markdown v0.0.0-20231214224604-88bb533a6020      h1:GqQcl3Kno/rOntek8/d8axYjau8r/c1zVFojXS6WJFI=
        build   -buildmode=exe
        build   -compiler=gc
        build   DefaultGODEBUG=httplaxcontentlength=1,httpmuxgo121=1,panicnil=1,tls10server=1,tlsrsakex=1,tlsunsafeekm=1
        build   CGO_ENABLED=1
        build   CGO_CFLAGS=
        build   CGO_CPPFLAGS=
        build   CGO_CXXFLAGS=
        build   CGO_LDFLAGS=
        build   GOARCH=amd64
        build   GOOS=windows
        build   GOAMD64=v3

I try in today , again look slow to download std when running locally.

I look

2024/01/20 08:39:06 Info: go/packages.Load(["all"]) loaded 463 packages from . in 1.5965688s
2024/01/20 08:39:06 Info: go/packages.Load(std) loaded 0 packages from C:\Users\26454\.go\current in 22.2076ms
2024/01/20 08:39:06 Error: loading packages from stdlib: err: fork/exec C:\Users\26454\.go\current\bin\go.exe: The parameter is incorrect.: stderr:
2024/01/20 08:39:06 Info: FetchDataSource: fetching std@latest
2024/01/20 08:39:06 Info: FetchDataSource: fetching golang.org/x/tools/gopls@v0.0.0
2024/01/20 08:39:06 Info: Listening on addr http://localhost:8080
2024/01/20 08:39:06 Info: FetchDataSource: fetched golang.org/x/tools/gopls@v0.0.0 using *fetch.goPackagesModuleGetter in 364.3211ms with error <nil>
2024/01/20 08:39:17 Warning: fetching url from deps.dev: Get "https://deps.dev/_/s/go/p/golang.org%2Fx%2Ftools%2Fgopls/v/v0.0.0/exists": context deadline exceeded
2024/01/20 08:39:20 Warning: fetching url from deps.dev: Get "https://deps.dev/_/s/go/p/golang.org%2Fx%2Ftools%2Fgopls/v/v0.0.0/exists": context deadline exceeded
2024/01/20 08:39:32 Info: FetchDataSource: fetching std@latest
2024/01/20 08:39:55 Info: FetchDataSource: fetched std@latest using *fetch.stdlibZipModuleGetter in 49.4576596s with error <nil>
2024/01/20 08:40:26 Info: FetchDataSource: fetched std@latest using *fetch.stdlibZipModuleGetter in 54.0362934s with error <nil>
2024/01/20 08:40:26 Warning: fetching url from deps.dev: Get "https://deps.dev/_/s/go/p/std/v/v1.21.6/exists": context deadline exceeded

Then I ran pkgsite in the src subdirectory of a git clone go directory, tried to get the std documentation, got no response for a long time, and finally saw 500 Internal Server Error and

2024/01/20 08:46:44 Info: FetchDataSource: fetched std@v0.0.0 using *fetch.stdlibZipModuleGetter in 2m10.086188s with error FetchModule("std", "v0.0.0"): stdlib.ZipInfo("v0.0.0"): semanticVersion("v0.0.0"): invalid argument: requested version unknown: "v0.0.0"
2024/01/20 08:47:38 Info: FetchDataSource: fetched std@latest using *fetch.stdlibZipModuleGetter in 2m37.7875152s with error FetchModule("std", "latest"): stdlib.ContentDir("latest"): remoteGoRepo.clone("v1.21.6"): context deadline exceeded
2024/01/20 08:47:38 Error: serveUnitPage(ctx, w, r, ds, &{std std latest}): FetchDataSource.GetUnitMeta("std", "std", "latest"): FetchDataSource.findModule("std", "std", "latest"): FetchDataSource.getModule("std", "latest"): FetchModule("std", "latest"): stdlib.ContentDir("latest"): remoteGoRepo.clone("v1.21.6"): context deadline exceeded

Note: I used the 5G network speed and I saw that during pkgsite operation, there were 1 second speeds of 4-5M, and the average was estimated to be about 1M when in use, it seems that this problem has not been solved. pkgsite downloads something too big causing slow.

See #64920 and #61976 , If my use godoc, my experience is don't have to use the Internet.