cmd/go: `go env GOMOD` is too slow for use in interactive commands

[heschik]

We're porting a lot of tools to support modules, and most of them need to know whether modules are enabled to turn on the newer, slower, potentially buggier code path. Running go env GOMOD takes about 10ms, which is not negligible for an interactive command like godef or guru. The algorithm for determining if modules are on is just simple enough to tempt people to roll their own. In fact, there's already a second implementation in go/build:

go/src/go/build/build.go

Line 977 in 429bae7

switch os.Getenv("GO111MODULE") {

This is unfortunate, especially since the algorithm changed in http://golang.org/cl/148517. (Maybe not in a way that matters for go/build? Not sure.)

I think it'd be helpful to have a standalone internal package that implemented this functionality that could be used for go env, go/build, and copied into x/tools for other tools to use.

@bcmills @jayconrod @ianthehat

[bcmills]

The algorithm has changed before, and will likely change again (whenever the default behavior switches over). Probably it's better to make go env GOMOD fast than to duplicate it with version skew.

I'm a total novice with the Linux perf tool, but it seems to imply that we're spending a significant fraction of our latency initializing maps, regexps, and templates that aren't actually going to be used.

From

$ GOMAXPROCS=1 perf record --freq=max --call-graph fp -- go env GOMOD
$ perf report

I get the following:

-   76.18%     0.00%  go       go                  [.] runtime.main                                                                  ◆
   - runtime.main                                                                                                                    ▒
      - 71.40% main.init                                                                                                             ▒
         - 61.68% cmd/go/internal/bug.init                                                                                           ▒
            - cmd/go/internal/envcmd.init                                                                                            ▒
               - 48.18% cmd/go/internal/modload.init                                                                                 ▒
                  - 46.52% cmd/go/internal/modfetch.init                                                                             ▒
                     - 41.80% cmd/go/internal/get.init                                                                               ▒
                        - 23.85% cmd/go/internal/work.init                                                                           ▒
                           + 22.31% regexp.MustCompile                                                                               ▒
                           + 0.82% cmd/go/internal/work.init.0                                                                       ▒
                           + 0.72% debug/elf.init                                                                                    ▒
                        - 8.70% cmd/go/internal/get.init.1                                                                           ▒
                             regexp.MustCompile                                                                                      ▒
                           + regexp.compile                                                                                          ▒
                        - 7.08% cmd/go/internal/web.init                                                                             ▒
                           - 4.61% net/http.init                                                                                     ▒
                              - 1.91% internal/x/net/http/httpguts.init                                                              ▒
                                   internal/x/net/idna.init                                                                          ▒
                                 - internal/x/text/unicode/norm.init                                                                 ▒
                                      1.25% runtime.mapassign_fast32                                                                 ▒
                                      0.66% runtime.aeshash32                                                                        ▒
                              - 1.37% internal/x/net/http2/hpack.init                                                                ▒
                                   internal/x/net/http2/hpack.newStaticTable                                                         ▒
                                 + runtime.mapassign                                                                                 ▒
                                0.70% runtime.mapassign_fast64                                                                       ▒
                              + 0.63% mime/multipart.init                                                                            ▒
                           - 2.47% crypto/tls.init                                                                                   ▒
                              + 1.82% crypto/des.init                                                                                ▒
                              + 0.65% crypto/x509.init                                                                               ▒
                        + 0.81% regexp.MustCompile                                                                                   ▒
                        + 0.75% cmd/go/internal/get.init.0                                                                           ▒
                          0.61% crypto/tls.init                                                                                      ▒
                     + 2.73% cmd/go/internal/modfetch/codehost.init                                                                  ▒
                     + 1.15% archive/zip.init                                                                                        ▒
                     + 0.83% regexp.MustCompile                                                                                      ▒
                  + 0.84% cmd/go/internal/imports.init                                                                               ▒
                  + 0.83% regexp.MustCompile                                                                                         ▒
               - 12.96% cmd/go/internal/load.init                                                                                    ▒
                    text/template.(*Template).Parse                                                                                  ▒
                    text/template/parse.Parse                                                                                        ▒
                    text/template/parse.(*Tree).Parse                                                                                ▒
                  + text/template/parse.(*Tree).parse                                                                                ▒
         - 4.54% cmd/go/internal/base.init                                                                                           ▒
            - cmd/go/internal/cfg.init                                                                                               ▒
               - 3.83% go/build.init                                                                                                 ▒
                  - 1.96% go/doc.init                                                                                                ▒
                     + 1.38% regexp.MustCompile                                                                                      ▒
                  + 0.78% go/build.defaultContext                                                                                    ▒
         + 1.64% cmd/go/internal/test.init                                                                                           ▒
         + 1.58% flag.init                                                                                                           ▒
         + 0.83% cmd/go/internal/list.init                                                                                           ▒
         + 0.83% cmd/go/internal/clean.init                                                                                          ▒
      + 3.87% main.main                                                                                                              ▒

[heschik]

I agree that'd be better, and I know Brad was looking at lazy-loading this kind of stuff, but in general keeping init functions out of something as big as the main go command seems pretty difficult. Maybe a new command in $GOROOT/bin, goenv or something?

[mark-rushakoff]

The lazy-loading issue was #26775.

[bcmills]

Let's see how far we can get making go env faster. I'd really rather not introduce a whole binary just to shave off (optimistically) ≤8ms, especially given that the result of go env GOMOD for a given working directory should be pretty amenable to caching for long-running tools like editors.

[mvdan]

I actually was thinking the same when I saw go env executions like go env GOARCH show up in the trace reports from a tool I've written that uses go/packages. I agree that shaving off a few milliseconds from go env should be a good start.

I agree with the sentiment that milliseconds matter for some tools, but given that go/packages is built around executing the go tool, I don't think it's a good idea to break that rule just for performance. Perhaps we could instead reduce the amount of times the go tool is executed as part of those interactive tools.

[gopherbot]

Change https://golang.org/cl/155540 mentions this issue: cmd/go: avoid compiling many regexes at init

[mvdan]

I had a stab at this and shaved a millisecond off on my machine (20%). The bigger offenders given by perf report are now a bit more tricky, like 5.20% net/http.init and 3.76% crypto/tls.init. A few percent still belong to regexp, but I've left them for later.

I wonder if packages like http and tls could reduce their init cost even further. If not, perhaps it does make sense to split go env into a much smaller binary with a tiny init function.

I'm also a bit puzzled by how time go env GOARCH on Bash seems to give numbers around 10ms, while the benchmark gives numbers around 4ms. I'm not sure what could explain the almost tripled run-time.

[mark-rushakoff]

time go env GOARCH on Bash seems to give numbers around 10ms, while the benchmark gives numbers around 4ms.

echo 'package main; func main(){}' > c.go
go build ./c.go
time ./c

time reports 5-8 milliseconds to run a no-op go program on my machine, so it's probably overhead from loading/starting the process and whatever go runtime initialization is involved. time /usr/bin/true (an actual 17k compiled executable) takes 3-6 milliseconds on my machine so the time for the no-op go program seems reasonable to me.

[mvdan]

Interesting - but the Go benchmark is also executing the go binary at each iteration. Perhaps it's faster since it's loading the same binary over and over again.

If anyone knows of a better way to write a benchmark for this, please let me know. Benchmarking a function directly, like the generated init function for the entire cmd/go main package, would be great - as we'd be able to get cpu and memory profiles directly from the benchmark. But I don't think that is possible at the moment.

[gopherbot]

Change https://golang.org/cl/155961 mentions this issue: cmd/go: add benchmark that execs 'go env GOARCH'

[gopherbot]

Change https://golang.org/cl/155962 mentions this issue: cmd/go: delay parsing the testmain template

[gopherbot]

Change https://golang.org/cl/155963 mentions this issue: cmd/go: delay compiling some more regexes

[mvdan]

I've found a few more ways to remove work from cmd/go, and now go env GOARCH went from ~5ms to ~2.3ms on the added benchmark. Similarly, time go env GOARCH has gone from ~15-20ms to ~10-12ms on my laptop, though those numbers jump around much more than the benchmark ones.

I think we should be able to fairly easily shave off some more from cmd/go's init for 1.13. On perf report, regexp.MustCompile still owns 7%, and go/build.init is an astonishing 10%, so I think we could realistically shave off another 20% or even more.

I'll stop with the CLs for now, to get some input on the benchmark and style of the CLs. Happy to work on more once these have been approved.

[bcmills]

Retitling to reflect the current approach.

[gopherbot]

Change https://golang.org/cl/164040 mentions this issue: internal/lazyregexp: add a lazy Regexp package

[gopherbot]

Change https://golang.org/cl/164337 mentions this issue: internal/lazytemplate: add a lazy template wrapper

[mvdan]

All previous changes merged; go env should now be about twice as fast.

A perf report now shows these remaining offenders:

• errors.New at 21%, most of which is runtime.Callers at 20%. Why do global errors need stack traces?
• archive/zip at 3%, which we might be able to improve slightly
• runtime.mapassign_faststr at 7%, so there's quite a bit of map filling work happening somewhere
• runtime.newobject at 6%, so there probably are quite a lot of global object allocs around

I think the only big low-hanging fruit is errors.New. There should definitely be a way to create global errors without incurring extra init cost beyond allocating the error. I presume the 1.13 error changes, which include stack frames, are partially to blame for this spike in init cost.

What should the solution be? Not to collect stack frames if errors.New is running as part of init? Using static error types, which would be a large refactor? Making runtime.Callers a no-op while init is running? /cc @jba @mpvl

[bcmills]

runtime.Caller, at least, can theoretically be evaluated (and inlined) at link time.

I filed #30468 for the general regression. In the meantime perhaps we should substitute an internal package for errors: it might be as simple as sed -i 's/ "errors"/ "cmd/go/internal/errors"/' $(find cmd/go -name '*.go'.

[mvdan]

Thanks! I don't think this is terribly urgent; we won ~50% and lost ~20%, so I'd wait another week or two to see if #30468 can have a good solution.

[gopherbot]

Change https://golang.org/cl/166459 mentions this issue: cmd/go: further reduce init work

[mvdan]

I've had a look around and managed to scrape another 3%. I don't think we're going to get much more without larger refactors, such as making cmd/go's init work be lazier, or having the compiler generate better code for static global maps.

If anyone is better at perf report than I am and manages to spot more low-hanging fruit, please do speak up :)

I'd say we should focus on #30468 before continuing; right now errors.New is just taking the top spot in too many generated package init functions.

[gopherbot]

Change https://golang.org/cl/167401 mentions this issue: errors: improve performance of New

[gopherbot]

Change https://golang.org/cl/167400 mentions this issue: errors: record only single frame

[rsc]

I'm seeing 20ms for 'go env GOMOD'. Is that really too slow?

[mvdan]

It's gotten ~50% faster since the issue was opened :) At least as far as the benchmark tells us.

[mvdan]

Now that the CLs to make errors.New faster have gone in, perf report shows errors.New at ~10%, and runtime.Callers at 8%, down from the previous 21% and 20%. #30820 would presumably let us get rid of that entire chunk.

[bcmills]

I'm seeing 11ms at this point. That's well below typical keypress latency for a USB keyboard,¹ and can generally be cached for a given working directory anyway.

That seems good enough for interactive use.

¹https://danluu.com/keyboard-latency/