Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/go/packages: Load is 2-3 orders of magnitude slower than go/build.Import #31087

Open
thockin opened this issue Mar 27, 2019 · 6 comments

Comments

@thockin
Copy link

commented Mar 27, 2019

As per @heschik in #29452

Converting a tool from go/build to go/packages incurs a huge slowdown. I tried to min-repro it here. Note that we have a full vendor/ dir, no un-vendored deps, and modules are NOT turned on yet.

What version of Go are you using (go version)?

1.12.1

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

linux, amd64

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/usr/local/google/home/thockin/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/usr/local/google/home/thockin/src/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/google-golang"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/google-golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build445939239=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Build and run the two programs below and time them.

What did you expect to see?

Roughly the same timing

What did you see instead?

0.008s vs 1.272s

go-build.go:

package main

import (
        "fmt"
        "go/build"
        "log"
        "os"

        "github.com/davecgh/go-spew/spew"
)

func main() {
        if err := visit("."); err != nil {
                fmt.Println(err)
                os.Exit(1)
        }
}

func visit(pkgName string) error {
        pkg, err := findPackage(pkgName)
        if err != nil {
                return err
        }
        spew := spew.ConfigState{DisableMethods: true, Indent: "  "}
        spew.Dump(pkg)
        return nil
}

func findPackage(pkgName string) (*build.Package, error) {
        log.Println("find", pkgName)
        pkg, err := build.Import(pkgName, getwd(), build.ImportComment)
        if err != nil {
                return nil, err
        }
        return pkg, nil
}

func getwd() string {
        pwd, err := os.Getwd()
        if err != nil {
                panic(fmt.Sprintf("can't get working directory: %v", err))
        }
        return pwd
}

go-packages.go:

package main

import (
	"fmt"
	"log"
	"os"

	"github.com/davecgh/go-spew/spew"

	"golang.org/x/tools/go/packages"
)

func main() {
	if err := visit("."); err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
}

func visit(pkgName string) error {
	pkgs, err := findPackage(pkgName)
	if err != nil {
		return err
	}
	spew := spew.ConfigState{DisableMethods: true, Indent: "  "}
	spew.Dump(pkgs[0])
	return nil
}

func findPackage(pkgName string) ([]*packages.Package, error) {
	log.Println("find", pkgName)
	pkgs, err := packages.Load(&packages.Config{Mode: packages.LoadFiles}, pkgName)
	if err != nil {
		return nil, err
	}
	return pkgs, nil
}

Testing:

$ ls
go-build.go  go-packages.go

$ go build ./go-build.go 

$ go build ./go-packages.go 

$ time ./go-build >/dev/null
2019/03/27 11:51:10 find .

real	0m0.008s
user	0m0.008s
sys	0m0.000s

$ time ./go-packages >/dev/null
2019/03/27 11:51:17 find .

real	0m1.272s
user	0m0.094s
sys	0m0.109s

@thockin thockin changed the title go/packes is 50x slower than go/build go/packages is 50x slower than go/build Mar 27, 2019

@thockin thockin changed the title go/packages is 50x slower than go/build go/packages is 2-3 orders of magnitude slower than go/build Mar 27, 2019

@matloob

This comment has been minimized.

Copy link
Contributor

commented Mar 27, 2019

This was discussed offline: The slowness was due to the Google-specific gopackagesdriver binary.

Unfortunately go/packages will always be slower than go/build because we need to execute go list as the true authority of package information, but this kind of slowness is unexpected and really bad. We'll get a google-specific fix in soon, but for those of you facing this problem, you can set GOPACKAGESDRIVER=off in the short term to bypass that driver.

@thockin

This comment has been minimized.

Copy link
Author

commented Mar 27, 2019

Thanks. I am still concerned at "go/packages will always be slower than go/build" even when we have vendored EVERYTHING. I will hang tight for the followup here, and try a proper rewrite of my real tool against go/packages (rather than a trivial adaptation as I did)

@heschik

This comment has been minimized.

Copy link
Contributor

commented Mar 27, 2019

re: always slower, it's true but I doubt it'll be a big problem for Kubernetes' build tools.

There's some amount of overhead involved in running go list, then marshalling and unmarshalling the results. In my experience that's usually small, say 50-100ms or so. That can be a problem for highly interactive tools like goimports and godef, but for a build tool like go2make I think it'd be okay.

After that the major issue is that packages.Load starts from scratch every time, so it's important to keep the number of calls low to avoid loading the same packages (e.g. the stdlib) over and over and over. That can be really easy, or require major refactorings. But for most people I don't think it's been a big problem.

@thockin

This comment has been minimized.

Copy link
Author

commented Mar 28, 2019

@josharian

This comment has been minimized.

Copy link
Contributor

commented Mar 28, 2019

Note that issues like #30826 and #30828 make it harder to avoid calling packages.Load multiple times. go-fuzz-build calls packages.Load three times, and properly dealing with the referenced issues (which I am currently hacking around in a way that will work until it doesn't) will require a fourth and possibly a fifth call.

@bcmills bcmills added this to the Unreleased milestone Apr 12, 2019

@bcmills bcmills changed the title go/packages is 2-3 orders of magnitude slower than go/build go/packages: Load is 2-3 orders of magnitude slower than go/build.Import Apr 15, 2019

@gopherbot

This comment has been minimized.

Copy link

commented Apr 16, 2019

Change https://golang.org/cl/170863 mentions this issue: cmd/gopherbot: CC triaged issues to owners

@dmitshur dmitshur changed the title go/packages: Load is 2-3 orders of magnitude slower than go/build.Import x/tools/go/packages: Load is 2-3 orders of magnitude slower than go/build.Import Apr 16, 2019

@gopherbot gopherbot added the Tools label Sep 12, 2019

@liggitt liggitt referenced this issue Sep 13, 2019
2 of 8 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.