Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/go/packages: Load function throws "argument list too long" error #36909

Open
awelc opened this issue Jan 30, 2020 · 2 comments
Open

x/tools/go/packages: Load function throws "argument list too long" error #36909

awelc opened this issue Jan 30, 2020 · 2 comments

Comments

@awelc
Copy link

@awelc awelc commented Jan 30, 2020

What version of Go are you using (go version)?

$ go version
go version go1.13.4 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"

What is the problem?

I am trying to analyze a large application using Go's static analysis support. I am using the "packages" package to load all application packages using the Load function. Unfortunately the application is really big and the list of patterns that the Load function takes the second parameter is really long. This causes a problem on Mac OS as in this case it leads to the "argument list too long" error popping up after the Load function is called. The reason for it is that internally, the Load function executes "go list" command passing it the list of patterns as arguments, which leads to exceeding the limit established by Mac OS (which, as far as I can tell, cannot be changed).

The usual workaround for this problem is to split the argument list and apply a given function incrementally to each list fragment - in this case it would mean calling the Load function multiple times with a different subset of the patterns list at each invocation. This, however, does not quite work, as it messes up the SSA representation that I need to generate afterwards using AllPackages function from the "ssautil" package. The AllPackages function constructs a program that contains a "file set" containing all files constituting the program. As far as I can tell, the only way to access this file set during analysis is via the program itself. And this is the gist of the problem, incremental application of the Load function creates a "file set" per increment, and the resulting program during SSA construction will assign only one to the program, making the remaining ones inaccessible during analysis.

What would be great is to have an API-level solution that would allow incremental loading of packages.

@cagedmantis
Copy link
Contributor

@cagedmantis cagedmantis commented Feb 7, 2020

/cc @matloob

@matloob
Copy link
Contributor

@matloob matloob commented Feb 13, 2020

I think the solution to this is to have go/packages split the arguments to go list (or to the underlying GOPACKAGESDRIVER if it's set) and make multiple calls to the driver. those results should be able to just be merged (removing duplicates if, for example, -deps is set as a go list argument). The pcakage structures returned by the driver are independent, and the driver should return consistent ids over several calls. Then those can be supplied to the "refining" stage where we parse/typecheck the packages.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants