Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/go/gcexportdata: reads entire object file, not just the needed type information #48954

Closed
dominikh opened this issue Oct 13, 2021 · 1 comment

Comments

@dominikh
Copy link
Member

@dominikh dominikh commented Oct 13, 2021

When using gcimportdata.NewReader and gcimportdata.Read, the entire object file will be read into memory, not just the portion containing type information. For packages that have large variable initializers, or that use embed, not doing this could save almost 100% of memory usage and I/O.

https://play.golang.org/p/kAXdktA3WTD contains an example that loads the archive for https://play.golang.org/p/MvxM7Jlxjhk – a package that embeds a 100 MB file. It demonstrates how much data gets read unnecessarily.

This should be fairly straightforward to fix. The necessary length information exists, gcimporter.FindExportData just doesn't expose it. If it did, we could use an io.LimitedReader.

A quickly hacked together patch drops the read amount from 100 MB to 100 bytes.

I'd be happy to send a change, but I'm not sure how the development between x/tools's gcimporter and the Go compiler's gcimporter get synchronized.

/cc @mdempsky because I'm curious how this interacts with his lazy loading work.

@seankhliao seankhliao changed the title golang.org/x/tools/go/gcexportdata: reads entire object file, not just the needed type information x/tools/go/gcexportdata: reads entire object file, not just the needed type information Oct 14, 2021
@gopherbot gopherbot added this to the Unreleased milestone Oct 14, 2021
@gopherbot
Copy link

@gopherbot gopherbot commented Oct 20, 2021

Change https://golang.org/cl/357209 mentions this issue: go/gcexportdata: limit reader to the export data section of the archive

Loading

@gopherbot gopherbot closed this in baf4e38 Oct 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants