Repository navigation
x/tools/go/packages: memory hungry in parsing the github.com/go-fonts/latin-modern module packages #57985
Description
What version of Go are you using (go version)?
$ go version go version go1.20rc3 linux/amd64
The version of the golang.org/x/tools module is v0.5.0.
Does this issue reproduce with the latest release?
Yes
What did you do?
main.go:
package main
import (
"log"
"runtime"
"runtime/metrics"
"golang.org/x/tools/go/packages"
_ "github.com/go-fonts/latin-modern/lmmath"
)
func printUsedMemory(prefix string) {
runtime.GC()
samples := []metrics.Sample{
{Name: "/memory/classes/total:bytes"},
{Name: "/memory/classes/heap/free:bytes"},
{Name: "/memory/classes/heap/objects:bytes"},
{Name: "/gc/heap/goal:bytes"},
}
metrics.Read(samples)
log.Println("=========", prefix)
for _, s := range samples {
log.Printf("%44s %dMiB", s.Name, s.Value.Uint64()>>20)
}
}
var allPkgs []*packages.Package
func main() {
log.SetFlags(0)
var configForParsing = &packages.Config{
Mode: packages.NeedName | packages.NeedImports | packages.NeedDeps |
packages.NeedTypes | packages.NeedExportsFile | packages.NeedFiles |
packages.NeedCompiledGoFiles | packages.NeedTypesSizes |
packages.NeedSyntax | packages.NeedTypesInfo,
Tests: false,
}
printUsedMemory("before parsing:")
pkgs, err := packages.Load(configForParsing, "github.com/go-fonts/latin-modern/...")
if err != nil {
log.Println("packages.Load (parse packages): %w", err)
return
}
printUsedMemory("after parsing:")
allPkgs = pkgs
}go,mod:
module a.b/c
go 1.20
require golang.org/x/tools v0.5.0
require github.com/go-fonts/latin-modern v0.2.0
require (
golang.org/x/mod v0.7.0 // indirect
golang.org/x/sys v0.4.0 // indirect
)
Output:
========= before parsing:
/memory/classes/total:bytes 11MiB
/memory/classes/heap/free:bytes 0MiB
/memory/classes/heap/objects:bytes 0MiB
/gc/heap/goal:bytes 4MiB
========= after parsing:
/memory/classes/total:bytes 3079MiB
/memory/classes/heap/free:bytes 1290MiB
/memory/classes/heap/objects:bytes 1669MiB
/gc/heap/goal:bytes 3338MiB
What did you expect to see?
About several hundred of memory consumed.
What did you see instead?
About 3G memory consumed.
Each of the font source files only contains a var TTF = []byte{...} variable.
It is unexpected to use so much memory.
When parsing the packages in the github.com/tdewolff/canvas module,
which depends on the github.com/go-fonts/latin-modern module,
my computer (8G memory) hangs for virtual memory are swapped in and out frequently.
The github.com/tdewolff/canvas module is a relative small module.
My computer is able to parse much larger modules.
Activity
mknyszek commentedon Jan 30, 2023
CC @golang/tools-team
findleyr commentedon Jan 30, 2023
CC @griesemer @adonovan
This is almost certainly caused by recording information in types.Info for each element of those byte literals. We could potentially optimize go/types for such cases.
findleyr commentedon Jan 31, 2023
Tentatively marking this for Go 1.21, to see if we can improve memory consumption in go/types in these cases.