x/tools/go/packages: memory hungry in parsing the github.com/go-fonts/latin-modern module packages

[zigo101]

What version of Go are you using (go version)?

$ go version
go version go1.20rc3 linux/amd64

The version of the golang.org/x/tools module is v0.5.0.

Does this issue reproduce with the latest release?

Yes

What did you do?

main.go:

package main

import (
	"log"
	"runtime"
	"runtime/metrics"

	"golang.org/x/tools/go/packages"

	_ "github.com/go-fonts/latin-modern/lmmath"
)

func printUsedMemory(prefix string) {
	runtime.GC()

	samples := []metrics.Sample{
		{Name: "/memory/classes/total:bytes"},
		{Name: "/memory/classes/heap/free:bytes"},
		{Name: "/memory/classes/heap/objects:bytes"},
		{Name: "/gc/heap/goal:bytes"},
	}
	metrics.Read(samples)
	
	log.Println("=========", prefix)
	for _, s := range samples {
		log.Printf("%44s %dMiB", s.Name, s.Value.Uint64()>>20)
	}
}

var allPkgs []*packages.Package

func main() {
	log.SetFlags(0)

	var configForParsing = &packages.Config{
		Mode: packages.NeedName | packages.NeedImports | packages.NeedDeps |
			packages.NeedTypes | packages.NeedExportsFile | packages.NeedFiles |
			packages.NeedCompiledGoFiles | packages.NeedTypesSizes |
			packages.NeedSyntax | packages.NeedTypesInfo,
		Tests: false,
	}

	printUsedMemory("before parsing:")

	pkgs, err := packages.Load(configForParsing, "github.com/go-fonts/latin-modern/...")
	if err != nil {
		log.Println("packages.Load (parse packages): %w", err)
		return
	}

	printUsedMemory("after parsing:")

	allPkgs = pkgs
}

go,mod:

module a.b/c

go 1.20

require golang.org/x/tools v0.5.0

require github.com/go-fonts/latin-modern v0.2.0

require (
	golang.org/x/mod v0.7.0 // indirect
	golang.org/x/sys v0.4.0 // indirect
)

Output:

========= before parsing:
                 /memory/classes/total:bytes 11MiB
             /memory/classes/heap/free:bytes 0MiB
          /memory/classes/heap/objects:bytes 0MiB
                         /gc/heap/goal:bytes 4MiB
========= after parsing:
                 /memory/classes/total:bytes 3079MiB
             /memory/classes/heap/free:bytes 1290MiB
          /memory/classes/heap/objects:bytes 1669MiB
                         /gc/heap/goal:bytes 3338MiB

What did you expect to see?

About several hundred of memory consumed.

What did you see instead?

About 3G memory consumed.

Each of the font source files only contains a var TTF = []byte{...} variable.
It is unexpected to use so much memory.

When parsing the packages in the github.com/tdewolff/canvas module,
which depends on the github.com/go-fonts/latin-modern module,
my computer (8G memory) hangs for virtual memory are swapped in and out frequently.

The github.com/tdewolff/canvas module is a relative small module.
My computer is able to parse much larger modules.

[mknyszek]

CC @golang/tools-team

[findleyr]

CC @griesemer @adonovan

This is almost certainly caused by recording information in types.Info for each element of those byte literals. We could potentially optimize go/types for such cases.

[findleyr]

Tentatively marking this for Go 1.21, to see if we can improve memory consumption in go/types in these cases.