Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/text/unicode/norm: reduce init-time memory usage #26752

Open
bradfitz opened this issue Aug 1, 2018 · 4 comments

Comments

Projects
None yet
4 participants
@bradfitz
Copy link
Member

commented Aug 1, 2018

The golang.org/x/text/unicode/norm package allocs 21.47kB in its init function, regardless of whether the package is ever used.

We should do it lazily, as done in e.g. https://go-review.googlesource.com/c/net/+/127275 and https://go-review.googlesource.com/c/go/+/127075

This would help people who explicitly import it (or import it via golang.org/x/net/http2), as then the std-vendored copy would not need to also allocate the 20KB.

Perhaps some of those tables should be const strings, too, so the linker can put it in read-only sections. Then we can slice into it at runtime to get integers out.

/cc @mpvl

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Aug 1, 2018

CC @mpvl

@kevinburke

This comment has been minimized.

Copy link
Contributor

commented Aug 4, 2018

Going to take a crack at this!

@gopherbot

This comment has been minimized.

Copy link

commented Aug 4, 2018

Change https://golang.org/cl/127924 mentions this issue: unicode/norm: lazy load decomps table

@gopherbot

This comment has been minimized.

Copy link

commented Aug 5, 2018

Change https://golang.org/cl/127926 mentions this issue: make recompMap a string

gopherbot pushed a commit to golang/text that referenced this issue Aug 7, 2018

unicode/norm: lazy load recompMap from string
recompMap gets populated on init, which requires 21kB of memory.
Instead populate it only when we need it.

This will put it in the rodata section of the binary, which means that
the Go compiler can apply better optimizations to it. The string is an
alternating series of 4 byte keys and 4 byte values, which get built
into a map the first time they are required.

Updates golang/go#26752.

Change-Id: I4f9b7d74af6df42b58999cf42cf5ada9646aa98d
Reviewed-on: https://go-review.googlesource.com/127926
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.