Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/text/unicode/norm: reduce init-time memory usage #26752

Open
bradfitz opened this issue Aug 1, 2018 · 7 comments
Open

x/text/unicode/norm: reduce init-time memory usage #26752

bradfitz opened this issue Aug 1, 2018 · 7 comments

Comments

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Aug 1, 2018

The golang.org/x/text/unicode/norm package allocs 21.47kB in its init function, regardless of whether the package is ever used.

We should do it lazily, as done in e.g. https://go-review.googlesource.com/c/net/+/127275 and https://go-review.googlesource.com/c/go/+/127075

This would help people who explicitly import it (or import it via golang.org/x/net/http2), as then the std-vendored copy would not need to also allocate the 20KB.

Perhaps some of those tables should be const strings, too, so the linker can put it in read-only sections. Then we can slice into it at runtime to get integers out.

/cc @mpvl

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Aug 1, 2018

CC @mpvl

@kevinburke
Copy link
Contributor

@kevinburke kevinburke commented Aug 4, 2018

Going to take a crack at this!

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 4, 2018

Change https://golang.org/cl/127924 mentions this issue: unicode/norm: lazy load decomps table

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 5, 2018

Change https://golang.org/cl/127926 mentions this issue: make recompMap a string

gopherbot pushed a commit to golang/text that referenced this issue Aug 7, 2018
recompMap gets populated on init, which requires 21kB of memory.
Instead populate it only when we need it.

This will put it in the rodata section of the binary, which means that
the Go compiler can apply better optimizations to it. The string is an
alternating series of 4 byte keys and 4 byte values, which get built
into a map the first time they are required.

Updates golang/go#26752.

Change-Id: I4f9b7d74af6df42b58999cf42cf5ada9646aa98d
Reviewed-on: https://go-review.googlesource.com/127926
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@agnivade
Copy link
Contributor

@agnivade agnivade commented Nov 12, 2019

Is there a way to generate the tables for all unicode versions ? If I run "go generate", it only generates the tables for the current Go version.

Also, how are you checking the memory allocation of the init function ? If I just do a blank import and run a mem profile, nothing gets caught.

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Nov 12, 2019

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 13, 2019

Change https://golang.org/cl/207017 mentions this issue: unicode/norm: make decomps to a const string

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.