/go

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

all: reduce init-time CPU & memory usage #26775

Open
bradfitz opened this Issue Aug 2, 2018 · 12 comments

Comments

Assignees
No one assigned
Labels
Projects
None yet
Milestone
  Unplanned
5 participants
@bradfitz @gopherbot @ysmolsky @josharian @mark-rushakoff
@bradfitz
Member

bradfitz commented Aug 2, 2018 •
edited

Tracking bug to reduce init-time CPU & memory usage.

Previously:

https://go-review.googlesource.com/c/go/+/127075 - html: lazily populate Unescape tables
https://go-review.googlesource.com/c/net/+/127275 - http2/hpack: lazily build huffman table on first use

Open:

#26752 - x/text/unicode/norm: reduce init-time memory usage

The program https://play.golang.org/p/9ervXCWzV_z is useful to find offenders:

$ go build all.go
$ GODEBUG=memprofilerate=1  ./all
346768
$ go tool pprof /tmp/all.mem.prof 
(pprof) top 50
Showing nodes accounting for 286.24kB, 88.87% of 322.09kB total
Dropped 228 nodes (cum <= 1.61kB)
Showing top 50 nodes out of 133
      flat  flat%   sum%        cum   cum%
   37.03kB 11.50% 11.50%    37.03kB 11.50%  runtime.procresize
      32kB  9.94% 21.43%       32kB  9.94%  hash/crc64.makeSlicingBy8Table
   21.39kB  6.64% 28.07%    21.39kB  6.64%  vendor/golang_org/x/text/unicode/norm.init
   19.98kB  6.20% 34.28%    19.98kB  6.20%  strings.NewReplacer
   18.95kB  5.88% 40.16%    18.97kB  5.89%  encoding/xml.init
   12.17kB  3.78% 43.94%    12.17kB  3.78%  unicode.init
   10.92kB  3.39% 47.33%    10.92kB  3.39%  regexp/syntax.appendRange
   10.61kB  3.29% 50.63%    10.61kB  3.29%  vendor/golang_org/x/net/http2/hpack.(*headerFieldTable).addEntry (inline)
   10.38kB  3.22% 53.85%    12.62kB  3.92%  sync.(*Map).LoadOrStore
    9.38kB  2.91% 56.76%     9.38kB  2.91%  runtime.malg
    9.17kB  2.85% 59.61%     9.22kB  2.86%  html/template.init
    8.70kB  2.70% 62.31%    69.47kB 21.57%  net/http.init
    6.84kB  2.12% 64.43%     6.84kB  2.12%  regexp/syntax.(*compiler).inst (inline)
    6.12kB  1.90% 66.34%    13.12kB  4.07%  runtime.allocm
    5.25kB  1.63% 67.97%     5.25kB  1.63%  math/rand.NewSource
    5.06kB  1.57% 69.54%     5.47kB  1.70%  net/http.init.0
    4.03kB  1.25% 70.79%     4.22kB  1.31%  net/http.init.2
    3.94kB  1.22% 72.01%     3.95kB  1.23%  debug/dwarf.init
    3.72kB  1.15% 73.17%     3.72kB  1.15%  errors.New
    3.62kB  1.13% 74.29%    12.13kB  3.77%  encoding/gob.validUserType
    3.62kB  1.13% 75.42%     3.62kB  1.13%  regexp/syntax.init
    2.91kB   0.9% 76.32%     2.91kB   0.9%  go/types.(*Scope).Insert
    2.70kB  0.84% 77.16%    26.97kB  8.37%  encoding/gob.init
    2.69kB  0.83% 77.99%     2.69kB  0.83%  flag.(*FlagSet).Var
    2.59kB  0.81% 78.80%     2.59kB  0.81%  net/mail.init.0
    2.25kB   0.7% 79.50%     2.25kB   0.7%  compress/flate.(*huffmanEncoder).generate
    2.25kB   0.7% 80.20%     2.25kB   0.7%  net/textproto.init.0
    2.09kB  0.65% 80.85%     2.09kB  0.65%  image/jpeg.(*huffmanLUT).init
    1.89kB  0.59% 81.43%     1.89kB  0.59%  compress/flate.newHuffmanEncoder
    1.80kB  0.56% 81.99%     1.80kB  0.56%  go/types.NewTypeName
    1.77kB  0.55% 82.54%     1.77kB  0.55%  text/template/parse.(*Tree).newText (inline)
    1.75kB  0.54% 83.08%     4.38kB  1.36%  runtime.mcommoninit
    1.66kB  0.51% 83.60%     8.84kB  2.74%  go/types.init
    1.64kB  0.51% 84.11%    28.67kB  8.90%  go/doc.init
    1.59kB  0.49% 84.60%     3.57kB  1.11%  encoding/gob.bootstrapType
    1.55kB  0.48% 85.08%     2.98kB  0.93%  net.init
    1.39kB  0.43% 85.51%    32.34kB 10.04%  go/build.init
    1.17kB  0.36% 85.88%     3.09kB  0.96%  text/template.init
    1.16kB  0.36% 86.24%     3.55kB  1.10%  encoding/gob.newTypeObject
    1.06kB  0.33% 86.57%    14.55kB  4.52%  encoding/gob.RegisterName
    1.02kB  0.32% 86.88%     4.57kB  1.42%  encoding/gob.buildTypeInfo
    0.94kB  0.29% 87.17%    21.06kB  6.54%  regexp.compile
    0.91kB  0.28% 87.45%    16.88kB  5.24%  encoding/gob.init.1
    0.75kB  0.23% 87.69%     1.84kB  0.57%  html/template.New
    0.69kB  0.21% 87.90%     2.13kB  0.66%  reflect.(*rtype).ptrTo
    0.67kB  0.21% 88.11%     7.20kB  2.24%  crypto/tls.init
    0.67kB  0.21% 88.32%    22.42kB  6.96%  vendor/golang_org/x/net/http/httpguts.init
    0.61kB  0.19% 88.51%    14.49kB  4.50%  archive/tar.init
    0.61kB  0.19% 88.70%     5.40kB  1.68%  net/http/pprof.init
    0.56kB  0.17% 88.87%     7.81kB  2.43%  runtime.systemstack

(pprof) top --cum 50
Showing nodes accounting for 222.77kB, 69.16% of 322.09kB total
Dropped 228 nodes (cum <= 1.61kB)
Showing top 50 nodes out of 133
      flat  flat%   sum%        cum   cum%
    0.09kB 0.029% 0.029%   268.74kB 83.44%  runtime.main
         0     0% 0.029%   266.68kB 82.80%  main.init
         0     0% 0.029%    70.38kB 21.85%  expvar.init
    8.70kB  2.70%  2.73%    69.47kB 21.57%  net/http.init
         0     0%  2.73%    37.41kB 11.61%  runtime.rt0_go
   37.03kB 11.50% 14.23%    37.03kB 11.50%  runtime.procresize
         0     0% 14.23%    37.03kB 11.50%  runtime.schedinit
    1.39kB  0.43% 14.66%    32.34kB 10.04%  go/build.init
         0     0% 14.66%       32kB  9.94%  hash/crc64.init
      32kB  9.94% 24.60%       32kB  9.94%  hash/crc64.makeSlicingBy8Table
    1.64kB  0.51% 25.10%    28.67kB  8.90%  go/doc.init
    2.70kB  0.84% 25.94%    26.97kB  8.37%  encoding/gob.init
    0.67kB  0.21% 26.15%    22.42kB  6.96%  vendor/golang_org/x/net/http/httpguts.init
         0     0% 26.15%    21.75kB  6.75%  vendor/golang_org/x/net/idna.init
   21.39kB  6.64% 32.79%    21.39kB  6.64%  vendor/golang_org/x/text/unicode/norm.init
         0     0% 32.79%    21.06kB  6.54%  regexp.Compile
         0     0% 32.79%    21.06kB  6.54%  regexp.MustCompile
    0.94kB  0.29% 33.09%    21.06kB  6.54%  regexp.compile
   19.98kB  6.20% 39.29%    19.98kB  6.20%  strings.NewReplacer
   18.95kB  5.88% 45.17%    18.97kB  5.89%  encoding/xml.init
    0.91kB  0.28% 45.46%    16.88kB  5.24%  encoding/gob.init.1
         0     0% 45.46%    14.55kB  4.52%  encoding/gob.Register
    1.06kB  0.33% 45.79%    14.55kB  4.52%  encoding/gob.RegisterName
         0     0% 45.79%    14.55kB  4.52%  encoding/gob.registerBasics
    0.61kB  0.19% 45.97%    14.49kB  4.50%  archive/tar.init
         0     0% 45.97%    13.70kB  4.25%  fmt.init
    6.12kB  1.90% 47.88%    13.12kB  4.07%  runtime.allocm
         0     0% 47.88%    12.72kB  3.95%  regexp/syntax.Parse
   10.38kB  3.22% 51.10%    12.62kB  3.92%  sync.(*Map).LoadOrStore
         0     0% 51.10%    12.17kB  3.78%  reflect.init
   12.17kB  3.78% 54.88%    12.17kB  3.78%  unicode.init
         0     0% 54.88%    12.13kB  3.77%  encoding/gob.userType
    3.62kB  1.13% 56.00%    12.13kB  3.77%  encoding/gob.validUserType
         0     0% 56.00%    11.94kB  3.71%  runtime.mstart
         0     0% 56.00%    11.50kB  3.57%  regexp/syntax.(*parser).parseClass
         0     0% 56.00%    11.25kB  3.49%  runtime.newm
   10.92kB  3.39% 59.39%    10.92kB  3.39%  regexp/syntax.appendRange
         0     0% 59.39%    10.81kB  3.36%  vendor/golang_org/x/net/http2/hpack.init
    0.05kB 0.015% 59.41%    10.75kB  3.34%  vendor/golang_org/x/net/http2/hpack.newStaticTable
   10.61kB  3.29% 62.70%    10.61kB  3.29%  vendor/golang_org/x/net/http2/hpack.(*headerFieldTable).addEntry (inline)
         0     0% 62.70%    10.50kB  3.26%  regexp/syntax.(*parser).parseUnicodeClass
         0     0% 62.70%    10.50kB  3.26%  regexp/syntax.appendTable
    9.38kB  2.91% 65.61%     9.38kB  2.91%  runtime.malg
    9.17kB  2.85% 68.46%     9.22kB  2.86%  html/template.init
         0     0% 68.46%     9.05kB  2.81%  go/importer.init
         0     0% 68.46%     9.05kB  2.81%  go/internal/gccgoimporter.init
    1.66kB  0.51% 68.97%     8.84kB  2.74%  go/types.init
    0.05kB 0.015% 68.99%     8.80kB  2.73%  mime/multipart.init
    0.56kB  0.17% 69.16%     7.81kB  2.43%  runtime.systemstack
         0     0% 69.16%     7.69kB  2.39%  html/template.(*Template).Parse

@bradfitz bradfitz added the Performance label Aug 2, 2018

@bradfitz bradfitz added this to the Unplanned milestone Aug 2, 2018

@bradfitz bradfitz added the help wanted label Aug 2, 2018

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Aug 2, 2018

Change https://golang.org/cl/127655 mentions this issue: net: lazily look up the listenerBacklog value on first use

gopherbot commented Aug 2, 2018

Change https://golang.org/cl/127655 mentions this issue: net: lazily look up the listenerBacklog value on first use

zegl added a commit to zegl/go that referenced this issue Aug 2, 2018

@zegl
encoding/gob: lazily initialize registerBasics 
Defer call to registerBasics until the first invocation of NewDecoder() or NewEncoder()

This saves ~13.5kB of allocation in init()

Updates golang#26775
f2e8552

@zegl zegl referenced this issue Aug 2, 2018

Closed

encoding/gob: lazily initialize registerBasics #26777

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Aug 2, 2018

Change https://golang.org/cl/127660 mentions this issue: encoding/gob: lazily initialize registerBasics

gopherbot commented Aug 2, 2018

Change https://golang.org/cl/127660 mentions this issue: encoding/gob: lazily initialize registerBasics
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Aug 2, 2018

Change https://golang.org/cl/127661 mentions this issue: strings: select Replacer algorithm and build machine lazily

gopherbot commented Aug 2, 2018

Change https://golang.org/cl/127661 mentions this issue: strings: select Replacer algorithm and build machine lazily
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Aug 2, 2018

Change https://golang.org/cl/127664 mentions this issue: http2: reduce init-time work & allocations

gopherbot commented Aug 2, 2018

Change https://golang.org/cl/127664 mentions this issue: http2: reduce init-time work & allocations

@bradfitz bradfitz referenced this issue Aug 3, 2018

Open

cmd/compile: static init maps should never generate code #2559

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Aug 3, 2018

Change https://golang.org/cl/127735 mentions this issue: hash/crc64: lazily initialize slice8Tables

gopherbot commented Aug 3, 2018

Change https://golang.org/cl/127735 mentions this issue: hash/crc64: lazily initialize slice8Tables
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Aug 3, 2018

Change https://golang.org/cl/127736 mentions this issue: net/mail: lazily initialize dateLayouts

gopherbot commented Aug 3, 2018

Change https://golang.org/cl/127736 mentions this issue: net/mail: lazily initialize dateLayouts
@ysmolsky

This comment has been minimized.

Show comment
Hide comment
@ysmolsky

ysmolsky Aug 3, 2018

Member

Wow, that was picked up real fast by the others!
Member

ysmolsky commented Aug 3, 2018

Wow, that was picked up real fast by the others!
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Aug 3, 2018

Change https://golang.org/cl/127875 mentions this issue: go/doc: compile regexps lazily

gopherbot commented Aug 3, 2018

Change https://golang.org/cl/127875 mentions this issue: go/doc: compile regexps lazily
@josharian

This comment has been minimized.

Show comment
Hide comment
@josharian

josharian Aug 5, 2018

Contributor

See also my last comment in #2559 about using go generate for some of this.
Contributor

josharian commented Aug 5, 2018

See also my last comment in #2559 about using go generate for some of this.

@gopherbot gopherbot referenced this issue Aug 20, 2018

Open

cmd/stringer: add the ability to optionally generate TypeFromString function #46

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Aug 21, 2018

Change https://golang.org/cl/119715 mentions this issue: encoding/json: change reflect.Type usage to be more idiomatic

gopherbot commented Aug 21, 2018

Change https://golang.org/cl/119715 mentions this issue: encoding/json: change reflect.Type usage to be more idiomatic

gopherbot pushed a commit that referenced this issue Aug 21, 2018

@bontibon @bradfitz
encoding/json: call reflect.TypeOf with nil pointers rather than allo… 
…cating

Updates #26775

Change-Id: I83c9eeda59769d2f35e0cc98f3a8579861d5978b
Reviewed-on: https://go-review.googlesource.com/119715
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
dc272a4

gopherbot pushed a commit to golang/net that referenced this issue Aug 21, 2018

@bradfitz
http2: reduce init-time work & allocations 
Updates golang/go#26775

Change-Id: Iea95ea07bb0fed42410efb4e8420d8e9a17704fe
Reviewed-on: https://go-review.googlesource.com/127664
Reviewed-by: Ian Lance Taylor <iant@golang.org>
922f481

gopherbot pushed a commit that referenced this issue Aug 21, 2018

@bradfitz
strings: select Replacer algorithm and build machine lazily 
Saves 22KB of memory in stdlib packages.

Updates #26775

Change-Id: Ia19fe7aff61f6e2ddd83cd35969d7ff94526591f
Reviewed-on: https://go-review.googlesource.com/127661
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
c544e0f

gopherbot pushed a commit that referenced this issue Aug 21, 2018

@bradfitz
go/doc: compile regexps lazily 
Compile go/doc's 4 regexps lazily, on demand.

Also, add a test for the one that had no test coverage.

This reduces init-time CPU as well as heap by ~20KB when they're not
used, which seems to be common enough. As an example, cmd/doc only
seems to use 1 of them. (as noted by temporary print statements)

Updates #26775

Change-Id: I85df89b836327a53fb8e1ace3f92480374270368
Reviewed-on: https://go-review.googlesource.com/127875
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
fc5107c

gopherbot pushed a commit that referenced this issue Aug 21, 2018

@bradfitz
net: lazily look up the listenerBacklog value on first use 
Don't open files or do sysctls in init.

Updates #26775

Change-Id: I017bed6c24ef1e4bc30040120349fb779f203225
Reviewed-on: https://go-review.googlesource.com/127655
Reviewed-by: Ian Lance Taylor <iant@golang.org>
80fe2e6

gopherbot pushed a commit that referenced this issue Aug 21, 2018

@choleraehyq @bradfitz
hash/crc64: lazily initialize slice8Tables 
Saves 36KB of memory in stdlib packages.

Updates #26775

Change-Id: I0f9d7b17d9768f6fb980d5fbba7c45920215a5fc
Reviewed-on: https://go-review.googlesource.com/127735
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2a5df06
@mark-rushakoff

This comment has been minimized.

Show comment
Hide comment
@mark-rushakoff

mark-rushakoff Aug 21, 2018

Contributor

Using all.go that Brad posted in the OP, and using go version devel +aa311fecda Tue Aug 21 22:11:05 2018 +0000 darwin/amd64, here are the current results:

$ gotip build all.go
$ GODEBUG=memprofilerate=1  ./all
278848
$ gotip tool pprof /tmp/all.mem.prof
Type: inuse_space
Time: Aug 21, 2018 at 3:15pm (PDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top 50
Showing nodes accounting for 228.01kB, 87.93% of 259.32kB total
Dropped 218 nodes (cum <= 1.30kB)
Showing top 50 nodes out of 139
      flat  flat%   sum%        cum   cum%
   37.03kB 14.28% 14.28%    37.03kB 14.28%  runtime.procresize
   21.39kB  8.25% 22.53%    21.39kB  8.25%  vendor/golang_org/x/text/unicode/norm.init
   18.95kB  7.31% 29.84%    18.97kB  7.31%  encoding/xml.init
   12.17kB  4.69% 34.53%    12.17kB  4.69%  unicode.init
   10.81kB  4.17% 38.70%    10.81kB  4.17%  vendor/golang_org/x/net/http2/hpack.(*headerFieldTable).addEntry (inline)
   10.58kB  4.08% 42.78%    12.82kB  4.94%  sync.(*Map).LoadOrStore
    9.17kB  3.54% 46.32%     9.22kB  3.55%  html/template.init
       9kB  3.47% 49.79%        9kB  3.47%  runtime.malg
    8.70kB  3.36% 53.14%    56.52kB 21.79%  net/http.init
       7kB  2.70% 55.84%       14kB  5.40%  runtime.allocm
    6.03kB  2.33% 58.17%     6.03kB  2.33%  syscall.copyenv
    5.25kB  2.02% 60.19%     5.25kB  2.02%  math/rand.NewSource
    4.78kB  1.84% 62.04%     5.19kB  2.00%  net/http.init.0
    4.03kB  1.55% 63.59%     4.22kB  1.63%  net/http.init.2
    3.94kB  1.52% 65.11%     3.95kB  1.52%  debug/dwarf.init
    3.81kB  1.47% 66.58%     3.81kB  1.47%  errors.New
    3.62kB  1.40% 67.98%    12.34kB  4.76%  encoding/gob.validUserType
    3.62kB  1.40% 69.38%     3.62kB  1.40%  regexp/syntax.init
    2.91kB  1.12% 70.50%     2.91kB  1.12%  go/types.(*Scope).Insert
    2.70kB  1.04% 71.54%    27.16kB 10.47%  encoding/gob.init
    2.69kB  1.04% 72.58%     2.69kB  1.04%  flag.(*FlagSet).Var
    2.59kB  1.00% 73.58%     2.59kB  1.00%  net/mail.init.0
    2.25kB  0.87% 74.44%     2.25kB  0.87%  compress/flate.(*huffmanEncoder).generate
    2.25kB  0.87% 75.31%     2.25kB  0.87%  net/textproto.init.0
    2.09kB  0.81% 76.12%     2.09kB  0.81%  image/jpeg.(*huffmanLUT).init
    1.89kB  0.73% 76.85%     1.89kB  0.73%  compress/flate.newHuffmanEncoder
    1.80kB  0.69% 77.54%     1.80kB  0.69%  go/types.NewTypeName
    1.77kB  0.68% 78.22%     1.77kB  0.68%  text/template/parse.(*Tree).newText
    1.75kB  0.67% 78.90%     4.38kB  1.69%  runtime.mcommoninit
    1.66kB  0.64% 79.53%     8.85kB  3.41%  go/types.init
    1.64kB  0.63% 80.17%    10.17kB  3.92%  go/doc.init
    1.59kB  0.61% 80.78%     3.74kB  1.44%  encoding/gob.bootstrapType
    1.56kB   0.6% 81.38%     1.56kB   0.6%  regexp/syntax.(*compiler).inst (inline)
    1.55kB   0.6% 81.98%     4.02kB  1.55%  net.init
    1.41kB  0.54% 82.52%     1.41kB  0.54%  go/types.newBuiltin (inline)
    1.39kB  0.54% 83.06%    13.84kB  5.34%  go/build.init
    1.38kB  0.53% 83.59%     1.38kB  0.53%  text/template.addValueFuncs
    1.31kB  0.51% 84.10%     1.31kB  0.51%  text/template/parse.(*Tree).newPipeline
    1.17kB  0.45% 84.55%     3.09kB  1.19%  text/template.init
    1.16kB  0.45% 85.00%     3.74kB  1.44%  encoding/gob.newTypeObject
    1.06kB  0.41% 85.41%    14.55kB  5.61%  encoding/gob.RegisterName
    1.02kB  0.39% 85.80%     4.76kB  1.83%  encoding/gob.buildTypeInfo
    0.75kB  0.29% 86.09%     1.84kB  0.71%  html/template.New
    0.75kB  0.29% 86.38%     7.88kB  3.04%  runtime.systemstack
    0.73kB  0.28% 86.66%    16.71kB  6.44%  encoding/gob.init.1
    0.69kB  0.27% 86.92%     2.13kB  0.82%  reflect.(*rtype).ptrTo
    0.67kB  0.26% 87.18%     6.92kB  2.67%  crypto/tls.init
    0.67kB  0.26% 87.44%    22.42kB  8.65%  vendor/golang_org/x/net/http/httpguts.init
    0.64kB  0.25% 87.69%     1.42kB  0.55%  net/http/cgi.init
    0.61kB  0.23% 87.93%    20.99kB  8.10%  archive/tar.init
(pprof) top --cum 50
Showing nodes accounting for 160.47kB, 61.88% of 259.32kB total
Dropped 218 nodes (cum <= 1.30kB)
Showing top 50 nodes out of 139
      flat  flat%   sum%        cum   cum%
    0.09kB 0.036% 0.036%   205.41kB 79.21%  runtime.main
         0     0% 0.036%   203.23kB 78.37%  main.init
         0     0% 0.036%    57.42kB 22.14%  expvar.init
    8.70kB  3.36%  3.39%    56.52kB 21.79%  net/http.init
         0     0%  3.39%    37.41kB 14.42%  runtime.rt0_go
   37.03kB 14.28% 17.67%    37.03kB 14.28%  runtime.procresize
         0     0% 17.67%    37.03kB 14.28%  runtime.schedinit
    2.70kB  1.04% 18.71%    27.16kB 10.47%  encoding/gob.init
    0.67kB  0.26% 18.97%    22.42kB  8.65%  vendor/golang_org/x/net/http/httpguts.init
         0     0% 18.97%    21.75kB  8.39%  vendor/golang_org/x/net/idna.init
   21.39kB  8.25% 27.22%    21.39kB  8.25%  vendor/golang_org/x/text/unicode/norm.init
    0.61kB  0.23% 27.46%    20.99kB  8.10%  archive/tar.init
         0     0% 27.46%    20.20kB  7.79%  fmt.init
   18.95kB  7.31% 34.77%    18.97kB  7.31%  encoding/xml.init
    0.73kB  0.28% 35.05%    16.71kB  6.44%  encoding/gob.init.1
         0     0% 35.05%    14.55kB  5.61%  encoding/gob.Register
    1.06kB  0.41% 35.46%    14.55kB  5.61%  encoding/gob.RegisterName
         0     0% 35.46%    14.55kB  5.61%  encoding/gob.registerBasics
       7kB  2.70% 38.16%       14kB  5.40%  runtime.allocm
    1.39kB  0.54% 38.69%    13.84kB  5.34%  go/build.init
   10.58kB  4.08% 42.77%    12.82kB  4.94%  sync.(*Map).LoadOrStore
         0     0% 42.77%    12.34kB  4.76%  encoding/gob.userType
    3.62kB  1.40% 44.17%    12.34kB  4.76%  encoding/gob.validUserType
         0     0% 44.17%    12.17kB  4.69%  reflect.init
   12.17kB  4.69% 48.87%    12.17kB  4.69%  unicode.init
         0     0% 48.87%       12kB  4.63%  runtime.newm
         0     0% 48.87%    11.02kB  4.25%  vendor/golang_org/x/net/http2/hpack.init
    0.05kB 0.018% 48.88%    10.95kB  4.22%  vendor/golang_org/x/net/http2/hpack.newStaticTable
   10.81kB  4.17% 53.05%    10.81kB  4.17%  vendor/golang_org/x/net/http2/hpack.(*headerFieldTable).addEntry (inline)
         0     0% 53.05%    10.25kB  3.95%  runtime.mstart
    1.64kB  0.63% 53.69%    10.17kB  3.92%  go/doc.init
    9.17kB  3.54% 57.22%     9.22kB  3.55%  html/template.init
         0     0% 57.22%     9.07kB  3.50%  go/importer.init
         0     0% 57.22%     9.07kB  3.50%  go/internal/gccgoimporter.init
       9kB  3.47% 60.69%        9kB  3.47%  runtime.malg
    1.66kB  0.64% 61.33%     8.85kB  3.41%  go/types.init
         0     0% 61.33%        8kB  3.08%  runtime.startm
    0.75kB  0.29% 61.62%     7.88kB  3.04%  runtime.systemstack
         0     0% 61.62%     7.82kB  3.02%  os.init
         0     0% 61.62%     7.80kB  3.01%  html/template.(*Template).Parse
         0     0% 61.62%     7.80kB  3.01%  text/template.(*Template).Parse
         0     0% 61.62%     7.39kB  2.85%  text/template/parse.Parse
         0     0% 61.62%     7.04kB  2.71%  go/types.init.0
    0.67kB  0.26% 61.88%     6.92kB  2.67%  crypto/tls.init
         0     0% 61.88%     6.89kB  2.66%  text/template/parse.(*Tree).Parse
         0     0% 61.88%     6.89kB  2.66%  text/template/parse.(*Tree).parse
         0     0% 61.88%     6.52kB  2.51%  text/template/parse.(*Tree).textOrAction
         0     0% 61.88%     6.16kB  2.37%  crypto/x509.init
         0     0% 61.88%     6.05kB  2.33%  os.Getwd
         0     0% 61.88%     6.03kB  2.33%  os.Getenv
Contributor

mark-rushakoff commented Aug 21, 2018

Using all.go that Brad posted in the OP, and using go version devel +aa311fecda Tue Aug 21 22:11:05 2018 +0000 darwin/amd64, here are the current results:

$ gotip build all.go
$ GODEBUG=memprofilerate=1  ./all
278848
$ gotip tool pprof /tmp/all.mem.prof
Type: inuse_space
Time: Aug 21, 2018 at 3:15pm (PDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top 50
Showing nodes accounting for 228.01kB, 87.93% of 259.32kB total
Dropped 218 nodes (cum <= 1.30kB)
Showing top 50 nodes out of 139
      flat  flat%   sum%        cum   cum%
   37.03kB 14.28% 14.28%    37.03kB 14.28%  runtime.procresize
   21.39kB  8.25% 22.53%    21.39kB  8.25%  vendor/golang_org/x/text/unicode/norm.init
   18.95kB  7.31% 29.84%    18.97kB  7.31%  encoding/xml.init
   12.17kB  4.69% 34.53%    12.17kB  4.69%  unicode.init
   10.81kB  4.17% 38.70%    10.81kB  4.17%  vendor/golang_org/x/net/http2/hpack.(*headerFieldTable).addEntry (inline)
   10.58kB  4.08% 42.78%    12.82kB  4.94%  sync.(*Map).LoadOrStore
    9.17kB  3.54% 46.32%     9.22kB  3.55%  html/template.init
       9kB  3.47% 49.79%        9kB  3.47%  runtime.malg
    8.70kB  3.36% 53.14%    56.52kB 21.79%  net/http.init
       7kB  2.70% 55.84%       14kB  5.40%  runtime.allocm
    6.03kB  2.33% 58.17%     6.03kB  2.33%  syscall.copyenv
    5.25kB  2.02% 60.19%     5.25kB  2.02%  math/rand.NewSource
    4.78kB  1.84% 62.04%     5.19kB  2.00%  net/http.init.0
    4.03kB  1.55% 63.59%     4.22kB  1.63%  net/http.init.2
    3.94kB  1.52% 65.11%     3.95kB  1.52%  debug/dwarf.init
    3.81kB  1.47% 66.58%     3.81kB  1.47%  errors.New
    3.62kB  1.40% 67.98%    12.34kB  4.76%  encoding/gob.validUserType
    3.62kB  1.40% 69.38%     3.62kB  1.40%  regexp/syntax.init
    2.91kB  1.12% 70.50%     2.91kB  1.12%  go/types.(*Scope).Insert
    2.70kB  1.04% 71.54%    27.16kB 10.47%  encoding/gob.init
    2.69kB  1.04% 72.58%     2.69kB  1.04%  flag.(*FlagSet).Var
    2.59kB  1.00% 73.58%     2.59kB  1.00%  net/mail.init.0
    2.25kB  0.87% 74.44%     2.25kB  0.87%  compress/flate.(*huffmanEncoder).generate
    2.25kB  0.87% 75.31%     2.25kB  0.87%  net/textproto.init.0
    2.09kB  0.81% 76.12%     2.09kB  0.81%  image/jpeg.(*huffmanLUT).init
    1.89kB  0.73% 76.85%     1.89kB  0.73%  compress/flate.newHuffmanEncoder
    1.80kB  0.69% 77.54%     1.80kB  0.69%  go/types.NewTypeName
    1.77kB  0.68% 78.22%     1.77kB  0.68%  text/template/parse.(*Tree).newText
    1.75kB  0.67% 78.90%     4.38kB  1.69%  runtime.mcommoninit
    1.66kB  0.64% 79.53%     8.85kB  3.41%  go/types.init
    1.64kB  0.63% 80.17%    10.17kB  3.92%  go/doc.init
    1.59kB  0.61% 80.78%     3.74kB  1.44%  encoding/gob.bootstrapType
    1.56kB   0.6% 81.38%     1.56kB   0.6%  regexp/syntax.(*compiler).inst (inline)
    1.55kB   0.6% 81.98%     4.02kB  1.55%  net.init
    1.41kB  0.54% 82.52%     1.41kB  0.54%  go/types.newBuiltin (inline)
    1.39kB  0.54% 83.06%    13.84kB  5.34%  go/build.init
    1.38kB  0.53% 83.59%     1.38kB  0.53%  text/template.addValueFuncs
    1.31kB  0.51% 84.10%     1.31kB  0.51%  text/template/parse.(*Tree).newPipeline
    1.17kB  0.45% 84.55%     3.09kB  1.19%  text/template.init
    1.16kB  0.45% 85.00%     3.74kB  1.44%  encoding/gob.newTypeObject
    1.06kB  0.41% 85.41%    14.55kB  5.61%  encoding/gob.RegisterName
    1.02kB  0.39% 85.80%     4.76kB  1.83%  encoding/gob.buildTypeInfo
    0.75kB  0.29% 86.09%     1.84kB  0.71%  html/template.New
    0.75kB  0.29% 86.38%     7.88kB  3.04%  runtime.systemstack
    0.73kB  0.28% 86.66%    16.71kB  6.44%  encoding/gob.init.1
    0.69kB  0.27% 86.92%     2.13kB  0.82%  reflect.(*rtype).ptrTo
    0.67kB  0.26% 87.18%     6.92kB  2.67%  crypto/tls.init
    0.67kB  0.26% 87.44%    22.42kB  8.65%  vendor/golang_org/x/net/http/httpguts.init
    0.64kB  0.25% 87.69%     1.42kB  0.55%  net/http/cgi.init
    0.61kB  0.23% 87.93%    20.99kB  8.10%  archive/tar.init
(pprof) top --cum 50
Showing nodes accounting for 160.47kB, 61.88% of 259.32kB total
Dropped 218 nodes (cum <= 1.30kB)
Showing top 50 nodes out of 139
      flat  flat%   sum%        cum   cum%
    0.09kB 0.036% 0.036%   205.41kB 79.21%  runtime.main
         0     0% 0.036%   203.23kB 78.37%  main.init
         0     0% 0.036%    57.42kB 22.14%  expvar.init
    8.70kB  3.36%  3.39%    56.52kB 21.79%  net/http.init
         0     0%  3.39%    37.41kB 14.42%  runtime.rt0_go
   37.03kB 14.28% 17.67%    37.03kB 14.28%  runtime.procresize
         0     0% 17.67%    37.03kB 14.28%  runtime.schedinit
    2.70kB  1.04% 18.71%    27.16kB 10.47%  encoding/gob.init
    0.67kB  0.26% 18.97%    22.42kB  8.65%  vendor/golang_org/x/net/http/httpguts.init
         0     0% 18.97%    21.75kB  8.39%  vendor/golang_org/x/net/idna.init
   21.39kB  8.25% 27.22%    21.39kB  8.25%  vendor/golang_org/x/text/unicode/norm.init
    0.61kB  0.23% 27.46%    20.99kB  8.10%  archive/tar.init
         0     0% 27.46%    20.20kB  7.79%  fmt.init
   18.95kB  7.31% 34.77%    18.97kB  7.31%  encoding/xml.init
    0.73kB  0.28% 35.05%    16.71kB  6.44%  encoding/gob.init.1
         0     0% 35.05%    14.55kB  5.61%  encoding/gob.Register
    1.06kB  0.41% 35.46%    14.55kB  5.61%  encoding/gob.RegisterName
         0     0% 35.46%    14.55kB  5.61%  encoding/gob.registerBasics
       7kB  2.70% 38.16%       14kB  5.40%  runtime.allocm
    1.39kB  0.54% 38.69%    13.84kB  5.34%  go/build.init
   10.58kB  4.08% 42.77%    12.82kB  4.94%  sync.(*Map).LoadOrStore
         0     0% 42.77%    12.34kB  4.76%  encoding/gob.userType
    3.62kB  1.40% 44.17%    12.34kB  4.76%  encoding/gob.validUserType
         0     0% 44.17%    12.17kB  4.69%  reflect.init
   12.17kB  4.69% 48.87%    12.17kB  4.69%  unicode.init
         0     0% 48.87%       12kB  4.63%  runtime.newm
         0     0% 48.87%    11.02kB  4.25%  vendor/golang_org/x/net/http2/hpack.init
    0.05kB 0.018% 48.88%    10.95kB  4.22%  vendor/golang_org/x/net/http2/hpack.newStaticTable
   10.81kB  4.17% 53.05%    10.81kB  4.17%  vendor/golang_org/x/net/http2/hpack.(*headerFieldTable).addEntry (inline)
         0     0% 53.05%    10.25kB  3.95%  runtime.mstart
    1.64kB  0.63% 53.69%    10.17kB  3.92%  go/doc.init
    9.17kB  3.54% 57.22%     9.22kB  3.55%  html/template.init
         0     0% 57.22%     9.07kB  3.50%  go/importer.init
         0     0% 57.22%     9.07kB  3.50%  go/internal/gccgoimporter.init
       9kB  3.47% 60.69%        9kB  3.47%  runtime.malg
    1.66kB  0.64% 61.33%     8.85kB  3.41%  go/types.init
         0     0% 61.33%        8kB  3.08%  runtime.startm
    0.75kB  0.29% 61.62%     7.88kB  3.04%  runtime.systemstack
         0     0% 61.62%     7.82kB  3.02%  os.init
         0     0% 61.62%     7.80kB  3.01%  html/template.(*Template).Parse
         0     0% 61.62%     7.80kB  3.01%  text/template.(*Template).Parse
         0     0% 61.62%     7.39kB  2.85%  text/template/parse.Parse
         0     0% 61.62%     7.04kB  2.71%  go/types.init.0
    0.67kB  0.26% 61.88%     6.92kB  2.67%  crypto/tls.init
         0     0% 61.88%     6.89kB  2.66%  text/template/parse.(*Tree).Parse
         0     0% 61.88%     6.89kB  2.66%  text/template/parse.(*Tree).parse
         0     0% 61.88%     6.52kB  2.51%  text/template/parse.(*Tree).textOrAction
         0     0% 61.88%     6.16kB  2.37%  crypto/x509.init
         0     0% 61.88%     6.05kB  2.33%  os.Getwd
         0     0% 61.88%     6.03kB  2.33%  os.Getenv
@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Aug 21, 2018

Member

Those top two vendor/golang_org/x ones fixed in x/ but need to be imported into std. That'll happen soon enough.
Member

bradfitz commented Aug 21, 2018

Those top two vendor/golang_org/x ones fixed in x/ but need to be imported into std. That'll happen soon enough.

gopherbot pushed a commit that referenced this issue Aug 22, 2018

@choleraehyq @bradfitz
net/mail: lazily initialize dateLayouts 
Saves 6KB of memory in stdlib packages.

Updates #26775

Change-Id: I1a6184cefa78e9a3c034fa84506fdfe0fec27add
Reviewed-on: https://go-review.googlesource.com/127736
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
39eda0d

@iand iand referenced this issue Oct 10, 2018

Open

encoding/xml: update character ranges for names to fifth edition (2008) specification #28124

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment