all: reduce init-time CPU & memory usage #26775

bradfitz · 2018-08-02T19:04:01Z

Tracking bug to reduce init-time CPU & memory usage.

Previously:

https://go-review.googlesource.com/c/go/+/127075 - html: lazily populate Unescape tables
https://go-review.googlesource.com/c/net/+/127275 - http2/hpack: lazily build huffman table on first use

Open:

#26752 - x/text/unicode/norm: reduce init-time memory usage

The program https://play.golang.org/p/9ervXCWzV_z is useful to find offenders:

$ go build all.go
$ GODEBUG=memprofilerate=1  ./all
346768
$ go tool pprof /tmp/all.mem.prof 
(pprof) top 50
Showing nodes accounting for 286.24kB, 88.87% of 322.09kB total
Dropped 228 nodes (cum <= 1.61kB)
Showing top 50 nodes out of 133
      flat  flat%   sum%        cum   cum%
   37.03kB 11.50% 11.50%    37.03kB 11.50%  runtime.procresize
      32kB  9.94% 21.43%       32kB  9.94%  hash/crc64.makeSlicingBy8Table
   21.39kB  6.64% 28.07%    21.39kB  6.64%  vendor/golang_org/x/text/unicode/norm.init
   19.98kB  6.20% 34.28%    19.98kB  6.20%  strings.NewReplacer
   18.95kB  5.88% 40.16%    18.97kB  5.89%  encoding/xml.init
   12.17kB  3.78% 43.94%    12.17kB  3.78%  unicode.init
   10.92kB  3.39% 47.33%    10.92kB  3.39%  regexp/syntax.appendRange
   10.61kB  3.29% 50.63%    10.61kB  3.29%  vendor/golang_org/x/net/http2/hpack.(*headerFieldTable).addEntry (inline)
   10.38kB  3.22% 53.85%    12.62kB  3.92%  sync.(*Map).LoadOrStore
    9.38kB  2.91% 56.76%     9.38kB  2.91%  runtime.malg
    9.17kB  2.85% 59.61%     9.22kB  2.86%  html/template.init
    8.70kB  2.70% 62.31%    69.47kB 21.57%  net/http.init
    6.84kB  2.12% 64.43%     6.84kB  2.12%  regexp/syntax.(*compiler).inst (inline)
    6.12kB  1.90% 66.34%    13.12kB  4.07%  runtime.allocm
    5.25kB  1.63% 67.97%     5.25kB  1.63%  math/rand.NewSource
    5.06kB  1.57% 69.54%     5.47kB  1.70%  net/http.init.0
    4.03kB  1.25% 70.79%     4.22kB  1.31%  net/http.init.2
    3.94kB  1.22% 72.01%     3.95kB  1.23%  debug/dwarf.init
    3.72kB  1.15% 73.17%     3.72kB  1.15%  errors.New
    3.62kB  1.13% 74.29%    12.13kB  3.77%  encoding/gob.validUserType
    3.62kB  1.13% 75.42%     3.62kB  1.13%  regexp/syntax.init
    2.91kB   0.9% 76.32%     2.91kB   0.9%  go/types.(*Scope).Insert
    2.70kB  0.84% 77.16%    26.97kB  8.37%  encoding/gob.init
    2.69kB  0.83% 77.99%     2.69kB  0.83%  flag.(*FlagSet).Var
    2.59kB  0.81% 78.80%     2.59kB  0.81%  net/mail.init.0
    2.25kB   0.7% 79.50%     2.25kB   0.7%  compress/flate.(*huffmanEncoder).generate
    2.25kB   0.7% 80.20%     2.25kB   0.7%  net/textproto.init.0
    2.09kB  0.65% 80.85%     2.09kB  0.65%  image/jpeg.(*huffmanLUT).init
    1.89kB  0.59% 81.43%     1.89kB  0.59%  compress/flate.newHuffmanEncoder
    1.80kB  0.56% 81.99%     1.80kB  0.56%  go/types.NewTypeName
    1.77kB  0.55% 82.54%     1.77kB  0.55%  text/template/parse.(*Tree).newText (inline)
    1.75kB  0.54% 83.08%     4.38kB  1.36%  runtime.mcommoninit
    1.66kB  0.51% 83.60%     8.84kB  2.74%  go/types.init
    1.64kB  0.51% 84.11%    28.67kB  8.90%  go/doc.init
    1.59kB  0.49% 84.60%     3.57kB  1.11%  encoding/gob.bootstrapType
    1.55kB  0.48% 85.08%     2.98kB  0.93%  net.init
    1.39kB  0.43% 85.51%    32.34kB 10.04%  go/build.init
    1.17kB  0.36% 85.88%     3.09kB  0.96%  text/template.init
    1.16kB  0.36% 86.24%     3.55kB  1.10%  encoding/gob.newTypeObject
    1.06kB  0.33% 86.57%    14.55kB  4.52%  encoding/gob.RegisterName
    1.02kB  0.32% 86.88%     4.57kB  1.42%  encoding/gob.buildTypeInfo
    0.94kB  0.29% 87.17%    21.06kB  6.54%  regexp.compile
    0.91kB  0.28% 87.45%    16.88kB  5.24%  encoding/gob.init.1
    0.75kB  0.23% 87.69%     1.84kB  0.57%  html/template.New
    0.69kB  0.21% 87.90%     2.13kB  0.66%  reflect.(*rtype).ptrTo
    0.67kB  0.21% 88.11%     7.20kB  2.24%  crypto/tls.init
    0.67kB  0.21% 88.32%    22.42kB  6.96%  vendor/golang_org/x/net/http/httpguts.init
    0.61kB  0.19% 88.51%    14.49kB  4.50%  archive/tar.init
    0.61kB  0.19% 88.70%     5.40kB  1.68%  net/http/pprof.init
    0.56kB  0.17% 88.87%     7.81kB  2.43%  runtime.systemstack

(pprof) top --cum 50
Showing nodes accounting for 222.77kB, 69.16% of 322.09kB total
Dropped 228 nodes (cum <= 1.61kB)
Showing top 50 nodes out of 133
      flat  flat%   sum%        cum   cum%
    0.09kB 0.029% 0.029%   268.74kB 83.44%  runtime.main
         0     0% 0.029%   266.68kB 82.80%  main.init
         0     0% 0.029%    70.38kB 21.85%  expvar.init
    8.70kB  2.70%  2.73%    69.47kB 21.57%  net/http.init
         0     0%  2.73%    37.41kB 11.61%  runtime.rt0_go
   37.03kB 11.50% 14.23%    37.03kB 11.50%  runtime.procresize
         0     0% 14.23%    37.03kB 11.50%  runtime.schedinit
    1.39kB  0.43% 14.66%    32.34kB 10.04%  go/build.init
         0     0% 14.66%       32kB  9.94%  hash/crc64.init
      32kB  9.94% 24.60%       32kB  9.94%  hash/crc64.makeSlicingBy8Table
    1.64kB  0.51% 25.10%    28.67kB  8.90%  go/doc.init
    2.70kB  0.84% 25.94%    26.97kB  8.37%  encoding/gob.init
    0.67kB  0.21% 26.15%    22.42kB  6.96%  vendor/golang_org/x/net/http/httpguts.init
         0     0% 26.15%    21.75kB  6.75%  vendor/golang_org/x/net/idna.init
   21.39kB  6.64% 32.79%    21.39kB  6.64%  vendor/golang_org/x/text/unicode/norm.init
         0     0% 32.79%    21.06kB  6.54%  regexp.Compile
         0     0% 32.79%    21.06kB  6.54%  regexp.MustCompile
    0.94kB  0.29% 33.09%    21.06kB  6.54%  regexp.compile
   19.98kB  6.20% 39.29%    19.98kB  6.20%  strings.NewReplacer
   18.95kB  5.88% 45.17%    18.97kB  5.89%  encoding/xml.init
    0.91kB  0.28% 45.46%    16.88kB  5.24%  encoding/gob.init.1
         0     0% 45.46%    14.55kB  4.52%  encoding/gob.Register
    1.06kB  0.33% 45.79%    14.55kB  4.52%  encoding/gob.RegisterName
         0     0% 45.79%    14.55kB  4.52%  encoding/gob.registerBasics
    0.61kB  0.19% 45.97%    14.49kB  4.50%  archive/tar.init
         0     0% 45.97%    13.70kB  4.25%  fmt.init
    6.12kB  1.90% 47.88%    13.12kB  4.07%  runtime.allocm
         0     0% 47.88%    12.72kB  3.95%  regexp/syntax.Parse
   10.38kB  3.22% 51.10%    12.62kB  3.92%  sync.(*Map).LoadOrStore
         0     0% 51.10%    12.17kB  3.78%  reflect.init
   12.17kB  3.78% 54.88%    12.17kB  3.78%  unicode.init
         0     0% 54.88%    12.13kB  3.77%  encoding/gob.userType
    3.62kB  1.13% 56.00%    12.13kB  3.77%  encoding/gob.validUserType
         0     0% 56.00%    11.94kB  3.71%  runtime.mstart
         0     0% 56.00%    11.50kB  3.57%  regexp/syntax.(*parser).parseClass
         0     0% 56.00%    11.25kB  3.49%  runtime.newm
   10.92kB  3.39% 59.39%    10.92kB  3.39%  regexp/syntax.appendRange
         0     0% 59.39%    10.81kB  3.36%  vendor/golang_org/x/net/http2/hpack.init
    0.05kB 0.015% 59.41%    10.75kB  3.34%  vendor/golang_org/x/net/http2/hpack.newStaticTable
   10.61kB  3.29% 62.70%    10.61kB  3.29%  vendor/golang_org/x/net/http2/hpack.(*headerFieldTable).addEntry (inline)
         0     0% 62.70%    10.50kB  3.26%  regexp/syntax.(*parser).parseUnicodeClass
         0     0% 62.70%    10.50kB  3.26%  regexp/syntax.appendTable
    9.38kB  2.91% 65.61%     9.38kB  2.91%  runtime.malg
    9.17kB  2.85% 68.46%     9.22kB  2.86%  html/template.init
         0     0% 68.46%     9.05kB  2.81%  go/importer.init
         0     0% 68.46%     9.05kB  2.81%  go/internal/gccgoimporter.init
    1.66kB  0.51% 68.97%     8.84kB  2.74%  go/types.init
    0.05kB 0.015% 68.99%     8.80kB  2.73%  mime/multipart.init
    0.56kB  0.17% 69.16%     7.81kB  2.43%  runtime.systemstack
         0     0% 69.16%     7.69kB  2.39%  html/template.(*Template).Parse

The text was updated successfully, but these errors were encountered:

gopherbot · 2018-08-02T19:16:34Z

Change https://golang.org/cl/127655 mentions this issue: net: lazily look up the listenerBacklog value on first use

Defer call to registerBasics until the first invocation of NewDecoder() or NewEncoder() This saves ~13.5kB of allocation in init() Updates golang#26775

gopherbot · 2018-08-02T20:04:29Z

Change https://golang.org/cl/127660 mentions this issue: encoding/gob: lazily initialize registerBasics

gopherbot · 2018-08-02T20:05:01Z

Change https://golang.org/cl/127661 mentions this issue: strings: select Replacer algorithm and build machine lazily

gopherbot · 2018-08-02T22:25:23Z

Change https://golang.org/cl/127664 mentions this issue: http2: reduce init-time work & allocations

gopherbot · 2018-08-03T06:36:12Z

Change https://golang.org/cl/127735 mentions this issue: hash/crc64: lazily initialize slice8Tables

gopherbot · 2018-08-03T06:57:26Z

Change https://golang.org/cl/127736 mentions this issue: net/mail: lazily initialize dateLayouts

ysmolsky · 2018-08-03T07:06:57Z

Wow, that was picked up real fast by the others!

gopherbot · 2018-08-03T19:24:54Z

Change https://golang.org/cl/127875 mentions this issue: go/doc: compile regexps lazily

josharian · 2018-08-05T22:47:43Z

See also my last comment in #2559 about using go generate for some of this.

gopherbot · 2018-08-21T02:24:34Z

Change https://golang.org/cl/119715 mentions this issue: encoding/json: change reflect.Type usage to be more idiomatic

…cating Updates #26775 Change-Id: I83c9eeda59769d2f35e0cc98f3a8579861d5978b Reviewed-on: https://go-review.googlesource.com/119715 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>

Updates golang/go#26775 Change-Id: Iea95ea07bb0fed42410efb4e8420d8e9a17704fe Reviewed-on: https://go-review.googlesource.com/127664 Reviewed-by: Ian Lance Taylor <iant@golang.org>

Saves 22KB of memory in stdlib packages. Updates #26775 Change-Id: Ia19fe7aff61f6e2ddd83cd35969d7ff94526591f Reviewed-on: https://go-review.googlesource.com/127661 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>

Compile go/doc's 4 regexps lazily, on demand. Also, add a test for the one that had no test coverage. This reduces init-time CPU as well as heap by ~20KB when they're not used, which seems to be common enough. As an example, cmd/doc only seems to use 1 of them. (as noted by temporary print statements) Updates #26775 Change-Id: I85df89b836327a53fb8e1ace3f92480374270368 Reviewed-on: https://go-review.googlesource.com/127875 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>

Don't open files or do sysctls in init. Updates #26775 Change-Id: I017bed6c24ef1e4bc30040120349fb779f203225 Reviewed-on: https://go-review.googlesource.com/127655 Reviewed-by: Ian Lance Taylor <iant@golang.org>

Saves 36KB of memory in stdlib packages. Updates #26775 Change-Id: I0f9d7b17d9768f6fb980d5fbba7c45920215a5fc Reviewed-on: https://go-review.googlesource.com/127735 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

mark-rushakoff · 2018-08-21T22:25:42Z

Using all.go that Brad posted in the OP, and using go version devel +aa311fecda Tue Aug 21 22:11:05 2018 +0000 darwin/amd64, here are the current results:

$ gotip build all.go
$ GODEBUG=memprofilerate=1  ./all
278848
$ gotip tool pprof /tmp/all.mem.prof
Type: inuse_space
Time: Aug 21, 2018 at 3:15pm (PDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top 50
Showing nodes accounting for 228.01kB, 87.93% of 259.32kB total
Dropped 218 nodes (cum <= 1.30kB)
Showing top 50 nodes out of 139
      flat  flat%   sum%        cum   cum%
   37.03kB 14.28% 14.28%    37.03kB 14.28%  runtime.procresize
   21.39kB  8.25% 22.53%    21.39kB  8.25%  vendor/golang_org/x/text/unicode/norm.init
   18.95kB  7.31% 29.84%    18.97kB  7.31%  encoding/xml.init
   12.17kB  4.69% 34.53%    12.17kB  4.69%  unicode.init
   10.81kB  4.17% 38.70%    10.81kB  4.17%  vendor/golang_org/x/net/http2/hpack.(*headerFieldTable).addEntry (inline)
   10.58kB  4.08% 42.78%    12.82kB  4.94%  sync.(*Map).LoadOrStore
    9.17kB  3.54% 46.32%     9.22kB  3.55%  html/template.init
       9kB  3.47% 49.79%        9kB  3.47%  runtime.malg
    8.70kB  3.36% 53.14%    56.52kB 21.79%  net/http.init
       7kB  2.70% 55.84%       14kB  5.40%  runtime.allocm
    6.03kB  2.33% 58.17%     6.03kB  2.33%  syscall.copyenv
    5.25kB  2.02% 60.19%     5.25kB  2.02%  math/rand.NewSource
    4.78kB  1.84% 62.04%     5.19kB  2.00%  net/http.init.0
    4.03kB  1.55% 63.59%     4.22kB  1.63%  net/http.init.2
    3.94kB  1.52% 65.11%     3.95kB  1.52%  debug/dwarf.init
    3.81kB  1.47% 66.58%     3.81kB  1.47%  errors.New
    3.62kB  1.40% 67.98%    12.34kB  4.76%  encoding/gob.validUserType
    3.62kB  1.40% 69.38%     3.62kB  1.40%  regexp/syntax.init
    2.91kB  1.12% 70.50%     2.91kB  1.12%  go/types.(*Scope).Insert
    2.70kB  1.04% 71.54%    27.16kB 10.47%  encoding/gob.init
    2.69kB  1.04% 72.58%     2.69kB  1.04%  flag.(*FlagSet).Var
    2.59kB  1.00% 73.58%     2.59kB  1.00%  net/mail.init.0
    2.25kB  0.87% 74.44%     2.25kB  0.87%  compress/flate.(*huffmanEncoder).generate
    2.25kB  0.87% 75.31%     2.25kB  0.87%  net/textproto.init.0
    2.09kB  0.81% 76.12%     2.09kB  0.81%  image/jpeg.(*huffmanLUT).init
    1.89kB  0.73% 76.85%     1.89kB  0.73%  compress/flate.newHuffmanEncoder
    1.80kB  0.69% 77.54%     1.80kB  0.69%  go/types.NewTypeName
    1.77kB  0.68% 78.22%     1.77kB  0.68%  text/template/parse.(*Tree).newText
    1.75kB  0.67% 78.90%     4.38kB  1.69%  runtime.mcommoninit
    1.66kB  0.64% 79.53%     8.85kB  3.41%  go/types.init
    1.64kB  0.63% 80.17%    10.17kB  3.92%  go/doc.init
    1.59kB  0.61% 80.78%     3.74kB  1.44%  encoding/gob.bootstrapType
    1.56kB   0.6% 81.38%     1.56kB   0.6%  regexp/syntax.(*compiler).inst (inline)
    1.55kB   0.6% 81.98%     4.02kB  1.55%  net.init
    1.41kB  0.54% 82.52%     1.41kB  0.54%  go/types.newBuiltin (inline)
    1.39kB  0.54% 83.06%    13.84kB  5.34%  go/build.init
    1.38kB  0.53% 83.59%     1.38kB  0.53%  text/template.addValueFuncs
    1.31kB  0.51% 84.10%     1.31kB  0.51%  text/template/parse.(*Tree).newPipeline
    1.17kB  0.45% 84.55%     3.09kB  1.19%  text/template.init
    1.16kB  0.45% 85.00%     3.74kB  1.44%  encoding/gob.newTypeObject
    1.06kB  0.41% 85.41%    14.55kB  5.61%  encoding/gob.RegisterName
    1.02kB  0.39% 85.80%     4.76kB  1.83%  encoding/gob.buildTypeInfo
    0.75kB  0.29% 86.09%     1.84kB  0.71%  html/template.New
    0.75kB  0.29% 86.38%     7.88kB  3.04%  runtime.systemstack
    0.73kB  0.28% 86.66%    16.71kB  6.44%  encoding/gob.init.1
    0.69kB  0.27% 86.92%     2.13kB  0.82%  reflect.(*rtype).ptrTo
    0.67kB  0.26% 87.18%     6.92kB  2.67%  crypto/tls.init
    0.67kB  0.26% 87.44%    22.42kB  8.65%  vendor/golang_org/x/net/http/httpguts.init
    0.64kB  0.25% 87.69%     1.42kB  0.55%  net/http/cgi.init
    0.61kB  0.23% 87.93%    20.99kB  8.10%  archive/tar.init
(pprof) top --cum 50
Showing nodes accounting for 160.47kB, 61.88% of 259.32kB total
Dropped 218 nodes (cum <= 1.30kB)
Showing top 50 nodes out of 139
      flat  flat%   sum%        cum   cum%
    0.09kB 0.036% 0.036%   205.41kB 79.21%  runtime.main
         0     0% 0.036%   203.23kB 78.37%  main.init
         0     0% 0.036%    57.42kB 22.14%  expvar.init
    8.70kB  3.36%  3.39%    56.52kB 21.79%  net/http.init
         0     0%  3.39%    37.41kB 14.42%  runtime.rt0_go
   37.03kB 14.28% 17.67%    37.03kB 14.28%  runtime.procresize
         0     0% 17.67%    37.03kB 14.28%  runtime.schedinit
    2.70kB  1.04% 18.71%    27.16kB 10.47%  encoding/gob.init
    0.67kB  0.26% 18.97%    22.42kB  8.65%  vendor/golang_org/x/net/http/httpguts.init
         0     0% 18.97%    21.75kB  8.39%  vendor/golang_org/x/net/idna.init
   21.39kB  8.25% 27.22%    21.39kB  8.25%  vendor/golang_org/x/text/unicode/norm.init
    0.61kB  0.23% 27.46%    20.99kB  8.10%  archive/tar.init
         0     0% 27.46%    20.20kB  7.79%  fmt.init
   18.95kB  7.31% 34.77%    18.97kB  7.31%  encoding/xml.init
    0.73kB  0.28% 35.05%    16.71kB  6.44%  encoding/gob.init.1
         0     0% 35.05%    14.55kB  5.61%  encoding/gob.Register
    1.06kB  0.41% 35.46%    14.55kB  5.61%  encoding/gob.RegisterName
         0     0% 35.46%    14.55kB  5.61%  encoding/gob.registerBasics
       7kB  2.70% 38.16%       14kB  5.40%  runtime.allocm
    1.39kB  0.54% 38.69%    13.84kB  5.34%  go/build.init
   10.58kB  4.08% 42.77%    12.82kB  4.94%  sync.(*Map).LoadOrStore
         0     0% 42.77%    12.34kB  4.76%  encoding/gob.userType
    3.62kB  1.40% 44.17%    12.34kB  4.76%  encoding/gob.validUserType
         0     0% 44.17%    12.17kB  4.69%  reflect.init
   12.17kB  4.69% 48.87%    12.17kB  4.69%  unicode.init
         0     0% 48.87%       12kB  4.63%  runtime.newm
         0     0% 48.87%    11.02kB  4.25%  vendor/golang_org/x/net/http2/hpack.init
    0.05kB 0.018% 48.88%    10.95kB  4.22%  vendor/golang_org/x/net/http2/hpack.newStaticTable
   10.81kB  4.17% 53.05%    10.81kB  4.17%  vendor/golang_org/x/net/http2/hpack.(*headerFieldTable).addEntry (inline)
         0     0% 53.05%    10.25kB  3.95%  runtime.mstart
    1.64kB  0.63% 53.69%    10.17kB  3.92%  go/doc.init
    9.17kB  3.54% 57.22%     9.22kB  3.55%  html/template.init
         0     0% 57.22%     9.07kB  3.50%  go/importer.init
         0     0% 57.22%     9.07kB  3.50%  go/internal/gccgoimporter.init
       9kB  3.47% 60.69%        9kB  3.47%  runtime.malg
    1.66kB  0.64% 61.33%     8.85kB  3.41%  go/types.init
         0     0% 61.33%        8kB  3.08%  runtime.startm
    0.75kB  0.29% 61.62%     7.88kB  3.04%  runtime.systemstack
         0     0% 61.62%     7.82kB  3.02%  os.init
         0     0% 61.62%     7.80kB  3.01%  html/template.(*Template).Parse
         0     0% 61.62%     7.80kB  3.01%  text/template.(*Template).Parse
         0     0% 61.62%     7.39kB  2.85%  text/template/parse.Parse
         0     0% 61.62%     7.04kB  2.71%  go/types.init.0
    0.67kB  0.26% 61.88%     6.92kB  2.67%  crypto/tls.init
         0     0% 61.88%     6.89kB  2.66%  text/template/parse.(*Tree).Parse
         0     0% 61.88%     6.89kB  2.66%  text/template/parse.(*Tree).parse
         0     0% 61.88%     6.52kB  2.51%  text/template/parse.(*Tree).textOrAction
         0     0% 61.88%     6.16kB  2.37%  crypto/x509.init
         0     0% 61.88%     6.05kB  2.33%  os.Getwd
         0     0% 61.88%     6.03kB  2.33%  os.Getenv

bradfitz · 2018-08-21T22:33:52Z

Those top two vendor/golang_org/x ones fixed in x/ but need to be imported into std. That'll happen soon enough.

Saves 6KB of memory in stdlib packages. Updates #26775 Change-Id: I1a6184cefa78e9a3c034fa84506fdfe0fec27add Reviewed-on: https://go-review.googlesource.com/127736 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

gopherbot · 2019-03-09T17:06:26Z

Change https://golang.org/cl/166459 mentions this issue: cmd/go: further reduce init work

The first biggest offender was crypto/des.init at ~1%. It's cryptographically broken and the init function is relatively expensive, which is unfortunate as both crypto/tls and crypto/x509 (and by extension, cmd/go) import it. Hide the work behind sync.Once. The second biggest offender was flag.sortFlags at just under 1%, used by the Visit flagset methods. It allocated two slices, which made a difference as cmd/go iterates over multiple flagsets during init. Use a single slice with a direct sort.Interface implementation. Another big offender is initializing global maps. Reducing this work in cmd/go/internal/imports and net/textproto gives us close to another whole 1% in saved work. The former can use map literals, and the latter can hide the work behind sync.Once. Finally, compress/flate used newHuffmanBitWriter as part of init, which allocates many objects and slices. Yet it only used one of the slice fields. Allocating just that slice saves a surprising ~0.3%, since we generated a lot of unnecessary garbage. All in all, these little pieces amount to just over 3% saved CPU time. name old time/op new time/op delta ExecGoEnv-8 3.61ms ± 1% 3.50ms ± 0% -3.02% (p=0.000 n=10+10) Updates #26775. Updates #29382. Change-Id: I915416e88a874c63235ba512617c8aef35c0ca8b Reviewed-on: https://go-review.googlesource.com/c/go/+/166459 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

gopherbot · 2019-04-01T14:10:56Z

Change https://golang.org/cl/170317 mentions this issue: net/textproto: simplify commonHeader initialization

gopherbot · 2019-12-06T19:29:15Z

Change https://golang.org/cl/210284 mentions this issue: text/template: avoid a global map to help the linker's deadcode elimination

…nation Fixes #36021 Updates #2559 Updates #26775 Change-Id: I2e6708691311035b63866f25d5b4b3977a118290 Reviewed-on: https://go-review.googlesource.com/c/go/+/210284 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org> Reviewed-by: Rob Pike <r@golang.org>

…nation Fixes golang#36021 Updates golang#2559 Updates golang#26775 Change-Id: I2e6708691311035b63866f25d5b4b3977a118290 Reviewed-on: https://go-review.googlesource.com/c/go/+/210284 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org> Reviewed-by: Rob Pike <r@golang.org>

cristaloleg · 2021-12-05T11:39:32Z

Isn't this resolved?

❯ go version
go version go1.17.2 darwin/amd64

❯ go build all.go

❯ GODEBUG=memprofilerate=1  ./all
289408

❯ go tool pprof /tmp/all.mem.prof 
Type: inuse_space
Time: Dec 5, 2021 at 12:35pm (CET)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top 50
Showing nodes accounting for 9751.47kB, 100% of 9751.47kB total
      flat  flat%   sum%        cum   cum%
 3585.42kB 36.77% 36.77%  3585.42kB 36.77%  runtime.malg
 3092.09kB 31.71% 68.48%  3092.09kB 31.71%  runtime.procresize
 1537.69kB 15.77% 84.25%  4610.75kB 47.28%  runtime.allocm
 1024.25kB 10.50% 94.75%  2048.66kB 21.01%  runtime.mcommoninit
  512.02kB  5.25%   100%   512.02kB  5.25%  text/template/parse.(*Tree).newList (inline)
         0     0%   100%   512.02kB  5.25%  html/template.(*Template).Parse
         0     0%   100%   512.02kB  5.25%  net/rpc.init
         0     0%   100%   512.02kB  5.25%  runtime.doInit
         0     0%   100%   512.02kB  5.25%  runtime.main
         0     0%   100%  2049.09kB 21.01%  runtime.main.func1
         0     0%   100%  1024.41kB 10.51%  runtime.mpreinit
         0     0%   100%  3073.86kB 31.52%  runtime.mstart
         0     0%   100%  3073.86kB 31.52%  runtime.mstart0
         0     0%   100%  3073.86kB 31.52%  runtime.mstart1
         0     0%   100%  2561.30kB 26.27%  runtime.mstartm0
         0     0%   100%  2561.30kB 26.27%  runtime.newextram
         0     0%   100%  2561.66kB 26.27%  runtime.newm
         0     0%   100%   512.20kB  5.25%  runtime.newproc
         0     0%   100%  1024.41kB 10.51%  runtime.newproc.func1
         0     0%   100%  1024.41kB 10.51%  runtime.newproc1
         0     0%   100%  2561.30kB 26.27%  runtime.oneNewExtraM
         0     0%   100%   512.56kB  5.26%  runtime.resetspinning
         0     0%   100%  3604.29kB 36.96%  runtime.rt0_go
         0     0%   100%  3092.09kB 31.71%  runtime.schedinit
         0     0%   100%   512.56kB  5.26%  runtime.schedule
         0     0%   100%   512.56kB  5.26%  runtime.startm
         0     0%   100%  2561.30kB 26.27%  runtime.systemstack
         0     0%   100%   512.56kB  5.26%  runtime.wakep
         0     0%   100%   512.02kB  5.25%  text/template.(*Template).Parse
         0     0%   100%   512.02kB  5.25%  text/template/parse.(*Tree).Parse
         0     0%   100%   512.02kB  5.25%  text/template/parse.(*Tree).action
         0     0%   100%   512.02kB  5.25%  text/template/parse.(*Tree).itemList
         0     0%   100%   512.02kB  5.25%  text/template/parse.(*Tree).parse
         0     0%   100%   512.02kB  5.25%  text/template/parse.(*Tree).parseControl
         0     0%   100%   512.02kB  5.25%  text/template/parse.(*Tree).rangeControl
         0     0%   100%   512.02kB  5.25%  text/template/parse.(*Tree).textOrAction
         0     0%   100%   512.02kB  5.25%  text/template/parse.Parse
(pprof)

Updates golang/go#26775 Change-Id: Iea95ea07bb0fed42410efb4e8420d8e9a17704fe Reviewed-on: https://go-review.googlesource.com/127664 Reviewed-by: Ian Lance Taylor <iant@golang.org>

bradfitz added the Performance label Aug 2, 2018

bradfitz added this to the Unplanned milestone Aug 2, 2018

bradfitz added the help wanted label Aug 2, 2018

zegl added a commit to zegl/go that referenced this issue Aug 2, 2018

encoding/gob: lazily initialize registerBasics

f2e8552

Defer call to registerBasics until the first invocation of NewDecoder() or NewEncoder() This saves ~13.5kB of allocation in init() Updates golang#26775

zegl mentioned this issue Aug 2, 2018

encoding/gob: lazily initialize registerBasics #26777

Closed

bradfitz mentioned this issue Aug 3, 2018

cmd/compile: static init maps should never generate code #2559

Open

gopherbot mentioned this issue Aug 20, 2018

cmd/stringer: add the ability to optionally generate TypeFromString function golang/tools#46

Closed

iand mentioned this issue Oct 10, 2018

proposal: encoding/xml: update character ranges for names to fifth edition (2008) specification #28124

Open

mark-rushakoff mentioned this issue Dec 21, 2018

cmd/go: go env GOMOD is too slow for use in interactive commands #29382

Closed

esote mentioned this issue May 9, 2019

net: Moving listenerBacklog from init interferes with use of OpenBSD's pledge syscall #31927

Closed

bradfitz mentioned this issue Nov 12, 2019

x/text/unicode/norm: reduce init-time memory usage #26752

Open

Tatskaari mentioned this issue Sep 3, 2021

proposal: spec: Go 2: allow manual control over imported package initialization #48174

Closed

all: reduce init-time CPU & memory usage #26775

all: reduce init-time CPU & memory usage #26775

bradfitz commented Aug 2, 2018 •

edited

gopherbot commented Aug 2, 2018

gopherbot commented Aug 2, 2018

gopherbot commented Aug 2, 2018

gopherbot commented Aug 2, 2018

gopherbot commented Aug 3, 2018

gopherbot commented Aug 3, 2018

ysmolsky commented Aug 3, 2018

gopherbot commented Aug 3, 2018

josharian commented Aug 5, 2018

gopherbot commented Aug 21, 2018

mark-rushakoff commented Aug 21, 2018

bradfitz commented Aug 21, 2018

gopherbot commented Mar 9, 2019

gopherbot commented Apr 1, 2019

gopherbot commented Dec 6, 2019

cristaloleg commented Dec 5, 2021

all: reduce init-time CPU & memory usage #26775

all: reduce init-time CPU & memory usage #26775

Comments

bradfitz commented Aug 2, 2018 • edited

gopherbot commented Aug 2, 2018

gopherbot commented Aug 2, 2018

gopherbot commented Aug 2, 2018

gopherbot commented Aug 2, 2018

gopherbot commented Aug 3, 2018

gopherbot commented Aug 3, 2018

ysmolsky commented Aug 3, 2018

gopherbot commented Aug 3, 2018

josharian commented Aug 5, 2018

gopherbot commented Aug 21, 2018

mark-rushakoff commented Aug 21, 2018

bradfitz commented Aug 21, 2018

gopherbot commented Mar 9, 2019

gopherbot commented Apr 1, 2019

gopherbot commented Dec 6, 2019

cristaloleg commented Dec 5, 2021

bradfitz commented Aug 2, 2018 •

edited