Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

0.55: High memory usage in multisite with un-needed clone of shortcode templates #5890

Closed
bep opened this issue Apr 23, 2019 · 1 comment

Comments

Projects
None yet
1 participant
@bep
Copy link
Member

commented Apr 23, 2019

See https://discourse.gohugo.io/t/out-of-memory-with-hugo-0-55/18072/4

The repo linked to in that comment shows a 3,5x increase in "Maximum resident set size (kbytes)" reported from gnutime (the runtime is, however, about the same on my MacBook).

  • Since the site building benchmarks indicate the opposite behaviour, this is probably something that is not covered by those.
  • Also, doing the same compare with the Hugo site also shows an increase, but small compared
  • Comparing the Hugo site with the other site can, at first glance, indicate that how we treat the so-called "Non-page files" has changed. Wil check.

@bep bep added the Bug label Apr 23, 2019

@bep bep self-assigned this Apr 23, 2019

@bep bep added this to the v0.55.4 milestone Apr 23, 2019

@bep

This comment has been minimized.

Copy link
Member Author

commented Apr 23, 2019

pprof is pretty useful:

(pprof) top
Showing nodes accounting for 2375.53MB, 81.77% of 2905.21MB total
Dropped 340 nodes (cum <= 14.53MB)
Showing top 10 nodes out of 114
      flat  flat%   sum%        cum   cum%
 1689.44MB 58.15% 58.15%  1689.44MB 58.15%  github.com/nicksnyder/go-i18n/i18n/bundle.(*Bundle).Translations
  273.51MB  9.41% 67.57%  2321.03MB 79.89%  reflect.Value.call
   87.51MB  3.01% 70.58%    87.51MB  3.01%  text/template/parse.(*Tree).newPipeline
      54MB  1.86% 72.44%       54MB  1.86%  reflect.FuncOf
      51MB  1.76% 74.19%       51MB  1.76%  text/template/parse.(*CommandNode).append
   48.04MB  1.65% 75.85%    48.04MB  1.65%  bytes.makeSlice
      46MB  1.58% 77.43%    58.50MB  2.01%  reflect.MakeSlice

Sadly, there still seems to be a "new memory" issue with that site, even if I get rid of that call to Translations above, which is fairly unusual I suspect (lots of untranslated strings). But the site builds twice as fast ...

@bep bep changed the title Check memory usage between 0.54 and 0.55 0.55: High memory usage in multisite with un-needed clone of shortcode templates Apr 24, 2019

bep added a commit to bep/hugo that referenced this issue Apr 24, 2019

hugolib: Avoid recloning of shortcode templates
```bash
benchmark                                    old ns/op     new ns/op     delta
BenchmarkSiteNew/Bundle_with_image-4         14572242      14382188      -1.30%
BenchmarkSiteNew/Bundle_with_JSON_file-4     13683922      13738196      +0.40%
BenchmarkSiteNew/Multiple_languages-4        41912231      25192494      -39.89%

benchmark                                    old allocs     new allocs     delta
BenchmarkSiteNew/Bundle_with_image-4         57496          57493          -0.01%
BenchmarkSiteNew/Bundle_with_JSON_file-4     57492          57501          +0.02%
BenchmarkSiteNew/Multiple_languages-4        242422         118809         -50.99%

benchmark                                    old bytes     new bytes     delta
BenchmarkSiteNew/Bundle_with_image-4         3845077       3844065       -0.03%
BenchmarkSiteNew/Bundle_with_JSON_file-4     3627442       3627798       +0.01%
BenchmarkSiteNew/Multiple_languages-4        13963502      7543885       -45.97%
```

Fixes gohugoio#5890

@bep bep closed this in #5897 Apr 24, 2019

bep added a commit that referenced this issue Apr 24, 2019

hugolib: Avoid recloning of shortcode templates
```bash
benchmark                                    old ns/op     new ns/op     delta
BenchmarkSiteNew/Bundle_with_image-4         14572242      14382188      -1.30%
BenchmarkSiteNew/Bundle_with_JSON_file-4     13683922      13738196      +0.40%
BenchmarkSiteNew/Multiple_languages-4        41912231      25192494      -39.89%

benchmark                                    old allocs     new allocs     delta
BenchmarkSiteNew/Bundle_with_image-4         57496          57493          -0.01%
BenchmarkSiteNew/Bundle_with_JSON_file-4     57492          57501          +0.02%
BenchmarkSiteNew/Multiple_languages-4        242422         118809         -50.99%

benchmark                                    old bytes     new bytes     delta
BenchmarkSiteNew/Bundle_with_image-4         3845077       3844065       -0.03%
BenchmarkSiteNew/Bundle_with_JSON_file-4     3627442       3627798       +0.01%
BenchmarkSiteNew/Multiple_languages-4        13963502      7543885       -45.97%
```

Fixes #5890
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.