Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(log): enhance memory efficiency for the zap logger with fields #340

Merged
merged 4 commits into from
Aug 3, 2023

Conversation

tessig
Copy link
Member

@tessig tessig commented Jun 22, 2023

An often used pattern for the flamingo.Logger is to store an instance with prepared fields in a struct field.
The logger is then initialized with the fields in the struct's Inject method.

The underlying zap logger is then cloned and in big project we allocate a lot of memory for this.

This change keeps the fields in the layer of the flamingo logger until it comes to a message write. Then the fields are passed to zap.

The benchmark in core/zap/logger_test.go promises better memory efficiency with this change.

@tessig
Copy link
Member Author

tessig commented Jul 3, 2023

In addition, let's also consider the upcoming structured logger in Go https://go.googlesource.com/proposal/+/master/design/56345-structured-logging.md

@tessig
Copy link
Member Author

tessig commented Jul 3, 2023

benchstat for this change (core/zap/logger_test.go)

goos: darwin
goarch: arm64
pkg: flamingo.me/flamingo/v3/core/zap
                               │  old.bench  │              new.bench              │
                               │   sec/op    │   sec/op     vs base                │
Logger/withField-1-log-10        3.354µ ± 0%   1.642µ ± 1%  -51.05% (n=50)
Logger/withFields-1-log-10       1.629µ ± 1%   1.312µ ± 1%  -19.46% (n=50)
Logger/withField-each-log-10     8.399µ ± 0%   6.760µ ± 1%  -19.52% (n=50)
Logger/withFields-many-logs-10   9.830µ ± 1%   9.402µ ± 0%   -4.35% (p=0.000 n=50)
geomean                          4.608µ        3.420µ       -25.78%

                               │   old.bench   │          new.bench           │
                               │     B/op      │     B/op      vs base        │
Logger/withField-1-log-10         9.009Ki ± 0%   2.813Ki ± 0%  -68.77% (n=50)
Logger/withFields-1-log-10        2.769Ki ± 0%   1.579Ki ± 0%  -42.96% (n=50)
Logger/withField-each-log-10     12.769Ki ± 0%   6.567Ki ± 0%  -48.57% (n=50)
Logger/withFields-many-logs-10    9.533Ki ± 0%   8.334Ki ± 0%  -12.58% (n=50)
geomean                           7.423Ki        3.949Ki       -46.80%

                               │ old.bench  │         new.bench          │
                               │ allocs/op  │ allocs/op   vs base        │
Logger/withField-1-log-10        64.00 ± 0%   28.00 ± 0%  -56.25% (n=50)
Logger/withFields-1-log-10       26.00 ± 0%   20.00 ± 0%  -23.08% (n=50)
Logger/withField-each-log-10     144.0 ± 0%   108.0 ± 0%  -25.00% (n=50)
Logger/withFields-many-logs-10   170.0 ± 0%   164.0 ± 0%   -3.53% (n=50)
geomean                          79.89        56.12       -29.75%

@tessig tessig marked this pull request as ready for review July 28, 2023 13:06
@github-actions
Copy link
Contributor

Code Coverage

Package Line Rate Health
flamingo.me/flamingo/v3/core/auth 13%
flamingo.me/flamingo/v3/core/auth/fake 50%
flamingo.me/flamingo/v3/core/auth/http 40%
flamingo.me/flamingo/v3/core/auth/oauth 49%
flamingo.me/flamingo/v3/core/cache 54%
flamingo.me/flamingo/v3/core/gotemplate 87%
flamingo.me/flamingo/v3/core/healthcheck/interfaces/controllers 76%
flamingo.me/flamingo/v3/core/healthcheck 66%
flamingo.me/flamingo/v3/core/internalauth/application 100%
flamingo.me/flamingo/v3/core/internalauth 100%
flamingo.me/flamingo/v3/core/locale/application 87%
flamingo.me/flamingo/v3/core/locale/domain 75%
flamingo.me/flamingo/v3/core/locale/interfaces/controllers 100%
flamingo.me/flamingo/v3/core/locale/interfaces/templatefunctions 92%
flamingo.me/flamingo/v3/core/locale 59%
flamingo.me/flamingo/v3/core/oauth/application 4%
flamingo.me/flamingo/v3/core/oauth/domain 62%
flamingo.me/flamingo/v3/core/oauth 55%
flamingo.me/flamingo/v3/core/requestlogger 100%
flamingo.me/flamingo/v3/core/requesttask 10%
flamingo.me/flamingo/v3/core/robotstxt 50%
flamingo.me/flamingo/v3/core/runtime 77%
flamingo.me/flamingo/v3/core/security/application/role 95%
flamingo.me/flamingo/v3/core/security/application 98%
flamingo.me/flamingo/v3/core/security/application/voter 100%
flamingo.me/flamingo/v3/core/security/interface/controller 100%
flamingo.me/flamingo/v3/core/security/interface/middleware 72%
flamingo.me/flamingo/v3/core/security 59%
flamingo.me/flamingo/v3/core/zap 43%
flamingo.me/flamingo/v3/framework/cmd 12%
flamingo.me/flamingo/v3/framework/config 49%
flamingo.me/flamingo/v3/framework/flamingo 40%
flamingo.me/flamingo/v3/framework 29%
flamingo.me/flamingo/v3/framework/opencensus 88%
flamingo.me/flamingo/v3/framework/prefixrouter 43%
flamingo.me/flamingo/v3/framework/systemendpoint 27%
flamingo.me/flamingo/v3/framework/testutil 0%
flamingo.me/flamingo/v3/framework/web 49%
Summary 49% (3339 / 6746)

@tessig tessig merged commit f977f5f into master Aug 3, 2023
5 checks passed
@tessig tessig deleted the feat/logger-memory branch August 3, 2023 13:43
@github-actions github-actions bot mentioned this pull request Aug 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants