New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
all: replace log15 with slog #28187
all: replace log15 with slog #28187
Conversation
The diff already looks promising, do you have some benchmarks for the new logging lib? |
I did some benchmarks (https://github.com/ethereum/go-ethereum/blob/bf91275fc6baf3de3028458ad772a6de3dbabd20/log/slog/log/slog_test.go), but the code has since become outdated and I need to update them and add new benchmarks.
I think those can be improved on to make slog on par with our current logger in terms of performance. |
e943023
to
35bc591
Compare
Do you have any newer? Seems not possible to run those. Also, would be interesting to see the performance difference for logging on not-enabled levels, e.g. perf on |
Looking into it a bit, you have this curious construction where we still use the
Could you please add some information to the PR description about the design/implementation choices that you have made, and why? |
@holiman updated the description with an explanation. |
Regarding benchmarks:
This is the |
@holiman Are we okay with removing the alignment of provided attributes when using i.e. This is actually a bit of a hack to implement given the rigidity of It will make the output less human-readable. But I figure the |
I agree. Imo we don't require the alignment |
292b2a2
to
1cf2157
Compare
I still need to add proper formatting for |
Updated benchmarks from my local machine: This PR (source):
Current Geth Log15-based logger (source):
|
Yes.
I think so, yes... (?) |
... and we would be inhibited from adding these in the future because of breaking compatibility with the interface we are about to introduce with this PR right? |
More or less, yes. But afaict, if someone wants to use |
Deleted my incorrect comment. Yeah we don't need |
|
Oh right. yeah. |
The json equivalence tester is actually a bit annoying. The plan was to go line-by-line comparing test data with received output, and ignoring the time. And this would be simple enough because I can easily transform each line string into a However, in cases where user-provided keys clash with the keys used by record object built-in attributes, slog produces invalid json with duplicate keys. So writing the tester is a bit hackier (have to "decode" each line by extracting fields/values directly from the string). still straightforward enough I think. |
I think
It's not so invalid that the golang json parser refuses to parse it, so let's just do that. That's what a "consumer" of this json stream would do, so that's IMO the best way to test it |
func TestJsonLogging(t *testing.T) {
t.Parallel()
haveB, err := runSelf("--log.format", "json", "logtest")
if err != nil {
t.Fatal(err)
}
readFile, err := os.Open("testdata/logging/logtest-json.txt")
if err != nil {
t.Fatal(err)
}
wantLines := split(readFile)
haveLines := split(bytes.NewBuffer(haveB))
for i, want := range wantLines {
if i > len(haveLines)-1 {
t.Fatalf("format %v, line %d missing, want:%v", "json", i, want)
}
have := haveLines[i]
for strings.Contains(have, "Unknown config environment variable") {
// This can happen on CI runs. Drop it.
haveLines = append(haveLines[:i], haveLines[i+1:]...)
have = haveLines[i]
}
var h map[string]any
var w map[string]any
if err := json.Unmarshal([]byte(have), &h); err != nil {
t.Fatal(err)
}
if err := json.Unmarshal([]byte(want), &w); err != nil {
t.Fatal(err)
}
h["t"] = "xxx"
w["t"] = "xxx"
outh, _ := json.Marshal(h)
outw, _ := json.Marshal(w)
if !bytes.Equal(outh, outw) {
// show an intelligent diff
t.Logf(nicediff(outh, outw))
t.Errorf("file content wrong")
}
}
} Works for me ... ? |
From RFC 8529:
So I guess it's not strictly required and may deviate between software implementations. Okay, then checking that |
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
This PR replaces Geth's logger package (a fork of [log15](https://github.com/inconshreveable/log15)) with an implementation using slog, a logging library included as part of the Go standard library as of Go1.21. Main changes are as follows: * removes any log handlers that were unused in the Geth codebase. * Json, logfmt, and terminal formatters are now slog handlers. * Verbosity level constants are changed to match slog constant values. Internal translation is done to make this opaque to the user and backwards compatible with existing `--verbosity` and `--vmodule` options. * `--log.backtraceat` and `--log.debug` are removed. The external-facing API is largely the same as the existing Geth logger. Logger method signatures remain unchanged. A small semantic difference is that a `Handler` can only be set once per `Logger` and not changed dynamically. This just means that a new logger must be instantiated every time the handler of the root logger is changed. ---- For users of the `go-ethereum/log` module. If you were using this module for your own project, you will need to change the initialization. If you previously did ```golang log.Root().SetHandler(log.LvlFilterHandler(log.LvlInfo, log.StreamHandler(os.Stderr, log.TerminalFormat(true)))) ``` You now instead need to do ```golang log.SetDefault(log.NewLogger(log.NewTerminalHandlerWithLevel(os.Stderr, log.LevelInfo, true))) ``` See more about reasoning here: ethereum#28558 (comment)
This PR replaces Geth's logger package (a fork of [log15](https://github.com/inconshreveable/log15)) with an implementation using slog, a logging library included as part of the Go standard library as of Go1.21. Main changes are as follows: * removes any log handlers that were unused in the Geth codebase. * Json, logfmt, and terminal formatters are now slog handlers. * Verbosity level constants are changed to match slog constant values. Internal translation is done to make this opaque to the user and backwards compatible with existing `--verbosity` and `--vmodule` options. * `--log.backtraceat` and `--log.debug` are removed. The external-facing API is largely the same as the existing Geth logger. Logger method signatures remain unchanged. A small semantic difference is that a `Handler` can only be set once per `Logger` and not changed dynamically. This just means that a new logger must be instantiated every time the handler of the root logger is changed. ---- For users of the `go-ethereum/log` module. If you were using this module for your own project, you will need to change the initialization. If you previously did ```golang log.Root().SetHandler(log.LvlFilterHandler(log.LvlInfo, log.StreamHandler(os.Stderr, log.TerminalFormat(true)))) ``` You now instead need to do ```golang log.SetDefault(log.NewLogger(log.NewTerminalHandlerWithLevel(os.Stderr, log.LevelInfo, true))) ``` See more about reasoning here: ethereum#28558 (comment)
This PR replaces Geth's logger package (a fork of [log15](https://github.com/inconshreveable/log15)) with an implementation using slog, a logging library included as part of the Go standard library as of Go1.21. Main changes are as follows: * removes any log handlers that were unused in the Geth codebase. * Json, logfmt, and terminal formatters are now slog handlers. * Verbosity level constants are changed to match slog constant values. Internal translation is done to make this opaque to the user and backwards compatible with existing `--verbosity` and `--vmodule` options. * `--log.backtraceat` and `--log.debug` are removed. The external-facing API is largely the same as the existing Geth logger. Logger method signatures remain unchanged. A small semantic difference is that a `Handler` can only be set once per `Logger` and not changed dynamically. This just means that a new logger must be instantiated every time the handler of the root logger is changed. ---- For users of the `go-ethereum/log` module. If you were using this module for your own project, you will need to change the initialization. If you previously did ```golang log.Root().SetHandler(log.LvlFilterHandler(log.LvlInfo, log.StreamHandler(os.Stderr, log.TerminalFormat(true)))) ``` You now instead need to do ```golang log.SetDefault(log.NewLogger(log.NewTerminalHandlerWithLevel(os.Stderr, log.LevelInfo, true))) ``` See more about reasoning here: ethereum#28558 (comment)
the pr prevents all logs
Can we remove this line? Lines 14 to 15 in 09e0208
|
#28738 is about this problem |
This PR is a bit in preparation for the slog work in ethereum#28187 . Our current test re logging mostly test the internals, but we have no real end-to-end test of the logging output. This PR introduces a simple reexec-based log tester. This also relies upon a special mode in geth, which can be made to eject a set of predefined log messages (only available if the build-tag `integrationtests` is used e.g. go run --tags=integrationtests ./cmd/geth --log.format terminal logtest While working on this, I also noticed a quirk in the setup: when geth was configured to use a file output, then two separate handlers were used (one handler for the file, one handler for the console). Using two separate handlers means that two formatters are used, thus the formatting of any/all records happened twice. This PR changes the mechanism to use two separate io.Writers instead, which is both more optimal and fixes a bug which occurs due to a global statefulness in the formatter.
This PR replaces Geth's logger package (a fork of [log15](https://github.com/inconshreveable/log15)) with an implementation using slog, a logging library included as part of the Go standard library as of Go1.21. Main changes are as follows: * removes any log handlers that were unused in the Geth codebase. * Json, logfmt, and terminal formatters are now slog handlers. * Verbosity level constants are changed to match slog constant values. Internal translation is done to make this opaque to the user and backwards compatible with existing `--verbosity` and `--vmodule` options. * `--log.backtraceat` and `--log.debug` are removed. The external-facing API is largely the same as the existing Geth logger. Logger method signatures remain unchanged. A small semantic difference is that a `Handler` can only be set once per `Logger` and not changed dynamically. This just means that a new logger must be instantiated every time the handler of the root logger is changed. ---- For users of the `go-ethereum/log` module. If you were using this module for your own project, you will need to change the initialization. If you previously did ```golang log.Root().SetHandler(log.LvlFilterHandler(log.LvlInfo, log.StreamHandler(os.Stderr, log.TerminalFormat(true)))) ``` You now instead need to do ```golang log.SetDefault(log.NewLogger(log.NewTerminalHandlerWithLevel(os.Stderr, log.LevelInfo, true))) ``` See more about reasoning here: ethereum#28558 (comment)
This PR replaces Geth's logger package (a fork of log15) with an implementation using slog, a logging library included as part of the Go standard library as of Go1.21.
Main changes are as follows:
--verbosity
and--vmodule
options.--log.backtraceat
and--log.debug
are removed.The external-facing API is largely the same as the existing Geth logger. Logger method signatures remain unchanged.
A small semantic difference is that a
Handler
can only be set once perLogger
and not changed dynamically. This just means that a new logger must be instantiated every time the handler of the root logger is changed.For users of the
go-ethereum/log
module. If you were using this module for your own project, you will need to change the initialization. If you previously didYou now instead need to do
See more about reasoning here: #28558 (comment)