New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enhance logging #310
Enhance logging #310
Conversation
13f80b3
to
e39abed
Compare
9ffc201
to
c582bdd
Compare
2f2d35f
to
3e593fc
Compare
a113bf0
to
acaa913
Compare
7aa9670
to
a4d4752
Compare
doc/03-Configuration.md
Outdated
Option | Description | ||
-------------------------|----------------------------------------------- | ||
level | **Optional.** Logger default level (debug, info, warn, error, dpanic, panic or fatal). Usually set to `debug` by default. | ||
options | **Optional.** Child loggers, with `Icinga Db component` => `logging level` map.<br /> `Icinga Db component`: database, redis, heartbeat, high-availability, config-sync, history, runtime-updates, overdue-sync, dump-signals, delta. <br /> `logging level`: debug, info, warn, error, dpanic, panic or fatal. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would not write long text in tables, but create a separate section under logging and document the components with a small description of what they do there. The description for options
could then be something like this:
Optional. Map of component name to logging level in order to set a different logging level for each component instead of the default one. See logging components for details.
bf7e836
to
38aaf26
Compare
38aaf26
to
d4f9b94
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Works great and the code is also fine. Just needs those 3 requested changes and a rebase and it's ready to merge.
d4f9b94
to
ba4b9d9
Compare
The logger for each component in icingadb must be set to a level described in `logging` section in `config.yml. If config file doesnot describe the logger level for the component, then the level for the corresponding component is set to `info`.
Logger field is removed from Command struct in command.go The logging package has been completely changed to handle fatal logs. Method Logging.FatalF is added to introduce a buffer time adter the Fatal log has been displayed on the console and before the service shuts down.
logging.enabled switch is used to disable the child loggers along with default logger
`output` field is added to `logging` section in `config.yml` file. If it is set to `systemd` then logs are sent to systemd journal. By default it is set to `console`. `journal.go` is added to `logging` package in which the functionality to send the messages to systemd journal is written.
Get whether the log outputs need to be written to stdErr or sent to systemd journal using daemon.SdNotify(false, daemon.SdNotifyReady) and os.LookupEnv("NOTIFY_SOCKET"). And in icingadb.service when `Type` is set to `simple` then the output will be `console` and when it is set to `notify` the output will be `systemd-journa`. In this commit there are also changes to the code document.
70f414d
to
66081ae
Compare
…e.Level Unmarshaller is added to config.go to read sub fields which are set to default, if the main field is not present in config.yml file. For example logging has all optional fields and hence may not be included sometimes in config.yml file, in which case the default logging level is set to debug.
66081ae
to
65dbb03
Compare
Packaging PRs needed for this PR: |
|
||
Option | Description | ||
-------------------------|----------------------------------------------- | ||
level | **Optional.** Specifies the default logging level. Can be set to `fatal`, `error`, `warning`, `info` or `debug`. Defaults to `debug`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not default to a more production-ready default value?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense.
// UnmarshalYAML implements the yaml.Unmarshaler interface. | ||
func (c *Config) UnmarshalYAML(unmarshal func(interface{}) error) error { | ||
if err := defaults.Set(c); err != nil { | ||
return errors.Wrap(err, "can't set default config") | ||
} | ||
// Prevent recursion. | ||
type self Config | ||
if err := unmarshal((*self)(c)); err != nil { | ||
return internal.CantUnmarshalYAML(err, c) | ||
} | ||
|
||
return nil | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why introduce this again? Setting defaults should now be handled there:
Lines 47 to 49 in aaf4a22
if err := defaults.Set(c); err != nil { | |
return nil, errors.Wrap(err, "can't set config defaults") | |
} |
// UnmarshalYAML implements the yaml.Unmarshaler interface. | ||
func (l *Logging) UnmarshalYAML(unmarshal func(interface{}) error) error { | ||
if err := defaults.Set(l); err != nil { | ||
return errors.Wrap(err, "can't set default logging config") | ||
} | ||
// Prevent recursion. | ||
type self Logging | ||
if err := unmarshal((*self)(l)); err != nil { | ||
return internal.CantUnmarshalYAML(err, l) | ||
} | ||
|
||
return nil | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should also be handled by this:
Lines 47 to 49 in aaf4a22
if err := defaults.Set(c); err != nil { | |
return nil, errors.Wrap(err, "can't set config defaults") | |
} |
@@ -184,7 +215,7 @@ func run() int { | |||
com.ErrgroupReceive(g, dbErrs) | |||
|
|||
g.Go(func() error { | |||
return s.ApplyDelta(ctx, icingadb.NewDelta(ctx, actualCvs, cvs1, cv, logger)) | |||
return s.ApplyDelta(ctx, icingadb.NewDelta(ctx, actualCvs, cvs1, cv, logs.GetChildLogger("delta"))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The delta implementation is not worth it's own logging configuration and should be seen as part of a bigger component IMHO. There's also a call to ApplyDelta()
within the config sync which just uses the "config-sync"
child logger. So either also use a delta child logger there or make this one here a "customvar"
child logger (or maybe even also "config-sync"
if we don't see custom var sync as an independent component).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
config-sync
for those components you've mentioned sounds good to me.
@@ -199,7 +230,7 @@ func run() int { | |||
com.ErrgroupReceive(g, dbErrs) | |||
|
|||
g.Go(func() error { | |||
return s.ApplyDelta(ctx, icingadb.NewDelta(ctx, actualCvFlats, cvFlats, cvFlat, logger)) | |||
return s.ApplyDelta(ctx, icingadb.NewDelta(ctx, actualCvFlats, cvFlats, cvFlat, logs.GetChildLogger("delta"))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same here.
if _, ok := os.LookupEnv("NOTIFY_SOCKET"); ok { | ||
output = "systemd-journal" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Or is Type=notify
only there to detect whether running under systemd?
Possible alternative would be to use a combination of https://pkg.go.dev/github.com/coreos/go-systemd/v22@v22.3.2/journal#Enabled and https://pkg.go.dev/golang.org/x/term#IsTerminal
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, Type = notify
is only there to say if it runs under systemd. I would leave it that way, because the alternative can write to the journal if systemd is available and Icinga DB is started somehow differently.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, what about just doing ExecStart=/usr/bin/icingadb -log journald
? Would make it obvious why it logs to which destination.
if err := json.NewDecoder(bytes.NewReader(p)).Decode(line); err != nil { | ||
panic(err) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Letting zap serialize to JSON just to deserialize it again within the same process feels unnecessary. Isn't there a better way to do this?
Apart from that, the io.Writer
interface would allow writing partial JSON objects here, so depending on how zap
handles this (and if it may change this in the future), this could panic at random.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe there is no better way right now without implementing a custom encoder. CoreOS does the same thing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you have a link to where CoreOS does that? Also haven't looked to much into zap but what would be the problem with implementing an encoder?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return w.Writer.Write(message.Bytes()) | ||
} | ||
|
||
return 0, nil |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should return the number of bytes written: https://pkg.go.dev/io#Writer
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Again, just return len(p)
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes
err = journal.Send(message.String(), pri, map[string]string{ | ||
"PACKAGE": filepath.Dir(line.Caller), | ||
"SYSLOG_IDENTIFIER": filepath.Base(os.Args[0]), | ||
}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't journald already set SYSLOG_IDENTIFIER
by itself?
Also, to be honest, is that PACKAGE
extra field really worth the effort of implementing anything systemd-specific? The log/syslog package already provides a Writer that could just be plugged into zap and should give mostly the same effect as journald provides syslog compatibility.
In addition, when testing with NOTIFY_SOCKET= go run ./cmd/icingadb
, this even just logs PACKAGE=.
(haven't tried if this behaves differently when properly compiling and running as a systemd unit).
panic(err) | ||
} | ||
|
||
return w.Writer.Write(message.Bytes()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also here, most likely len(p) != len(message.Bytes())
, so incorrect implementation of io.Writer
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That is true. Should we just return len(p)
here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On success, yes, on error, maybe that, maybe 0. io.Writer
docs doesn't seem to forbid returning len(p)
with err != nil
, however it sound strange to say everything was written and return an error at the same time.
This PR was basically split in the following two ones. Is there anything in here that's not covered by those preventing this one from being closed? |
Nope, I'll close this one. |
Enhance logging by providing configuration for logging in configuration file
config.yml
.