Not omit error in logger #1096
Not omit error in logger #1096
Conversation
|
Our goal here is to reduce alert noise for our alerts which refer to logged errors. A lot of errors we see are transient and can be treated with lower criticality in our case. As of now, we have no way of differentiating between errors we care about and those transient ones. To get a better handle on the different kinds of errors without hacky string matching, we would like to be able to check and filter via the actual error types in our middleware. This way, we can more cleanly filter and decide what to bubble up and what to treat as intermittent failure. |
|
@nlsun @achille-roussel |
|
@rhansen2 |
reader.go
Outdated
| @@ -1346,7 +1346,7 @@ func (r *reader) run(ctx context.Context, offset int64) { | |||
|
|
|||
| case errors.Is(err, UnknownTopicOrPartition): | |||
| r.withErrorLogger(func(log Logger) { | |||
| log.Printf("failed to read from current broker for partition %d of %s at offset %d, topic or parition not found on this broker, %v", r.partition, r.topic, toHumanOffset(offset), r.brokers) | |||
| log.Printf("failed to read from current broker for partition %d of %s at offset %d, topic or parition not found on this broker, %v: %v", r.partition, r.topic, toHumanOffset(offset), r.brokers, err) | |||
There was a problem hiding this comment.
It looks like with these new formats we'll have
failed to read from current broker for partition %d of %s at offset %d, topic or parition not found on this broker, %v: Unknown Topic Or Partition
Could we possible reformat these a bit so we don't log redundant messages?
Potentially like
failed to read from current broker for partition %d of %s at offset %d, Unknown Topic Or Partition, %v
There was a problem hiding this comment.
Changed the messages in logs
itechdima
force-pushed
the
add-error-value-into-logs
branch
from
8c94a74
to
ce68d4d
Compare
Add explicit errors as arguments so anyone can handle them in the logger implementation.
itechdima
force-pushed
the
add-error-value-into-logs
branch
from
ce68d4d
to
914d863
Compare
Add explicit errors as arguments so anyone can handle them in the logger implementation.
We stumbled across issue that during Kafka rebalancing we have plenty of different error logs (EOF, Unexpected EOF, not a leader, etc.). The only possible way to handle that noise is to add some middleware logic into our implementation of logger. So, to have better backward compatibility it's better to not check against string values, but against errors explicitly.