Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

transport: mute errors when too verbose #249

Merged
merged 4 commits into from
Jan 6, 2024
Merged

transport: mute errors when too verbose #249

merged 4 commits into from
Jan 6, 2024

Conversation

lspgn
Copy link
Member

@lspgn lspgn commented Dec 6, 2023

This adds two CLI arguments that will by default reduce the amount of logs if there is an issue (transport like kafka logs or decoder templates).

Arguments:

  • err.int: interval to log errors
  • err.cnt: threshold of errors per interval

If the threshold of errors during an interval is crossed, subsequent errors will be muted until the interval clears.

One of the common case is when Kafka is down:

ERRO[0043] transport error                               error="kafka transport kafka: Failed to produce message to topic flow-messages: circuit breaker is open"
WARN[0043] too many transport errors, muting
WARN[0052] skipped 85 transport errors
ERRO[0052] transport error                               error="kafka transport kafka: Failed to produce message to topic flow-messages: circuit breaker is open"
ERRO[0052] transport error                               error="kafka transport kafka: Failed to produce message to topic flow-messages: circuit breaker is open"
ERRO[0052] transport error                               error="kafka transport kafka: Failed to produce message to topic flow-messages: circuit breaker is open"

Will require updating #199

@lspgn lspgn added the transport Related to transport libraries (Kafka) label Dec 6, 2023
@lspgn lspgn changed the title transport: mute when too verbose transport: mute errors when too verbose Dec 6, 2023
@lspgn
Copy link
Member Author

lspgn commented Dec 6, 2023

Idea: add "skipped x error messages" when counter is reset

@lspgn lspgn added the enhancement New feature or request label Dec 6, 2023
@lspgn lspgn merged commit c38607d into main Jan 6, 2024
1 check passed
@lspgn lspgn deleted the transport/mute branch January 6, 2024 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request transport Related to transport libraries (Kafka)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant