-
Notifications
You must be signed in to change notification settings - Fork 177
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enhancement : Add "discard_kafka_delivery_failed_regex" option #496
Enhancement : Add "discard_kafka_delivery_failed_regex" option #496
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems nice, thanks for your proposal!
To merge this, could you address following things?
- Could you follow DCO?: https://github.com/fluent/fluent-plugin-kafka/pull/496/checks?check_run_id=17580570217
- Could you add a description about this parameter to README.md?
- (It would be nice if we can add a test for it, but not mandatory for now.)
As a future task for us, it would be nice if we can match errors by error codes of Rdkafka to avoid ambiguity: https://docs.confluent.io/platform/current/clients/librdkafka/html/rdkafkacpp_8h.html#a4c6b7af48c215724c323c60ea4080dbf
but it's not needed in this pull request.
…failed events by regexp Signed-off-by: kubotat <tkubota@ctc-america.com>
…failed events by regexp Signed-off-by: kubotat <tkubota@ctc-america.com>
Signed-off-by: kubotat <tkubota@ctc-america.com>
a94c27f
to
3a9191c
Compare
@ashie Thanks for reviewing. I've done two items you mentioned. I will keep working on a test code.
It seems some of the check processes were failed after updating the contents. I would appreciate it if you could give me an advice on how to pass the check process. |
Don't worry, they aren't caused by this change.
Please let me know if it's hard to add. I'll merge & release it without test for now. |
@ashie Thanks. I would appreciate it if could merge my request. |
Thanks! |
I've released v0.19.2 |
When with Forwarder and Aggregator architecture, forwarders sometimes send invalid data which cause delivery failure at aggregators.
out_rdkafka2
plugin hasdiscard_kafka_delivery_failed
but it potentially discards not only unnecessary events but also required events, e.g. users expect Fluentd to keep events in buffer files during the outage of Kafka cluster (due to maintenance for instance) but all events are discarded when withdiscard_kafka_delivery_failed
.This enhancement request proposes having
discard_kafka_delivery_failed_regex
option onout_rdkafka2
plugin to nullify invalid data by checking the error message with given regexp pattern.Here is a sample use case:
In the following configuration, dummy events are generated with tag
test-topic0001
and Fluentd try to ship message totest-topic0001
. Iftest-topic0001
does not exist in Kafka cluster,out_rdkafka2
emits aLocal: Unknown topic (unknown_topic)
warning message. Withdiscard_kafka_delivery_failed_regex /Unknown topic/
,out_rdkafka2
discards events which emits aLocal: Unknown topic (unknown_topic)
warning message.