New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix({kafka,pulsar}_producer): correctly handle metrics for connectors that have internal buffers #11724
fix({kafka,pulsar}_producer): correctly handle metrics for connectors that have internal buffers #11724
Conversation
f991615
to
52bc762
Compare
Pull Request Test Coverage Report for Build 6436313820
💛 - Coveralls |
a1c17c0
to
7c690a1
Compare
… internal buffers Fixes https://emqx.atlassian.net/browse/EMQX-11086 There’s currently a metric inconsistency due to the internal buffering nature of Kafka Producer (wolff). We use simple_sync_query to call the Kafka Producer bridge. If that times out, the call is accounted as failed, even though the message is buffered in wolff and later sent successfully.
…ridges with internal buffering Since authn/authz backends also use simple async/sync queries, we may want to avoid them calling the connector when it's not connected.
…or Pulsar Since it has internal buffering, it necessitates the same fix as Kafka producer.
c3bd534
to
d6781ef
Compare
there is no need to keep the |
It was removed here: https://github.com/emqx/emqx/pull/11722/files#diff-3803c5a6f7df225ee8e37ccda7a79158464e838d842bd5632f25e6356fffb7c2L309 But there's still a reference to it in |
targeting
release-53
Fixes https://emqx.atlassian.net/browse/EMQX-11086
There’s currently a metric inconsistency due to the internal buffering nature of Kafka Producer (wolff).
We use simple_sync_query to call the Kafka Producer bridge. If that times out, the call is accounted as failed, even though the message is buffered in wolff and later sent successfully.
Summary
🤖 Generated by Copilot at 887f0ad
This pull request adds a new query mode
simple_sync_internal_buffer
to theemqx_resource
application, which enables synchronous queries with internal buffering and retry logic for resource workers that support it. This mode is used by the Kafka bridge producer to improve message delivery reliability. The pull request also updates the type definitions, version number, and query function of theemqx_resource
application.PR Checklist
Please convert it to a draft if any of the following conditions are not met. Reviewers may skip over until all the items are checked:
changes/(ce|ee)/(feat|perf|fix)-<PR-id>.en.md
filesChecklist for CI (.github/workflows) changes
changes/
dir for user-facing artifacts update