Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix({kafka,pulsar}_producer): correctly handle metrics for connectors that have internal buffers #11724

Merged
merged 5 commits into from
Oct 10, 2023
Merged

fix({kafka,pulsar}_producer): correctly handle metrics for connectors that have internal buffers #11724

merged 5 commits into from
Oct 10, 2023

Commits on Oct 9, 2023

  1. fix(kafka_producer): correctly handle metrics for connector that have… 

    … internal buffers

Fixes https://emqx.atlassian.net/browse/EMQX-11086

There’s currently a metric inconsistency due to the internal buffering nature of Kafka
Producer (wolff).

We use simple_sync_query to call the Kafka Producer bridge.  If that times out, the call
is accounted as failed, even though the message is buffered in wolff and later sent
successfully.
    @thalesmg
    thalesmg committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    79cf0a2 View commit details
    Browse the repository at this point in the history

  2. fix(resource): create simple_async_internal_buffer query mode for b… 

    …ridges with internal buffering

Since authn/authz backends also use simple async/sync queries, we may want to avoid them
calling the connector when it's not connected.
    @thalesmg
    thalesmg committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    eebfb44 View commit details
    Browse the repository at this point in the history

  3. fix(pulsar_producer): use simple_async_internal_buffer query mode f… 

    …or Pulsar

Since it has internal buffering, it necessitates the same fix as Kafka producer.
    @thalesmg
    thalesmg committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    902b1d6 View commit details
    Browse the repository at this point in the history

  4. fix(resource): change how buffer workers are started

    @thalesmg
    thalesmg committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    d6781ef View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2023

  1. chore: remove mention of is_buffer_supported from typespec

    @thalesmg
    thalesmg committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    cf2075d View commit details
    Browse the repository at this point in the history