Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Producer failover and promise reject when a broker is disconnected #1686

Open
byrne-greg opened this issue May 20, 2024 · 0 comments
Open

Producer failover and promise reject when a broker is disconnected #1686

byrne-greg opened this issue May 20, 2024 · 0 comments

Comments

@byrne-greg
Copy link
Contributor

Currently running an app as HTTP server on node that emits events on requests. On startup of the app, we connect the producer; and when a request comes in, we emit a message through producer.send.

When one of the kafka brokers are turned off/on during a maintenance window, the application will start reporting issues when emitting events and then throw an unhandled error uncaught by any error handler. Can't see where the error bubbles up from kafkajs and its leading the application crashing.

/Users/myapp/node_modules/kafkajs/src/network/requestQueue/socketRequest.js:108
    this.rejected(new KafkaJSRequestTimeoutError(`Request ${requestInfo} timed out`, eventData))
                  ^
KafkaJSNonRetriableError
  Caused by: KafkaJSRequestTimeoutError: Request Produce(key: 0, version: 7) timed out
    at SocketRequest.timeoutRequest (/Users/myapp/node_modules/kafkajs/src/network/requestQueue/socketRequest.js:108:19)
    ... 4 lines matching cause stack trace ...
    at processTimers (node:internal/timers:512:7) {
  name: 'KafkaJSNumberOfRetriesExceeded',
  retriable: false,
  helpUrl: undefined,
  retryCount: 3,
  retryTime: 596,
  [cause]: KafkaJSRequestTimeoutError: Request Produce(key: 0, version: 7) timed out
      at SocketRequest.timeoutRequest (/Users/myapp/node_modules/kafkajs/src/network/requestQueue/socketRequest.js:108:19)
      at /Users/myapp/node_modules/kafkajs/src/network/requestQueue/index.js:100:21
      at Map.forEach (<anonymous>)
      at Timeout._onTimeout (/Users/myapp/node_modules/kafkajs/src/network/requestQueue/index.js:94:23)
      at listOnTimeout (node:internal/timers:569:17)
      at processTimers (node:internal/timers:512:7) {
    retriable: true,
    helpUrl: undefined,
    broker: 'broker-3.com:9097',
    correlationId: 7,
    createdAt: 1716207461754,
    sentAt: 1716207461755,
    pendingDuration: 1,
    [cause]: undefined
  }
}

What is the correct way to handle the producer in these cases? Ideally i'd prefer the HTTP server stay alive even if we lose some events and some hook that I could restart the producer in the interim. I didn't expect the error to surface from a place that doesn't look like i can wrap with an error handler

@byrne-greg byrne-greg changed the title Producer failover when broker is disconnected Producer failover and promise reject when a broker is disconnected May 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant