Should metadata response errors trigger `connections:connect:error`?

[jsumners-nr]

index.js

const dc = require('node:diagnostics_channel')
const { Producer, stringSerializers } = require('@platformatic/kafka')

dc.subscribe('tracing:plt:kafka:connections:connects:error', (event) => {
  process._rawDebug('!!! error.event: ', event)
})

const producer = new Producer({
  clientId: 'test-producer',
  bootstrapBrokers: ['localhost:9092'],
  serializers: stringSerializers
})

main()
  .then(() => {
    console.log('done')
  })
  .catch((error) => {
    console.error('!!! promise.error: ', error)
  })
  .finally(async () => {
    await producer.close()
  })

async function main() {
  try {
    await producer.send({
      messages: [{
        topic: 'events',
        key: 'user-123',
        value: JSON.stringify({name: 'John', action: 'login'}),
        headers: {source: 'web-app'}
      }]
    })
  } catch (error) {
    console.error('!!! catch.error:', error)
  }
}

If you run that script against a Kafka host that is 1. actively listening on localhost:9092, 2. can serve good connects, and 3. does not have the defined events topic declared, then we'll hit the catch.error but not the error.event. Is this by design or by bug? Note, if you change the port from 9092 to some port that is not listening for anything, e.g. 13337, then the error.event will be hit.

In the case that the connection happens, but the target topic does not exist, the returned error is 4 instances of:

• msg: Received response with error while executing API Metadata(v12)
• cause: This server does not host this topic-partition.

If tracing:plt:kafka:connections:connects:error is not the right channel for these metadata errors, which one is?

[ShogunPanda]

Hi!

Yes, tracing:plt:kafka:connections:connects:error is specifically only for connection errors.
Side note promise.error: will never be triggered in your code since main can never throw.

The right channel is tracing:plt:kafka:base:metadata:error.

Anyway, I spotted a bug where the channel was not invoked as intended and sent a PR for it.

[jsumners-nr]

I'm kind of torn on this. I can see the argument that the error is post connection. But it seems a lot like a connection error in that a workable connection cannot be established. If anything, I'd expect the error to come in through producer:sends:error.

[ShogunPanda]

Unfortunately that's exactly the point: the connection has been established.
Given the current architecture, show in the producer:sends:error (or it's counterpart in the consumers) is not possible.

[jsumners-nr]

Unfortunately that's exactly the point: the connection has been established.

But not really. What is happening is:

1. producer.send is invoked
2. A connection is retried from the pool
3. Connection is established
4. producer.send is actually attempted
5. An error occurs because the connection is not usable

From an end user perspective, and what one would expect to trace, the .send failed because a usable connection couldn't be established.

[ShogunPanda]

No, the connection is usable.
If you do

const producer = new Producer(...)

try {
    // Send to a non existing topic
	await producer.send(...)
}catch(e) {
    // Send to an existing topic
    await producer.send(...)
}

You will see that the connection is only established once.
Or am I missing anything in your comment?

[jsumners-nr]

I understand that technically the connection is usable. I am saying that the technicality is irrelevant from a user perspective. The .send failed because the connection the .send tried establish for its work cannot be used to complete the work.

[ShogunPanda]

I see your point but, as I said, this cannot be done with the current architecture.