message.key is null in eachMessage implementation

[johesoman]

Describe the bug
The message provided to my eachMessage function has .key = null sometimes, even though the type KafkaMessage type definition states that .key: Buffer and not .key: Buffer | null.

To Reproduce
My implementation is part of a closed source project and I cannot share details. But the implementation of eachMessage is just something like:

eachMessage: async ({ message }) => {
  if (messagePayload.value) {
    const key = message.key.toString("utf-8");
    const value = message.value?.toString("utf-8");
    
    ...
  }
}

Expected behavior
Since the KafkaMessage type definition states that .key: Buffer, I was expecting it to not be null.

Observed behavior
.key has been consistently null for a subset of the consumed messages. Here's the output from the KafkaJS logger when the line message.key.toSring("utf-8") triggered the error:

2021-03-02T13:59:40.372Z	f1c814a0-38e6-44ec-a54f-d23203552cd7	ERROR	{"level":"ERROR","timestamp":"2021-03-02T13:59:40.372Z","logger":"kafkajs","message":"[Consumer] Crash: KafkaJSNumberOfRetriesExceeded: Cannot read property 'toString' of null","groupId":*****, "retryCount":0,"stack":"KafkaJSNonRetriableError\n  Caused by: TypeError: Cannot read property 'toString' of null\n    at Runner.eachMessage *****\n    at Runner.processEachMessage (/var/task/node_modules/kafkajs/src/consumer/runner.js:187:20)\n    at onBatch (/var/task/node_modules/kafkajs/src/consumer/runner.js:323:20)\n    at /var/task/node_modules/kafkajs/src/consumer/runner.js:375:21\n    at invoke (/var/task/node_modules/kafkajs/src/utils/concurrency.js:38:5)\n    at push (/var/task/node_modules/kafkajs/src/utils/concurrency.js:51:7)\n    at /var/task/node_modules/kafkajs/src/utils/concurrency.js:60:53\n    at new Promise (<anonymous>)\n    at /var/task/node_modules/kafkajs/src/utils/concurrency.js:60:20\n    at /var/task/node_modules/kafkajs/src/consumer/runner.js:365:11"}

Environment:

• OS: AWS Lambda
• KafkaJS version: ^1.13.0
• NodeJS version: 12 something

Additional context
Add any other context about the problem here.

[Nevon]

Right, that must be a mistake in the typings. If the message is produced without a key, it'll be null when you consume it.

[johesoman]

That makes sense! Thanks for your quick response and action :)

[Nevon]

This is fixed in 1.16.0-beta.10.