Predetermine supported API versions before making requests

[peter-evans]

Thank you for this excellent library 🥇

I'm working on a small client to handle some admin requests and I'm trying to get my head around how to handle different Kafka API versions for the features I would like to support. Take for example the command you have to alter config in kcl here. I notice it requires the user to opt-in with --inc for use of the incremental method introduced in Kafka 2.3.0.

My question is, is there a way to predetermine what API version the broker(s) support and in a client automatically chose to issue a IncrementalAlterConfigsRequest or AlterConfigsRequest?

[twmb]

There is, but it's a pretty big risk to have any fallback to the old AlterConfigsRequest.

What you could do is to send ApiVersionsRequest, pass the response to kversion.FromApiVersionsResponse, and then check HasKey(<request key>).

If the response does not have the incremental key, then you could fallback to non-incremental.

The big risk is that AlterConfigs does not preserve any prior configuration, and even if you DescribeConfigs beforehand to try to preserve configuration, DescribeConfigs does not return secrets. Secrets cannot be preserved with AlterConfigs -- you may have seen this in the kcl repo where I have a prompt for configuration that will be lost.

[peter-evans]

Thank you! I'll take a look at using ApiVersionsRequest as you suggested.

I understand your explanation of the risk and I'll make sure it's clear to the user if my client is going to do something that might be destructive. 👍

[twmb]

One other small risk is if your ApiVersions request goes to a new broker in a cluster that supports incremental update, while old brokers that do not are still in the cluster. This gets really tricky to solve correctly and IMO it's not worth it; Kafka authors have seen the problem here and are actually introducing a new "supported features" bit in MetadataResponse, but that's very new and they return nothing in it yet.

Happy to have helped, and happy you're checking out this library!

[peter-evans]

One other small risk is if your ApiVersions request goes to a new broker in a cluster that supports incremental update, while old brokers that do not are still in the cluster.

In this situation, if the client chose to use incremental update and the request went to a broker that didn't support the feature, do you know off-hand what would happen? Would it return one of these errors, perhaps INVALID_REQUEST?

[twmb]

Kafka actually closes the connection, and the client has no knowledge of why. The client will retry, if it retries enough it should hit the broker that supported the request anyway. But, if you're trying to modify configs on the broker that doesn't support the request, the connection will be closed a bunch and then the client will hit the request retry limit.

[peter-evans]

Um, sounds complicated. I can see why you think it's not worth it!

Thanks for the info. Very helpful.