Returned config_type in DescribeConfigs is always 0

[weeco]

Version & Environment

Redpanda version: (use rpk version): v22.1.4
Kafka client: franz-go v1.6.0

What went wrong?

When describing configs (either broker or topic) the returned config_type is always zero (unknown). The reported API version for DescribeConfigs in the API versions response is v2. This property has been introduced in v3 of this request.

Version 2:

DescribeConfigs Response (Version: 2) => throttle_time_ms [results] 
  throttle_time_ms => INT32
  results => error_code error_message resource_type resource_name [configs] 
    error_code => INT16
    error_message => NULLABLE_STRING
    resource_type => INT8
    resource_name => STRING
    configs => name value read_only config_source is_sensitive [synonyms] 
      name => STRING
      value => NULLABLE_STRING
      read_only => BOOLEAN
      config_source => INT8
      is_sensitive => BOOLEAN
      synonyms => name value source 
        name => STRING
        value => NULLABLE_STRING
        source => INT8

Version 3:

A couple metadata properties have been added as per KIP-569

DescribeConfigs Response (Version: 3) => throttle_time_ms [results] 
  throttle_time_ms => INT32
  results => error_code error_message resource_type resource_name [configs] 
    error_code => INT16
    error_message => NULLABLE_STRING
    resource_type => INT8
    resource_name => STRING
    configs => name value read_only config_source is_sensitive [synonyms] config_type documentation 
      name => STRING
      value => NULLABLE_STRING
      read_only => BOOLEAN
      config_source => INT8
      is_sensitive => BOOLEAN
      synonyms => name value source 
        name => STRING
        value => NULLABLE_STRING
        source => INT8
      config_type => INT8
      documentation => NULLABLE_STRING

What should have happened instead?

How to reproduce the issue?

1. Send a DescribeConfigs request and check the response for the reported config_type (either topics or brokers is where I have observed this)

Additional information

Usecase is Kowl:

[dotnwat]

Thanks @weeco I'm surprised this field isn't filled it. But this should be an easy fix.

[peter-evans]

Took a look at this to see what might be involved to fix it. This is what I found.

In franz-go v1.4.0 it shows that the version of the Kafka API is expected to be at least 3 in order to have config property types included in the response.
https://github.com/twmb/franz-go/blob/v1.4.0/pkg/kmsg/generated.go#L22764-L22772

However, Redpanda only seems to support up to version 2 for the DescribeConfigs API:

redpanda/src/v/kafka/server/handlers/describe_configs.h

Line 17 in a4b097c

using describe_configs_handler = handler<describe_configs_api, 0, 2>;

I tested with kcl against a Redpanda cluster and sure enough, it negotiates with DescribeConfigs v2. So even though I've specified the --with-types flag, it can't show them because they don't exist in the response payload.

kcl admin configs describe -tb --with-types --log-level debug
[DEBUG] sharded request; destinations: [any]
[DEBUG] opening connection to broker; addr: 127.0.0.1:57421, broker: seed 0
[DEBUG] connection opened to broker; addr: 127.0.0.1:57421, broker: seed 0
[DEBUG] issuing api versions request; broker: seed 0, version: 3
[DEBUG] wrote ApiVersions v3; broker: seed 0, bytes_written: 30, write_wait: 399.442µs, time_to_write: 17.634µs, err: <nil>
[DEBUG] read ApiVersions v3; broker: seed 0, bytes_read: 180, read_wait: 138.576µs, time_to_read: 1.097272ms, err: <nil>
[DEBUG] connection initialized successfully; addr: 127.0.0.1:57421, broker: seed 0
[DEBUG] wrote DescribeConfigs v2; broker: seed 0, bytes_written: 29, write_wait: 4.449721ms, time_to_write: 8.021µs, err: <nil>
[DEBUG] read DescribeConfigs v2; broker: seed 0, bytes_read: 450, read_wait: 34.82µs, time_to_read: 1.642178ms, err: <nil>
advertised.listeners        internal://172.24.1.2:9092,external://127.0.0.1:57421  STATIC_BROKER_CONFIG
auto.create.topics.enable   true                                                   STATIC_BROKER_CONFIG
default.replication.factor  1                                                      DEFAULT_CONFIG
listeners                   internal://0.0.0.0:9092,external://172.24.1.2:9093     STATIC_BROKER_CONFIG
log.dirs                    /var/lib/redpanda/data                                 STATIC_BROKER_CONFIG
log.retention.bytes         18446744073709551615                                   DEFAULT_CONFIG
log.retention.ms            604800000                                              DEFAULT_CONFIG
log.segment.bytes           1073741824                                             DEFAULT_CONFIG
num.partitions              1                                                      DEFAULT_CONFIG

Assuming my understanding of this is correct, it looks like Redpanda would need to support version 3 which includes the property types in the response. I think this is the KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-569%3A+DescribeConfigsResponse+-+Update+the+schema+to+include+additional+metadata+information+of+the+field

[peter-evans]

Apologies, I realise now that I confused the property type and resource type the property is associated with. I've hidden my comment because it's unrelated to the issue at hand.

[weeco]

@peter-evans You were actually completely right, I used the wrong property name in my initial issue description. I have updated the issue description accordingly, thanks for looking at this too!

[peter-evans]

Ah, good to know! 😆

[Feediver1]

@graphcareful Is this customer-facing/does it require a doc update?

[piyushredpanda]

@NyaliaLui should comment on this as he is DRIing this feature.

[NyaliaLui]

@graphcareful Is this customer-facing/does it require a doc update?

@Feediver1 Yes this is customer-facing and it will require a doc update. After I resolve the ticket, there will be a UI/UX change on the rpk side since users can issue a DescribeConfigs request with rpk topic describe <topic name>.

I have not met with the RPK team yet about this but there will likely be two new rows in the output.

EDIT: I had a brief slack convo with @graphcareful. This change is not customer facing and it does not require a doc update.