Ccompat api - cannot use protobuf from dotnet or Python

[tomikiss]

Hello,

I'm testing out Apicurio using apicurio/apicurio-registry-mem:2.2.2.Final docker image and testing confluent python and dotnet packages using protobuf. When I try to use proto schema I'm receiving status code 422.

It seems like there is an inconsistency how proto schemas represented between the confluent schema-registry and apicurio. After debugging the issue a bit more confluent packages use Protobuf base64 encoded format while apicurio expect the input being string and return "Failed to discover artifact type from content. (HTTP status code 422, SR code 42201)" error.

Python package:

• confluent-kafka 1.8.2

Endpoint used :
http://localhost:8080/apis/ccompat/v6
Failing request:
POST
http://localhost:8080/apis/ccompat/v6/subjects/test-value/versions
Request content:

'{"schema": "Cgp1c2VyLnByb3RvIiYKBFVzZXISEAoIdXNlck5hbWUYASABKAkSDAoEdGltZRgCIAEoCWIGcHJvdG8z", "schemaType": "PROTOBUF", "references": []}'

Example reproducer following confluent python example - I have a similar one for dotnet as well if that helps.

import proto.user_pb2
from confluent_kafka import SerializingProducer
from confluent_kafka.serialization import StringSerializer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.protobuf import ProtobufSerializer


def delivery_report(err, msg):
    if err is not None:
        print("Delivery failed for User record {}: {}".format(msg.key(), err))
        return
    print('User record {} successfully produced to {} [{}] at offset {}'.format(
        msg.key(), msg.topic(), msg.partition(), msg.offset()))


def send_message():
    topic = "test"
    schema_registry_conf = {'url': "http://localhost:8080/apis/ccompat/v6"}
    schema_registry_client = SchemaRegistryClient(schema_registry_conf)

    protobuf_serializer = ProtobufSerializer(proto.user_pb2.User,
                                             schema_registry_client,
                                             {'use.deprecated.format': False})

    producer_conf = {'bootstrap.servers': "localhost:9092",
                     'key.serializer': StringSerializer('utf_8'),
                     'value.serializer': protobuf_serializer}

    producer = SerializingProducer(producer_conf)

    print("Producing user records to topic {}. ^C to exit.".format(topic))

    # Serve on_delivery callbacks from previous calls to produce()
    producer.poll(0.0)
    try:
        user_name = "test"
        user = proto.user_pb2.User(userName=user_name)
        producer.produce(topic=topic, partition=0, key="1", value=user,
                             on_delivery=delivery_report)
    except ValueError:
        print("Invalid input, discarding record...")

    print("Flushing records...")
    producer.flush()
    print("Done")


if __name__ == '__main__':
    send_message()

user.proto:

syntax = "proto3";

message User {
	string userName = 1;
}

On python side this is where the conversion happening before sending the schema to the registry:

https://github.com/confluentinc/confluent-kafka-python/blob/e7e066120f05ca6f9c37b18d77f974f1a6cad222/src/confluent_kafka/schema_registry/protobuf.py#L290

The equivalent happening on dotnet side as well when sending protobuf schemas to the registry.

[carlesarnal]

First of all, thanks for raising this.

I have created a PR that will solve this issue. We were not handling the situation of encoded protobuf properly in our automated type discovery tool.

I have used your example request to test the fix and now everything should work as expected.

[tomikiss]

Thank you very much for this. Will test it out after merge with .net and python side as well.