CCompat Layer SchemaType with JSON

[christofluethi]

Hi there.

Related to #1660 we are currently we are having issues with JSON schema type and the confluent compatibility layer.

Confluent

Avro

# OK.
curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{\"type\": \"string\"}"}' http://localhost:8081/subjects/avro-schema-value/versions

# schemaType is not set (default=AVRO)
curl http://localhost:8081/subjects/avro-schema-value/versions/1
curl http://localhost:8081/schemas/ids/102

Json

# OK.
curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{\"type\":\"object\",\"properties\":{\"f1\":{\"type\":\"string\"}}}", "schemaType": "JSON"}' http://localhost:8081/subjects/json-schema-value/versions

# schemaType on both endpoints is set to JSON
curl http://localhost:8081/subjects/json-schema-value/versions/1
curl http://localhost:8081/schemas/ids/103

Apicurio Registry

Avro

# OK.
curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{\"type\": \"string\"}"}' http://localhost:8082/apis/ccompat/v6/subjects/avro-schema-value/versions

# schemaType is not set (default=AVRO; ok, same as confluent)
curl http://localhost:8082/apis/ccompat/v6/subjects/avro-schema-value/versions/1

# schemaType is set to AVRO (different from confluent)
curl http://localhost:8082/apis/ccompat/v6/schemas/ids/10

Json

# Fails with {"message":"Failed to discover artifact type from content.","error_code":0}
curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{\"type\":\"object\",\"properties\":{\"f1\":{\"type\":\"string\"}}}", "schemaType": "JSON"}' http://localhost:8082/apis/ccompat/v6/subjects/json-schema-value/versions

# Created a JSON schema in Apicurio UI and fetched via CCompat API
# Works but schemaType is not set and therefore defaults to AVRO which is wrong
curl http://localhost:8082/apis/ccompat/v6/subjects/json-ui-schema-value/versions/1

# Results in {"message":"Failed to discover artifact type from content.","error_code":0}
curl http://localhost:8082/apis/ccompat/v6/schemas/ids/12

From my perspective:

• Posting JSON schemas using the ccompat api is not working
• Fetching JSON schemas using the ccompat api /schemas/ids is not working
• Fetching JSON schemas using the ccompat api /subjects is missing the schemaType and therefore defaults to AVRO
• schemaType is returned for avro on the schemas endpoint (that is probably not really a problem but different to confluent).

We are using 2.1.0.Final and this is currently blocking us from moving forward.

[carlesarnal]

Hi @christofluethi. We fixed some of the problems mentioned above already and I'm working to get the other ones fixed as soon as possible. Do you mind sharing the jsonschema you used in your testing so I can confirm when it's working with the same example?

Thanks

[christofluethi]

Hi @carlesarnal. that sounds great.

You may find the tests i did in the curls above. That is the one which failed for me with a simple JSON-Schema:

curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{\"type\":\"object\",\"properties\":{\"f1\":{\"type\":\"string\"}}}", "schemaType": "JSON"}' http://localhost:8082/apis/ccompat/v6/subjects/json-schema-value/versions

[trombka]

Hi @carlesarnal

What was the reason for #1677? Even the title says something that is not compatible with CP.
There should be no schemaType field in the response for a schema query by id.
No such field is described here:
https://docs.confluent.io/platform/current/schema-registry/develop/api.html#get--schemas-ids-int-%20id
Couldn't the whole merge be just reverted?

[carlesarnal]

Hi,

Although AFAIK it's not documented, the API returns the schemaType when the registered schema is not AVRO. You can check the rest object being used here. And you have an example I created locally where I registered a json schema and the response contains the schemaType:

{
    "schemaType": "JSON",
    "schema": "{\"$id\":\"https://example.com/person.schema.json\",\"$schema\":\"https://json-schema.org/draft/2020-12/schema\",\"title\":\"Person\",\"type\":\"object\",\"properties\":{\"firstName\":{\"type\":\"string\",\"description\":\"The person's first name.\"},\"lastName\":{\"type\":\"string\",\"description\":\"The person's last name.\"},\"age\":{\"description\":\"Age in years which must be equal to or greater than zero.\",\"type\":\"integer\",\"minimum\":0}}}"
}

As is stated on this thread, there are some issues when working with json schemas that are being fixed right now, but the schemaType must be returned when working with either json schema or protobuf. If you still think there's an issue, please, open a new issue (since the question you're arising is not directly related to this one) and I will take a look.

Thanks!

[trombka]

My comment was related to this issue because the unexpected behaviour is described in the original post by @christofluethi.

# schemaType is set to AVRO (different from confluent)
curl http://localhost:8082/apis/ccompat/v6/schemas/ids/10

[carlesarnal]

Then your issue is that the schemaType is also returned for avro schemas? If that's the case, is a known issue and we're working to fix it (and others) but, as I said, the schemaType must be returned for json and protobuf. The issue with it being returned now with a wrong value is something we're aware of and working to fix it.

[trombka]

Yes.
I'm using a tool that requires a schema registry and I'm using Apicurio Registry as a drop-in replacement. However, I cannot use your latest release, because the tool, unfortunately, fails with an "unknown schemaType field" error.

Ok, I see your reason for the field for JSON and Protobuf which, you're right, is undocumented CP behaviour.

[carlesarnal]

@christofluethi these problems should've been fixed on the master branch so I'm closing this. Once you have some time, can you confirm it's working as expected for you? If there are any issues, feel free to re-open. Thanks