Issues 1007 and 1017 are not yet fixed

[hudec]

Hi
Ii have tested the next realeses:

• apicurio-registry-streams:1.3.2.Final
• apicurio-registry-streams:1.3.x-snapshot
• apicurio-registry-streams: latest
  In all there's incompatibility with Confluent API
  curl -H "accept: application/json" --silent -k -X GET https://apicurio.rob.i.cz/api/ccompat/schemas/types
  {"message":"RESTEASY003635: No match for accept header","error_code":406}
  This works fine, but is incompatible
  curl --silent -k -X GET https://apicurio.rob.i.cz/api/ccompat/schemas/types
  ["JSON","PROTOBUF","AVRO"]
  Regards
  Vlado Hudec

[carlesarnal]

Hi @hudec,

This has been fixed in versions later than 1.3.2.Final but note that the streams storage has been removed in 2.0.0.Final. That's probably the reason why you're still seeing this.

[EricWittmann]

@hudec could you verify whether version 2.0.0.Final works for you?

[hudec]

Yes, of course
What is the correct URL for compatibility mode for apicurio/apicurio-registry-mem:2.0.0.Final?
k -n kafka1 exec -it apicurioregistry-d58b486cc-xwx67 -- sh
/ # curl --silent -X GET localhost:8080/api/ccompat/schemas/types
{"message":"RESTEASY003210: Could not find resource for full path: http://localhost:8080/apis/registry/v1/ccompat/schemas/types","error_code":404}/ #

[hudec]

I have discovered
curl --silent -X GET localhost:8080/apis/ccompat/v6/config
{"compatibilityLevel":"NONE"}/ # - what does it mean?

and
curl --silent -X GET localhost:8080/apis/ccompat/v6/schemas/types
["JSON","PROTOBUF","AVRO"]/ #

it seems it works

[hudec]

At the end - I can see schemas in AKHQ using URL localhost:8080/apis/ccompat/v6
I can't create a new schema using AKHQ nor KSQL.
For example

ksql> CREATE STREAM TEST01 (KEY_COL VARCHAR KEY, COL1 INT, COL2 VARCHAR)
>  WITH (KAFKA_TOPIC='test01', PARTITIONS=1, VALUE_FORMAT='AVRO');
Could not get latest schema for subject test01-value
Caused by: No artifact with ID 'test01-value' in group 'null' was found.; error
	code: 404

and in the KSQL log

[2021-06-03 12:00:28,057] INFO Processed unsuccessfully: KsqlRequest{ksql='CREATE STREAM TEST01 (KEY_COL VARCHAR KEY, COL1 INT, COL2 VARCHAR)
  WITH (KAFKA_TOPIC='test01', PARTITIONS=1, VALUE_FORMAT='AVRO');', configOverrides={}, requestProperties={}, sessionVariables={}, commandSequenceNumber=Optional[-1]}, reason:  (io.confluent.ksql.rest.server.resources.KsqlResource:317)
io.confluent.ksql.util.KsqlException: Could not get latest schema for subject test01-value
	at io.confluent.ksql.schema.registry.SchemaRegistryUtil.getLatestSchema(SchemaRegistryUtil.java:107)
	at io.confluent.ksql.schema.registry.SchemaRegistryUtil.subjectExists(SchemaRegistryUtil.java:83)
	at io.confluent.ksql.schema.ksql.inference.SchemaRegisterInjector.registerSchema(SchemaRegisterInjector.java:208)
	at io.confluent.ksql.schema.ksql.inference.SchemaRegisterInjector.registerSchemas(SchemaRegisterInjector.java:168)
	at io.confluent.ksql.schema.ksql.inference.SchemaRegisterInjector.registerForCreateSource(SchemaRegisterInjector.java:103)
	at io.confluent.ksql.schema.ksql.inference.SchemaRegisterInjector.inject(SchemaRegisterInjector.java:75)
	at io.confluent.ksql.statement.InjectorChain.inject(InjectorChain.java:42)
	at io.confluent.ksql.statement.InjectorChain.inject(InjectorChain.java:42)
	at io.confluent.ksql.rest.server.validation.RequestValidator.validate(RequestValidator.java:163)
	at io.confluent.ksql.rest.server.validation.RequestValidator.validate(RequestValidator.java:130)
	at io.confluent.ksql.rest.server.resources.KsqlResource.handleKsqlStatements(KsqlResource.java:280)
	at io.confluent.ksql.rest.server.KsqlServerEndpoints.lambda$executeKsqlRequest$2(KsqlServerEndpoints.java:190)
	at io.confluent.ksql.rest.server.KsqlServerEndpoints.lambda$executeOldApiEndpointOnWorker$23(KsqlServerEndpoints.java:341)
	at io.confluent.ksql.rest.server.KsqlServerEndpoints.lambda$executeOnWorker$22(KsqlServerEndpoints.java:327)
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: No artifact with ID 'test01-value' in group 'null' was found.; error code: 404
	at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:295)
	at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:365)
	at io.confluent.kafka.schemaregistry.client.rest.RestService.getLatestVersion(RestService.java:849)
	at io.confluent.kafka.schemaregistry.client.rest.RestService.getLatestVersion(RestService.java:840)
	at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getLatestSchemaMetadata(CachedSchemaRegistryClient.java:362)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at io.confluent.ksql.util.LimitedProxyBuilder.lambda$buildForwader$7(LimitedProxyBuilder.java:230)
	at io.confluent.ksql.util.LimitedProxyBuilder$SandboxProxy.invoke(LimitedProxyBuilder.java:312)
	at com.sun.proxy.$Proxy38.getLatestSchemaMetadata(Unknown Source)
	at io.confluent.ksql.schema.registry.SchemaRegistryUtil.getLatestSchema(SchemaRegistryUtil.java:101)
	... 18 more

[EricWittmann]

@hudec thanks for testing and providing feedback! The error messages are consistent with those clients trying to consume schemas that don't exist. Do those clients normally create the schema in the registry first? If so then I would expect it to work.

@famartinrh or @carlesarnal Any chance you have a few minutes to look at this?

[famarting]

@hudec based on your feedback I think the issue #1007 has been fixed

Now the error you are reporting when using AKHQ or KSQL, looks like a new thing. To not mix things, could you please open a new issue with some more detailed information of your new issue? Thank you :)

Just to clarify, the error No artifact with ID 'test01-value' in group 'null' was found.; error code: 404 should be normal if you know what you are doing. I don't know the details of what entails doing CREATE STREAM TEST01 (KEY_COL VARCHAR KEY, COL1 INT, COL2 VARCHAR) WITH (KAFKA_TOPIC='test01', PARTITIONS=1, VALUE_FORMAT='AVRO') , but I'm guessing you have to create first an artifact in the registry first

[hudec]

I think, the problem is incompatibility between Apicurio and Confluent.
The problem is, which return code is used in the case the resource is not found.
Ant the consequence is, if AKHW or KSQL are checking registry before the create
nee schema, they failed
For comparison

curl -k -D - https://apicurio.rob.i.cz/apis/ccompat/v6/subjects/nvlog-3-value/versions
HTTP/1.1 404 Not Found
Date: Tue, 08 Jun 2021 10:51:38 GMT
Content-Type: application/json
Content-Length: 93
Connection: keep-alive
Expires: Mon, 07 Jun 2021 10:51:38 GMT
Pragma: no-cache
Cache-control: no-cache, no-store, must-revalidate
Strict-Transport-Security: max-age=15724800; includeSubDomains

{"message":"No artifact with ID 'nvlog-3-value' in group 'null' was found.","error_code":404}

and

-k -D - https://schemaregistry.rob.i.cz/subjects/nvlog-3-value/versions
HTTP/1.1 404 Not Found
Date: Tue, 08 Jun 2021 10:52:14 GMT
Content-Type: application/vnd.schemaregistry.v1+json
Content-Length: 67
Connection: keep-alive
Strict-Transport-Security: max-age=15724800; includeSubDomains

{"error_code":40401,"message":"Subject 'nvlog-3-value' not found."}

[famarting]

interesting, @carlesarnal is possible confluent api changed error codes?

[carlesarnal]

No, I don't think they've changed the behaviour. There's probably a mismatch in the error codes being returned by the registry in the error payload, we're returning (I think always) 404 and confluent returns 40401, 40402, 40403 and so on depending on the missing resource. It is not a big deal in regular use cases but it is of course a problem when things like ksql enter the table.

That said, I'm not sure if fixing the error_code issue will fix your problem or not (it depends on how ksql handles error codes) but it's something we need to solve.

@hudec as @famartinrh, this worth a new issue since it is a completely different problem. I will create it and cc you there.