Invalid SSL configs breaking connector

[dhuang]

We're seeing the following error when booting up our connector that uses the SnowflakeAvroConverter with the schema registry enabled (and accessible):

com.snowflake.kafka.connector.internal.SnowflakeKafkaConnectorException: 
[SF_KAFKA_CONNECTOR] Exception: Failed to connect schema registry service 
[SF_KAFKA_CONNECTOR] Error Code: 0012 
[SF_KAFKA_CONNECTOR] Detail: Schema registry service is not available 
[SF_KAFKA_CONNECTOR] Message: Invalid value for configuration schema.registry.ssl.engine.factory.class: Class could not be found. 
[SF_KAFKA_CONNECTOR] io.confluent.common.config.ConfigDef.parseType(ConfigDef.java:392) 
[SF_KAFKA_CONNECTOR] io.confluent.common.config.ConfigDef.define(ConfigDef.java:110) 
[SF_KAFKA_CONNECTOR] io.confluent.common.config.ConfigDef.define(ConfigDef.java:154) 
[SF_KAFKA_CONNECTOR] io.confluent.kafka.schemaregistry.client.SchemaRegistryClientConfig.withClientSslSupport(SchemaRegistryClientConfig.java:44) 
[SF_KAFKA_CONNECTOR] io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig.baseConfigDef(AbstractKafkaAvroSerDeConfig.java:155) 
[SF_KAFKA_CONNECTOR] com.snowflake.kafka.connector.records.AvroConverterConfig.<init>(AvroConverterConfig.java:27) 
[SF_KAFKA_CONNECTOR] com.snowflake.kafka.connector.records.SnowflakeAvroConverter.configure(SnowflakeAvroConverter.java:56) 
[SF_KAFKA_CONNECTOR] org.apache.kafka.connect.runtime.isolation.Plugins.newConverter(Plugins.java:298) 
[SF_KAFKA_CONNECTOR] org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:532) 
[SF_KAFKA_CONNECTOR] org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1251) 
[SF_KAFKA_CONNECTOR] org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1700(DistributedHerder.java:127) 
[SF_KAFKA_CONNECTOR] org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1266) 
[SF_KAFKA_CONNECTOR] org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1262) 
[SF_KAFKA_CONNECTOR] java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 
[SF_KAFKA_CONNECTOR] java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
[SF_KAFKA_CONNECTOR] java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
[SF_KAFKA_CONNECTOR] java.base/java.lang.Thread.run(Thread.java:834) at 
com.snowflake.kafka.connector.internal.SnowflakeErrors.getException(SnowflakeErrors.java:407) at 
com.snowflake.kafka.connector.internal.SnowflakeErrors.getException(SnowflakeErrors.java:383) at 
com.snowflake.kafka.connector.internal.SnowflakeErrors.getException(SnowflakeErrors.java:370) at 
com.snowflake.kafka.connector.records.SnowflakeAvroConverter.configure(SnowflakeAvroConverter.java:65) at 
org.apache.kafka.connect.runtime.isolation.Plugins.newConverter(Plugins.java:298) at 
org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:532) at 
org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1251) at 
org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1700(DistributedHerder.java:127) at 
org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1266) at 
org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1262) at 
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at 
java.base/java.lang.Thread.run(Thread.java:834)

This only showed up after upgrading the connector from 1.2.3 to 1.4.4. We're still investigating, but currently suspect confluentinc/schema-registry#957 and maybe confluentinc/common#241 are related, which are changes that would've been pulled in as part of the upgrade since the dependent confluent version was also bumped from 5.3.0 to 5.4.0.

We are not actually using SSL for the schema registry, but the default values seem to be an issue. This has been observed on both Confluent 5.4.0 and 5.5.1 for Kafka Connect and Schema Registry, along with Kafka 2.2.1 and 2.5.0. This may be more of an upstream issue, but I was wondering if anyone else has seen this?

Also confluentinc/schema-registry#1576 and apache/kafka#8338 are not in the versions we're using, but may be relevant in the future.

[dhuang]

We ended up getting around this by building the connector ourselves with Confluent 5.5.1, which unintentionally removed the new SSL configs in a refactor starting in 5.5.0 (confluentinc/schema-registry#1280). However, I expect this will break again in a future release as it has since been re-added (confluentinc/schema-registry#1577).

[ivanyu]

We're having this issue as well.

schema.registry.ssl.engine.factory.class was introduced to Kafka by https://issues.apache.org/jira/browse/KAFKA-8890, that is Kafka 2.6+ only should be affected.

[sfc-gh-zli]

Are you using SSL in your Kafka brokers? I cannot reproduce this with non-SSL tests.

[dhuang]

Yes we are using SSL, I guess not terribly surprising it doesn't occur for non-SSL.