Add protobuf schema registry #10839

bananaaggle · 2021-02-03T04:11:42Z

Confluent's Schema Registry supports for deserializing protobuf now. So I add this feature to druid-protobuf-extension module. This module only work when user provide a .desc file in old version. Now it can get schema from schema registry dynamically.
This submission adds the io.confluent:kafka-schema-registry-client:6.0.1/io.confluent:kafka-protobuf-provider:6.0.1 dependency to the druid-protobuf-extension module.
Old spec is:

"parser": {
      "type": "protobuf",
      "descriptor": "file:///tmp/metrics.desc",
      "protoMessageType": "Metrics"
      ......
}

New spec is:

"parser": {
      "type": "protobuf",
      "protoBytesDecoder": {
           "type": "schema_registry",
          "url": "http://XXXXXX:8081",
          "capacity": 10,
        }
      ......
}

and

"parser": {
      "type": "protobuf",
      "protoBytesDecoder": {
          "type": "file",
          "descriptor": "file:///tmp/metrics.desc",
          "protoMessageType": "Metrics"
        }
      ......
}

Description

Fixed the bug ...

Renamed the class ...

Added a forbidden-apis entry ...

This PR has:

been self-reviewed.
- using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
added documentation for new or modified features or behaviors.
added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
added or updated version, license, or notice information in licenses.yaml
added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
added integration tests.
been tested in a test Druid cluster.

Key changed/added classes in this PR

MyFoo
OurBar
TheirBaz

clintropolis · 2021-02-03T04:46:56Z

Hi @bananaaggle, thanks for the contribution it seems like a useful addition and I would like to review it, but it is impossible if you keep closing and opening as a new PR. Could you please just stick to one PR and do any updates in place so that we can review it?

bananaaggle · 2021-02-03T05:03:49Z

Hi @bananaaggle, thanks for the contribution it seems like a useful addition and I would like to review it, but it is impossible if you keep closing and opening as a new PR. Could you please just stick to one PR and do any updates in place so that we can review it?

Sorry, I will stick to one PR.

clintropolis

overall lgtm

clintropolis · 2021-02-05T11:00:44Z

distribution/bin/check-licenses.py

@@ -225,6 +225,9 @@ def build_compatible_license_names():
    compatible_licenses['Apache License v2.0'] = 'Apache License version 2.0'
    compatible_licenses['Apache License, version 2.0'] = 'Apache License version 2.0'
    compatible_licenses['Apache 2.0 License'] = 'Apache License version 2.0'
+    compatible_licenses['Apache License, 2.0'] = 'Apache License version 2.0'
+    compatible_licenses['Confluent Community License'] = 'Confluent Community License'


hmm, what requires the confluent community license? I don't see anything in licenses.yaml mentioning it. Last i knew, the clients to confluent things were still apache licensed (i'm not sure we can distribute anything with this license in Druid packages, would have to check into that, but I suspect not since it has field of use restrictions)

This is dependency report generated by druid. I think it means kafka-protobuf-provider is licensed by Confluent Community License and Apache License. I change this python script because it can not work if this map lose key 'Confluent Community License'.

In MVN Repository, it marked as Apache license. So can I use it in Druid packages? I'm not very sure about that.

I looked at the source files for kafka-protobuf-provider here, https://github.com/confluentinc/schema-registry/tree/master/protobuf-provider/src/main/java/io/confluent/kafka/schemaregistry/protobuf

and there appears to be a mix of Apache and Confluent Community License headers, which I assume it means both licenses apply.

Unfortunately, I think this means that we cannot distribute this jar, but I think that instead we can provide instructions in the documentation on how users can download it (see https://www.apache.org/legal/resolved.html#optional), similar to how we do with the MySQL connector.

The pom.xml will need to mark this jar as provided, and you will need to remove this license from this python script, and remove kafka-protobuf-provider from license.yaml. I think maybe it is ok if tests still depend on it, since it wouldn't cause packaging to distribute it? but i'm unsure exactly...

clintropolis · 2021-02-05T11:07:51Z

...uf-extensions/src/main/java/org/apache/druid/data/input/protobuf/ProtobufInputRowParser.java

  private Parser<String, Object> parser;
  private final List<String> dimensions;

  @JsonCreator
  public ProtobufInputRowParser(
      @JsonProperty("parseSpec") ParseSpec parseSpec,
-      @JsonProperty("descriptor") String descriptorFilePath,
-      @JsonProperty("protoMessageType") String protoMessageType
+      @JsonProperty("protoBytesDecoder") ProtobufBytesDecoder protobufBytesDecoder


So that this change is not incompatible with previous druid versions, I think we should probably consider leaving the old constructor properties in place and marking them as deprecated instead of removing them, and then enforce that only one of protoBytesDecoder or descriptor/protoMessageType can be set, and then automatically construct a 'FileBasedProtobufBytesDecoder' if the old properties are set and the new is null.

Then in a future version we can consider dropping them

Good idea, I'll implement it.

clintropolis · 2021-02-05T11:08:50Z

...nsions/src/main/java/org/apache/druid/data/input/protobuf/FileBasedProtobufBytesDecoder.java

+    }
+  }
+
+  @SuppressWarnings("checkstyle:RightCurly")


what is this suppression for?

I'll delete it.

clintropolis · 2021-02-05T11:21:13Z

.../main/java/org/apache/druid/data/input/protobuf/SchemaRegistryBasedProtobufBytesDecoder.java

+      DynamicMessage message = DynamicMessage.parseFrom(descriptor, rawMessage);
+      return message;
+    }
+    catch (Exception e) {


maybe this "try" should block be split up error handling so that error messaging for failing to fetch the schema from the registry, which doesn't exactly seem like a parse exception, can be distinguished from an actual parse exception

I'll split it.

…pt to old version

bananaaggle · 2021-02-22T02:27:58Z

@clintropolis Hi, I change code follow your suggestion two weeks ago, could you please help me review this change?

clintropolis

Sorry very for the delay @bananaaggle! The code changes lgtm 👍, but I think we need to fix the license stuff. One minor nitpick is that some of the exception handling could maybe be a bit more specific than catching Exception probably?

Also, could you update the documentation for the protobuf extension, https://github.com/apache/druid/blob/master/docs/development/extensions-core/protobuf.md, to include the new stuff, and instructions to get the jar to use it? You can do something similar to here: https://github.com/apache/druid/blob/master/docs/development/extensions-core/mysql.md#installing-the-mysql-connector-library.

clintropolis · 2021-02-22T07:50:39Z

distribution/bin/check-licenses.py

@@ -225,6 +225,9 @@ def build_compatible_license_names():
    compatible_licenses['Apache License v2.0'] = 'Apache License version 2.0'
    compatible_licenses['Apache License, version 2.0'] = 'Apache License version 2.0'
    compatible_licenses['Apache 2.0 License'] = 'Apache License version 2.0'
+    compatible_licenses['Apache License, 2.0'] = 'Apache License version 2.0'
+    compatible_licenses['Confluent Community License'] = 'Confluent Community License'


I looked at the source files for kafka-protobuf-provider here, https://github.com/confluentinc/schema-registry/tree/master/protobuf-provider/src/main/java/io/confluent/kafka/schemaregistry/protobuf

and there appears to be a mix of Apache and Confluent Community License headers, which I assume it means both licenses apply.

Unfortunately, I think this means that we cannot distribute this jar, but I think that instead we can provide instructions in the documentation on how users can download it (see https://www.apache.org/legal/resolved.html#optional), similar to how we do with the MySQL connector.

The pom.xml will need to mark this jar as provided, and you will need to remove this license from this python script, and remove kafka-protobuf-provider from license.yaml. I think maybe it is ok if tests still depend on it, since it wouldn't cause packaging to distribute it? but i'm unsure exactly...

bananaaggle · 2021-02-23T08:07:41Z

Sorry very for the delay @bananaaggle! The code changes lgtm 👍, but I think we need to fix the license stuff. One minor nitpick is that some of the exception handling could maybe be a bit more specific than catching Exception probably?

Also, could you update the documentation for the protobuf extension, https://github.com/apache/druid/blob/master/docs/development/extensions-core/protobuf.md, to include the new stuff, and instructions to get the jar to use it? You can do something similar to here: https://github.com/apache/druid/blob/master/docs/development/extensions-core/mysql.md#installing-the-mysql-connector-library.

It is a good idea. I've separated this jar and write document for its usage.

bananaaggle · 2021-03-01T06:15:17Z

@clintropolis Hi, I've written document for this feature. Can you help me refine it? Thanks.

clintropolis

lgtm, just a few minor suggestions on docs, thanks 👍

clintropolis · 2021-03-03T09:29:27Z