[schema] expose the native record for struct schema #9614

eolivelli · 2021-02-18T09:46:12Z

Motivation

Allow GenericRecord consumers to access the underlying implementation, such as Avro GenericRecord, Protobug DynamicMessage or JSON JSONNode.

Modifications

This is patch introduces support for retrieving such information, with two methods:

GenericRecord#getSchemaType: return the type of schema (JSON,ProfobufNative,AVRO)
GenericRecord#getNativeRecod(Class expectedType): return the internal record

Consumer<GenericRecord> consumer = client.newConsumer(Schema.AUTO_CONSUME());
GenericRecord value = consumer.receive().getValue();
assertEquals(SchemaType.AVRO, res.getSchemaType());
org.apache.avro.generic.GenericRecord nativeRecord = res.getNativeRecord(org.apache.avro.generic.GenericRecord.class);
org.apache.avro.Schema schema = nativeRecord.getSchema();

Verifying this change

New tests cases are added

Does this pull request potentially affect one of the following parts:

This change introduces a new API

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? JavaDocs

eolivelli · 2021-02-18T09:48:41Z

@sijie @shiv4289 @aahmed-se please take a look

eolivelli · 2021-02-18T09:49:09Z

I will spend time in adding tests only when we reach consensus on the best API to provide to our users

eolivelli · 2021-02-19T08:17:29Z

@sijie I have added tests, please take a look, the patch is ready

eolivelli · 2021-02-19T10:57:26Z

/pulsarbot run-failure-checks

eolivelli · 2021-02-22T07:25:50Z

@sijie the patch passed CI, please take a look.
This way we can close this topic

@codelipenghui @rdhabalia PTAL as well

codelipenghui · 2021-02-23T23:41:36Z

@congbobo184 Could you please help review this PR?

codelipenghui

When the consumer uses AUTO_CONSUME schema, how to distinguish the unwrap type? Seems the implementation is assuming the users already know the schema is Avro or protobuf. Users can use the ALWAYS_COMPATIBLE policy, the schemas of the topic might have avro and protobuf schema. So shall we need to get the Avro type from the GenericRecord first?

And, for the GenericAvroRecord, GenericProtobufNativeRecord, etc. They have exposed the native schema, I think it's easy to get the field type from the native schema since the field exposed the index and the name.

congbobo184 · 2021-02-24T02:43:39Z

I also have the same confusion as @codelipenghui said.

eolivelli · 2021-02-24T07:48:31Z

@codelipenghui @congbobo184

When you are inside a Sink you have this situation and you can access all of the schema info from the Record

Record<GenericRecord> record = ....;
GenericRecord value = record.getValue();
SchemaInfo schemaInfo = record.getSchema().getSchemaInfo();
SchemaType type = schemaInfo.getType();
value.getFields().forEach( f -> {
    org.apache.avro.Schema.Field avroField = f.unwrap(org.apache.avro.Schema.Field.class);
    org.apache.avro.Schema fieldSchema = avroField.schema();
    log.info("Field {} unwrapped as {} schema {}", f, avroField, fieldSchema);
});

Unfortunately GenericRecord is used both to receive and to send values from/to Pulsar.

We cannot easily add GenericRecord#getSchema
We already have GenericRecord#getSchemaVersion and this is already akward for people that use GenericRecord to produce messages.

Do you think we should add something like

interface GenericRecord {
     /**
       * Access Schema information from the underlying Schema implementation.
       * This API is useful only while receiving messages.
       * Returns null in case that the requested Schema type is not available
      */
     default <T> T unwrapSchema(Class<T> schemaType) {
         return null;
     }
     /**
       * Access Schema information.
       * This API is useful only while receiving messages.
       * Returns null in case that the requested information is not available.
      */
     default Optional<SchemaInfo> getSchemaInfo() {
         return Options.empty();
     }
}

I will also like it, as it will help users that are not inside a Sink and they are receiving messages with the Consumer or Reader API.

In that case I would make getSchemaVersion() an optional method by adding a default implementation and adding a note on the JavaDoc that the method is intended to be used only while receiving messages.

codelipenghui · 2021-02-24T08:30:01Z

@eolivelli Currently, the PulsarClient maintains a schema cache for all topics, you can see getSchemaProviderLoadingCache method in the PulsarClientImpl, so that you can get the schema by the topic name and the schema version.

eolivelli · 2021-02-24T08:39:27Z

@codelipenghui I cannot get your point.
Are you suggesting to expose an API in the PulsarClient to access the Schema given a topic and a SchemaVersion ?

sijie · 2021-02-24T09:02:47Z

@eolivelli I shared the same question with @codelipenghui.

Why can't you use a code snippet like the below?

Record<GenericRecord> record = ....;
GenericRecord value = record.getValue();
SchemaInfo schemaInfo = record.getSchema().getSchemaInfo();
SchemaType type = schemaInfo.getType();

if (SchemaTye.AVRO == avro) {
   // the record is an AVRO record
    org.apache.avro.generic.GenericRecord avroRecord = ((GenericAvroRecord) value).getAvroRecord();
    handleAvroRecord(avro);
} else if (SchemaType.JSON == avro) {
   // the record is a JSON record
   JsonNode jsonRecord = ((GenericJsonRecord) value).getJsonNode();
   handleJsonRecord(jsonRecord);
} else if (SchemaType....) {
   ...
}

eolivelli · 2021-02-24T10:09:44Z

@sijie @codelipenghui
The answer is that GenericAvroRecord is not part of the public API, it is in org.apache.pulsar.client.impl.schema.generic package.
GenericRecord is in org.apache.pulsar.client.api.schema and so it is safe to be used by clients and by Pulsar Functions/Pulsar IO modules.

I believe that we need a strong and feature complete public API, that we can maintain in the future and can be used by users.

eolivelli · 2021-03-01T07:16:01Z

@sijie @codelipenghui ping

codelipenghui · 2021-03-04T00:54:48Z

@eolivelli @sijie Is it better to add method getNativeRecord or getOriginalRecord in the GenericRecord interface?

sijie · 2021-03-04T01:57:12Z

I think what @codelipenghui suggested can be a good solution.

eolivelli · 2021-03-04T06:06:47Z

Good idea. I will update the patch soon

eolivelli · 2021-03-04T17:23:12Z

/pulsarbot run-failure-tests

pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/GenericRecord.java

sijie · 2021-03-08T05:54:47Z

pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/GenericRecord.java

+     * @return the internal representation of the record, or null if the requested information is not available.
+     */
+    default <T> T getNativeRecord(Class<T> clazz) {
+        return null;


I am confused why do you throw UnsupportedOperationException for getSchemaType and return null for this method.

because we do not have a good "default value" for getSchemaType

but for this method the expected behaviour is that you get a null value if you are asking for an unsupported interface, so returning null here is the expected behaviour.

Why not throw UnsupportedOperationException for getNativeRecord? It is a much clear behavior than returning null.

I believe that UnsupportedOperationException does not work well,
because code that calls that method will have to catch it, and catching UnsupportedOperationException smells.

I thinking about generic code that handles multiple different types of records.

sijie · 2021-03-08T05:55:21Z

pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/GenericRecord.java

+     *
+     * @return the internal representation of the record, or null if the requested information is not available.
+     */
+    default <T> T getNativeRecord(Class<T> clazz) {


Suggested change

default <T> T getNativeRecord(Class<T> clazz) {

default <T> T getNativeRecord() {

Why do we need Class<T> here?

A couple of reasons:

I am adding this specification because this way the user won't have to deal with ClassCastExceptions or to use 'instanceof'.

Also the implementation of GenericRecord will be allowed to implement some "compatibility" feature.
For instance if in the future we will move away from JsonNode but the client code still expects to receive a JsonNode, we will be able to return a properly crafted JsonNode instance and to not break clients.
I am not saying we will do this soon, but if we do not add such support we won't be able to add it in the future.

I am not sure about that. Having Class<T> clazz means that the users need to know what exactly class that each schema type will use. This will be exactly the same as what I comment on #9614 (comment). There is no difference between these two approaches.

Because people anyway need to know what exactly class to use, I will instead suggest keeping the interface as simple as Object getNativeRecord(). Returning Object is very commonly seen in a lot of data processing engines that are dealing with different object types. I don't see it is a big problem to do the same thing here.

dlg99

lgtm

sijie · 2021-03-08T21:53:31Z

pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/GenericRecord.java

+     * @return the internal representation of the record, or null if the requested information is not available.
+     */
+    default <T> T getNativeRecord(Class<T> clazz) {
+        return null;


Why not throw UnsupportedOperationException for getNativeRecord? It is a much clear behavior than returning null.

sijie · 2021-03-08T21:59:56Z

pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/GenericRecord.java

+     *
+     * @return the internal representation of the record, or null if the requested information is not available.
+     */
+    default <T> T getNativeRecord(Class<T> clazz) {


I am not sure about that. Having Class<T> clazz means that the users need to know what exactly class that each schema type will use. This will be exactly the same as what I comment on #9614 (comment). There is no difference between these two approaches.

Because people anyway need to know what exactly class to use, I will instead suggest keeping the interface as simple as Object getNativeRecord(). Returning Object is very commonly seen in a lot of data processing engines that are dealing with different object types. I don't see it is a big problem to do the same thing here.

eolivelli · 2021-03-09T08:39:31Z

@sijie I have updated getNativeRecord to returning Object with a "throw new UnsupportedOperationException" as default implementation.

PTAL

@codelipenghui @dlg99 you already Approved this patch in the old form, please take a look again.

eolivelli · 2021-03-09T10:58:25Z

/pulsarbot rerun-failure-checks

eolivelli · 2021-03-09T13:32:28Z

/pulsarbot rerun-failure-checks

eolivelli · 2021-03-10T08:17:50Z

Thank you @sijie @codelipenghui and @dlg99 for your reviews and suggestions

eolivelli mentioned this pull request Feb 18, 2021

Issue 9004: Pulsar Schema API: provide Type information for Fields #9343

Closed

eolivelli added 3 commits February 19, 2021 08:17

Schema API: expose underlying Schema information for Fields

aa56de1

remove useless code

15c0a88

Add test cases

12369f5

eolivelli marked this pull request as ready for review February 19, 2021 08:16

eolivelli force-pushed the fix/field-types-2 branch from c341a28 to 12369f5 Compare February 19, 2021 08:17

codelipenghui assigned eolivelli Feb 23, 2021

codelipenghui added the component/schemaregistry label Feb 23, 2021

codelipenghui added this to the 2.8.0 milestone Feb 23, 2021

codelipenghui reviewed Feb 24, 2021

View reviewed changes

eolivelli requested a review from codelipenghui February 24, 2021 12:18

eolivelli added 3 commits March 4, 2021 07:58

Merge branch 'master' into fix/field-types-2

58a4e2f

Implement getNativeRecord and getSchemaType

6f4d52a

handle UOE

76f5d6c

codelipenghui requested review from hangc0276, sijie, 315157973, jiazhai, merlimat and zymap March 4, 2021 08:52

fix tests

fb75471

eolivelli added 2 commits March 5, 2021 08:10

Merge branch 'master' into fix/field-types-2

02de471

fix test

360cc7b

sijie requested changes Mar 8, 2021

View reviewed changes

eolivelli added 2 commits March 8, 2021 11:43

Merge branch 'master' into fix/field-types-2

6fe0a1e

Remove default method implementation

16f968d

eolivelli requested a review from sijie March 8, 2021 10:45

dlg99 approved these changes Mar 8, 2021

View reviewed changes

sijie requested changes Mar 8, 2021

View reviewed changes

eolivelli added 2 commits March 9, 2021 09:30

Merge branch 'master' into fix/field-types-2

41433b9

Address Sijie's comments

7d0ecf1

fix build

65b2795

codelipenghui approved these changes Mar 10, 2021

View reviewed changes

sijie approved these changes Mar 10, 2021

View reviewed changes

sijie changed the title ~~Schema API: expose underlying Schema information for Fields~~ [schema] expose the native record for struct schema Mar 10, 2021

sijie merged commit 487d2af into apache:master Mar 10, 2021

eolivelli deleted the fix/field-types-2 branch March 10, 2021 08:17

eolivelli mentioned this pull request Mar 30, 2021

Implement GenericObject - Allow GenericRecord to wrap any Java Object #10057

Merged

tisonkun mentioned this pull request Dec 7, 2022

Pulsar Schema API: provide Type information for Fields #9004

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[schema] expose the native record for struct schema #9614

[schema] expose the native record for struct schema #9614

eolivelli commented Feb 18, 2021 •

edited

Loading

eolivelli commented Feb 18, 2021

eolivelli commented Feb 18, 2021

eolivelli commented Feb 19, 2021

eolivelli commented Feb 19, 2021

eolivelli commented Feb 22, 2021

codelipenghui commented Feb 23, 2021

codelipenghui left a comment

congbobo184 commented Feb 24, 2021

eolivelli commented Feb 24, 2021

codelipenghui commented Feb 24, 2021

eolivelli commented Feb 24, 2021

sijie commented Feb 24, 2021

eolivelli commented Feb 24, 2021

eolivelli commented Mar 1, 2021

codelipenghui commented Mar 4, 2021

sijie commented Mar 4, 2021

eolivelli commented Mar 4, 2021

eolivelli commented Mar 4, 2021

sijie Mar 8, 2021

eolivelli Mar 8, 2021

sijie Mar 8, 2021

eolivelli Mar 9, 2021

sijie Mar 8, 2021

eolivelli Mar 8, 2021

sijie Mar 8, 2021

dlg99 left a comment

sijie Mar 8, 2021

sijie Mar 8, 2021

eolivelli commented Mar 9, 2021

eolivelli commented Mar 9, 2021

eolivelli commented Mar 9, 2021

eolivelli commented Mar 10, 2021

	default <T> T getNativeRecord(Class<T> clazz) {
	default <T> T getNativeRecord() {

[schema] expose the native record for struct schema #9614

[schema] expose the native record for struct schema #9614

Conversation

eolivelli commented Feb 18, 2021 • edited Loading

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

eolivelli commented Feb 18, 2021

eolivelli commented Feb 18, 2021

eolivelli commented Feb 19, 2021

eolivelli commented Feb 19, 2021

eolivelli commented Feb 22, 2021

codelipenghui commented Feb 23, 2021

codelipenghui left a comment

Choose a reason for hiding this comment

congbobo184 commented Feb 24, 2021

eolivelli commented Feb 24, 2021

codelipenghui commented Feb 24, 2021

eolivelli commented Feb 24, 2021

sijie commented Feb 24, 2021

eolivelli commented Feb 24, 2021

eolivelli commented Mar 1, 2021

codelipenghui commented Mar 4, 2021

sijie commented Mar 4, 2021

eolivelli commented Mar 4, 2021

eolivelli commented Mar 4, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dlg99 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eolivelli commented Mar 9, 2021

eolivelli commented Mar 9, 2021

eolivelli commented Mar 9, 2021

eolivelli commented Mar 10, 2021

eolivelli commented Feb 18, 2021 •

edited

Loading