Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a JSON RecordBuilder to the GenericJsonSchema #10052

Merged
merged 1 commit into from
Apr 20, 2021

Conversation

vroyer
Copy link
Contributor

@vroyer vroyer commented Mar 26, 2021

Motivation

Provide a JSON GenericRecord builder allowing to produce JsonGenericRecord from the GenericJsonSchema.

Modifications

Add a new class org.apache.pulsar.client.impl.schema.generic.JsonRecordBuilderImpl
Modify the org.apache.pulsar.client.impl.schema.generic.GenericJsonSchema.newRecordBuilder()

Verifying this change

Add a unit test in org.apache.pulsar.client.impl.schema.JSONSchemaTest

Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice work
I left a few comments

Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

please remove the "star import", then the patch is good to go from my point of view.

@sijie @codelipenghui can you please take a look ?

private static ObjectMapper objectMapper = new ObjectMapper();

private final GenericSchemaImpl genericSchema;
private Map<String, Object> map = new HashMap<>();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use the ObjectNode directly? Use the ObjectMapper to generate the JsonNode also create an internal map there,

Looks like:

ObjectNode node = JsonNodeFactory.instance.objectNode();
node.putPOJO("a", "a");

So that we don't need to create a HashMap for each JsonRecoredBuilder.

And we can reuse the ObjectNode by adding it to the ThreadLocal? We can call ObjectNode.removeAll() after build the GenericRecord.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@codelipenghui
I prefer to not use ThreadLocal, as it will complicate the usage, as the user is not able to have two builder on the same thread, with the risk of mixing data from two different records.

We can have a simple implementation now and enhance it when needed.
Creating just one object is not a big deal for the JVM.

we do not have such complexity for the AVRO record builder

class AvroRecordBuilderImpl implements GenericRecordBuilder {

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jackson already have a pretty good json node factory to build json node. We should use it instead of building the logic again that Jackson already did.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ObjectNode builder also uses a Map to build a JsonNode (a LinkedHashMap), but requires to call the a dedicated put() method for each native types or putPOJO() for Objects (like in such a code).

The ObjectMapper.valueToTree(map) produce the same result, a JsonNode, and the underlying serialiser properly deals with native and POJO objects in the same way without serializing it into "basic" nodes. The ObjectNode allows to filter or alter nodes, but we don't need here in the build() method, so the ObjectMapper.valueToTree() it is simple and efficient here.

Moreover, the ObjectMapper is thread-safe and a static instance is enough to do the job.

private static ObjectMapper objectMapper = new ObjectMapper();

private final GenericSchemaImpl genericSchema;
private Map<String, Object> map = new HashMap<>();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jackson already have a pretty good json node factory to build json node. We should use it instead of building the logic again that Jackson already did.

@vroyer vroyer force-pushed the json_generic_record_builder branch 2 times, most recently from 7977f0e to ae19f43 Compare April 13, 2021 08:19
@vroyer vroyer requested a review from sijie April 16, 2021 13:11
@eolivelli
Copy link
Contributor

@sijie as you told that you are okay with this patch, can you please click on "Approve" ? otherwise we cannot merge it

@vroyer vroyer force-pushed the json_generic_record_builder branch from 6558e5b to 23cb3ad Compare April 16, 2021 17:09
@vroyer vroyer force-pushed the json_generic_record_builder branch from 23cb3ad to 40889a5 Compare April 16, 2021 17:12
@codelipenghui codelipenghui merged commit 08cbdc7 into apache:master Apr 20, 2021
eolivelli pushed a commit to datastax/pulsar that referenced this pull request May 12, 2021
Provide a JSON GenericRecord builder allowing to produce JsonGenericRecord from the GenericJsonSchema.

Add a new class org.apache.pulsar.client.impl.schema.generic.JsonRecordBuilderImpl
Modify the org.apache.pulsar.client.impl.schema.generic.GenericJsonSchema.newRecordBuilder()

Add a unit test in org.apache.pulsar.client.impl.schema.JSONSchemaTest

(cherry picked from commit 08cbdc7)
eolivelli pushed a commit to datastax/pulsar that referenced this pull request May 12, 2021
Provide a JSON GenericRecord builder allowing to produce JsonGenericRecord from the GenericJsonSchema.

Add a new class org.apache.pulsar.client.impl.schema.generic.JsonRecordBuilderImpl
Modify the org.apache.pulsar.client.impl.schema.generic.GenericJsonSchema.newRecordBuilder()

Add a unit test in org.apache.pulsar.client.impl.schema.JSONSchemaTest
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants