Pulsar IO: allow Sinks to use native AVRO and JSON #11322
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nlu90
reviewed
Jul 15, 2021
...ons/runtime-all/src/test/java/org/apache/pulsar/functions/instance/JavaInstanceDepsTest.java
Show resolved
Hide resolved
|
Thanks for your contribution. For this PR, do we need to update docs? (The PR template contains info about doc, which helps others know more about the changes. Can you provide doc info in future PR descriptions? Thanks) |
The Sink should be able to manage the result of GenericRecord.getNativeObject(). In order to do this Apache AVRO must be loaded from the same classloader that is loading Apache Pulsar Runtime. The same problem applies in the case of a JsonNode returned by getNativeObject();
|
@Anonymitaet when we will find a good solution for this problem I will be happy to help with the docs. currently this is only a draft |
eolivelli
force-pushed
the
fix/core-api-deps
branch
2 times, most recently
from
910014a
to
01dea55
Compare
sijie
approved these changes
Jul 19, 2021
lhotari
added a commit
to lhotari/pulsar
that referenced
this pull request
Jul 21, 2021
)" This reverts commit f35766f.
Technoboy-
pushed a commit
to Technoboy-/pulsar
that referenced
this pull request
Jul 22, 2021
The Sink should be able to manage the result of GenericRecord.getNativeObject(). In order to do this Apache AVRO must be loaded from the same classloader that is loading Apache Pulsar Runtime. The same problem applies in the case of a JsonNode returned by getNativeObject(); *Modifications* - Add AVRO to the list of classes (like slf4j) to be loaded from the Pulsar runtime (this in turn imports Commons Compress and Jackson Databind). - Enhance the existing integration tests, that tested about the Schema definition, but it didn't actually try to "use" the `org.apache.avro.GenericRecord` object as well as the `Jackson JsonNode` object
codelipenghui
pushed a commit
that referenced
this pull request
Jul 23, 2021
The Sink should be able to manage the result of GenericRecord.getNativeObject(). In order to do this Apache AVRO must be loaded from the same classloader that is loading Apache Pulsar Runtime. The same problem applies in the case of a JsonNode returned by getNativeObject(); *Modifications* - Add AVRO to the list of classes (like slf4j) to be loaded from the Pulsar runtime (this in turn imports Commons Compress and Jackson Databind). - Enhance the existing integration tests, that tested about the Schema definition, but it didn't actually try to "use" the `org.apache.avro.GenericRecord` object as well as the `Jackson JsonNode` object (cherry picked from commit f35766f)
bharanic-dev
pushed a commit
to bharanic-dev/pulsar
that referenced
this pull request
Mar 18, 2022
The Sink should be able to manage the result of GenericRecord.getNativeObject(). In order to do this Apache AVRO must be loaded from the same classloader that is loading Apache Pulsar Runtime. The same problem applies in the case of a JsonNode returned by getNativeObject(); *Modifications* - Add AVRO to the list of classes (like slf4j) to be loaded from the Pulsar runtime (this in turn imports Commons Compress and Jackson Databind). - Enhance the existing integration tests, that tested about the Schema definition, but it didn't actually try to "use" the `org.apache.avro.GenericRecord` object as well as the `Jackson JsonNode` object
eolivelli
added a commit
to eolivelli/pulsar
that referenced
this pull request
Apr 3, 2022
The Sink should be able to manage the result of GenericRecord.getNativeObject(). In order to do this Apache AVRO must be loaded from the same classloader that is loading Apache Pulsar Runtime. The same problem applies in the case of a JsonNode returned by getNativeObject(); *Modifications* - Add AVRO to the list of classes (like slf4j) to be loaded from the Pulsar runtime (this in turn imports Commons Compress and Jackson Databind). - Enhance the existing integration tests, that tested about the Schema definition, but it didn't actually try to "use" the `org.apache.avro.GenericRecord` object as well as the `Jackson JsonNode` object (cherry picked from commit f35766f)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The Sink should be able to manage the result of GenericRecord.getNativeObject().
In order to do this Apache AVRO must be loaded from the same classloader that is loading Apache Pulsar Runtime.
The same problem applies in the case of a JsonNode returned by getNativeObject();
Modifications
org.apache.avro.GenericRecordobject as well as theJackson JsonNodeobject