Add DatumReader Cache to improve de-serialization performance #65
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hhkkxxx133
reviewed
Jul 30, 2021
.../main/java/com/amazonaws/services/schemaregistry/deserializers/avro/DatumReaderInstance.java
Outdated
Show resolved
Hide resolved
blacktooth
added a commit
that referenced
this pull request
Aug 23, 2021
* Add DatumReader Cache to improve de-serialization performance * Fix exception message Co-authored-by: Ravindranath Kakarla <rnath@amazon.com>
blacktooth
added a commit
that referenced
this pull request
Feb 11, 2022
* Add message indices parser for Protobuf schemas. (#52) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Protobuf support (#60) * Add message indices parser for Protobuf schemas. * Encode Protobuf messages with message index Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Fix checkstyle build error and reduce tests for canaries * Update README.md * Update README.md * Pull everit from maven central instead of jitpack (#54) Co-authored-by: mohit <mpaliwa@amazon.com> * Bump up version to v1.1.1 * Update ReadMe to v1.1.1 * Add message indices parser for Protobuf schemas. (#52) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Add message indices parser for Protobuf schemas. (#52) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Protobuf support (#60) * Add message indices parser for Protobuf schemas. * Encode Protobuf messages with message index Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Add message indices parser for Protobuf schemas. (#52) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Update dependencies for Protobuf and create class structure cr: https://code.amazon.com/reviews/CR-52122396 * Add ProtobufMessageType to aid Protobuf deserialization cr: https://code.amazon.com/reviews/CR-54151956 * Protobuf DynamicMessage deserialization cr: https://code.amazon.com/reviews/CR-54424261 * Fix serializer build issue * Fix serializer build issue * Pull everit from maven central instead of jitpack (#54) Co-authored-by: mohit <mpaliwa@amazon.com> * Fix resource clean up in Kafka integration test * Add documentation for configuring a specific GSR resource with the Kafka Connect converter (#58) * Reduce logging (#64) * Add .gitignore To avoid having files accidentally added to the project * Reduce and update logging It looks like these changes were lost in a merge at some point. Created based on the diff on GitHub. * Add DatumReader Cache to improve de-serialization performance (#65) * Add DatumReader Cache to improve de-serialization performance * Fix exception message Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Introduce cache to improve serialization performance (#67) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Bump up version to v1.1.2 (#70) Co-authored-by: mohit <mpaliwa@amazon.com> * Bump up v1.1.2 in README.md * Refactor GlueSchemaRegistryDataFormatDeserializer to accept Schema object (#72) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Add a utility to derive class name from message descriptor (#74) * Refactor GlueSchemaRegistryDataFormatDeserializer to accept Schema object * Add a utility to derive Protobuf compiled Java class name from FileDescriptor. Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Add support to deserialize to Protobuf POJOs. (#76) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Handle ProtobufMessageType not set. (#85) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Introduce cache to improve Protobuf serializer/de-serializer performance. (#90) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Merging latest approved changes to protobuf-support branch. (#98) * Modify UserAgent to emit usage metrics (#77) * Add tests to include key and value schemas both (#79) Co-authored-by: Mohit Paliwal <mpaliwa@amazon.com> * Bump to 1.1.3 (#78) * Modify UserAgent to emit usage metrics * Bump version to 1.1.3 Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Update README.md * Upgrade Kafka version to 2.8.1 * Bump up version to 1.1.4 * Upgrade transitive dependencies to latest versions. Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Remove configuration logging information (#89) * Bump up version to 1.1.5 (#91) * Specify the avro-maven-plugin version explicitly (#97) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Bump up version to v1.1.1 * Add message indices parser for Protobuf schemas. (#52) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> Co-authored-by: Mohit Paliwal <iammohitpaliwal@gmail.com> Co-authored-by: Mohit Paliwal <mpaliwa@amazon.com> Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> Co-authored-by: Kexin Hui <khi@amazon.com> Co-authored-by: Linyu Yao <71672182+LinyuYao1021@users.noreply.github.com> * Add integration tests for Protobuf support (#99) * Protobuf integration tests for Kinesis use-cases * Protobuf integration test cases for Kafka use-cases. Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Upgrade to latest versions of Protobuf dependencies. (#103) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Protobuf support (#117) * Modify UserAgent to emit usage metrics (#77) * Add tests to include key and value schemas both (#79) Co-authored-by: Mohit Paliwal <mpaliwa@amazon.com> * Bump to 1.1.3 (#78) * Modify UserAgent to emit usage metrics * Bump version to 1.1.3 Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Update README.md * Upgrade Kafka version to 2.8.1 * Bump up version to 1.1.4 * Upgrade transitive dependencies to latest versions. Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Remove configuration logging information (#89) * Bump up version to 1.1.5 (#91) * Specify the avro-maven-plugin version explicitly (#97) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Fix bug with canary tests running the entire suite. (#102) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Upgrade to latest versions of Protobuf dependencies. * Bump log4j-core from 2.13.2 to 2.15.0 (#107) * Bump log4j-api from 2.13.2 to 2.15.0 (#106) Bumps log4j-api from 2.13.2 to 2.15.0. --- updated-dependencies: - dependency-name: org.apache.logging.log4j:log4j-api dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump up version from 1.1.5 to 1.1.6 (#108) Co-authored-by: Mohit Paliwal <mpaliwa@amazon.com> * Bump log4j-api from 2.15.0 to 2.16.0 (#109) Bumps log4j-api from 2.15.0 to 2.16.0. --- updated-dependencies: - dependency-name: org.apache.logging.log4j:log4j-api dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump up version from 1.1.6 to 1.1.7 (#111) Co-authored-by: Mohit Paliwal <mpaliwa@amazon.com> * Pull current region for integration tests (#112) Co-authored-by: Mohit Paliwal <mpaliwa@amazon.com> * Bump log4j-api from 2.16.0 to 2.17.0 (#113) Bumps log4j-api from 2.16.0 to 2.17.0. --- updated-dependencies: - dependency-name: org.apache.logging.log4j:log4j-api dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump log4j-core from 2.16.0 to 2.17.0 (#114) Bumps log4j-core from 2.16.0 to 2.17.0. --- updated-dependencies: - dependency-name: org.apache.logging.log4j:log4j-core dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump up version from 1.1.7 to 1.1.8 (#115) Co-authored-by: Mohit Paliwal <mpaliwa@amazon.com> Co-authored-by: Mohit Paliwal <iammohitpaliwal@gmail.com> Co-authored-by: Mohit Paliwal <mpaliwa@amazon.com> Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> Co-authored-by: Kexin Hui <khi@amazon.com> Co-authored-by: Linyu Yao <71672182+LinyuYao1021@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Remove extra text from Protobuf schema generation (#132) * Remove extra text from Protobuf schema generation Co-authored-by: Kexin Hui <khi@amazon.com> * Add documentation for Protobuf support (#134) * Add documentation for Protobuf support Co-authored-by: Kexin Hui <khi@amazon.com> * Add message indices parser for Protobuf schemas. (#52) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Protobuf support (#60) * Add message indices parser for Protobuf schemas. * Encode Protobuf messages with message index Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Bump up version to v1.1.1 * Add message indices parser for Protobuf schemas. (#52) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Pull everit from maven central instead of jitpack (#54) Co-authored-by: mohit <mpaliwa@amazon.com> * Fix resource clean up in Kafka integration test * Add message indices parser for Protobuf schemas. (#52) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Update dependencies for Protobuf and create class structure cr: https://code.amazon.com/reviews/CR-52122396 * Add ProtobufMessageType to aid Protobuf deserialization cr: https://code.amazon.com/reviews/CR-54151956 * Protobuf DynamicMessage deserialization cr: https://code.amazon.com/reviews/CR-54424261 * Fix serializer build issue * Fix serializer build issue * Refactor GlueSchemaRegistryDataFormatDeserializer to accept Schema object (#72) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Add a utility to derive class name from message descriptor (#74) * Refactor GlueSchemaRegistryDataFormatDeserializer to accept Schema object * Add a utility to derive Protobuf compiled Java class name from FileDescriptor. Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Add support to deserialize to Protobuf POJOs. (#76) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Handle ProtobufMessageType not set. (#85) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Introduce cache to improve Protobuf serializer/de-serializer performance. (#90) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Add integration tests for Protobuf support (#99) * Protobuf integration tests for Kinesis use-cases * Protobuf integration test cases for Kafka use-cases. Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Upgrade to latest versions of Protobuf dependencies. * Protobuf support (#117) * Modify UserAgent to emit usage metrics (#77) * Add tests to include key and value schemas both (#79) Co-authored-by: Mohit Paliwal <mpaliwa@amazon.com> * Bump to 1.1.3 (#78) * Modify UserAgent to emit usage metrics * Bump version to 1.1.3 Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Update README.md * Upgrade Kafka version to 2.8.1 * Bump up version to 1.1.4 * Upgrade transitive dependencies to latest versions. Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Remove configuration logging information (#89) * Bump up version to 1.1.5 (#91) * Specify the avro-maven-plugin version explicitly (#97) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Fix bug with canary tests running the entire suite. (#102) Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> * Upgrade to latest versions of Protobuf dependencies. * Bump log4j-core from 2.13.2 to 2.15.0 (#107) * Bump log4j-api from 2.13.2 to 2.15.0 (#106) Bumps log4j-api from 2.13.2 to 2.15.0. --- updated-dependencies: - dependency-name: org.apache.logging.log4j:log4j-api dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump up version from 1.1.5 to 1.1.6 (#108) Co-authored-by: Mohit Paliwal <mpaliwa@amazon.com> * Bump log4j-api from 2.15.0 to 2.16.0 (#109) Bumps log4j-api from 2.15.0 to 2.16.0. --- updated-dependencies: - dependency-name: org.apache.logging.log4j:log4j-api dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump up version from 1.1.6 to 1.1.7 (#111) Co-authored-by: Mohit Paliwal <mpaliwa@amazon.com> * Pull current region for integration tests (#112) Co-authored-by: Mohit Paliwal <mpaliwa@amazon.com> * Bump log4j-api from 2.16.0 to 2.17.0 (#113) Bumps log4j-api from 2.16.0 to 2.17.0. --- updated-dependencies: - dependency-name: org.apache.logging.log4j:log4j-api dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump log4j-core from 2.16.0 to 2.17.0 (#114) Bumps log4j-core from 2.16.0 to 2.17.0. --- updated-dependencies: - dependency-name: org.apache.logging.log4j:log4j-core dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump up version from 1.1.7 to 1.1.8 (#115) Co-authored-by: Mohit Paliwal <mpaliwa@amazon.com> Co-authored-by: Mohit Paliwal <iammohitpaliwal@gmail.com> Co-authored-by: Mohit Paliwal <mpaliwa@amazon.com> Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> Co-authored-by: Kexin Hui <khi@amazon.com> Co-authored-by: Linyu Yao <71672182+LinyuYao1021@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Remove extra text from Protobuf schema generation (#132) * Remove extra text from Protobuf schema generation Co-authored-by: Kexin Hui <khi@amazon.com> * Add documentation for Protobuf support (#134) * Add documentation for Protobuf support Co-authored-by: Kexin Hui <khi@amazon.com> Co-authored-by: Ravindranath Kakarla <rnath@amazon.com> Co-authored-by: mohit <mpaliwa@amazon.com> Co-authored-by: Mohit Paliwal <iammohitpaliwal@gmail.com> Co-authored-by: Kexin Hui <khi@amazon.com> Co-authored-by: Jake Baker <jakbak@amazon.com> Co-authored-by: Mark Lambert <mark.lambert@gmail.com> Co-authored-by: Thomas Vestergaard Trolle <tveon@users.noreply.github.com> Co-authored-by: Kexin <31828975+hhkkxxx133@users.noreply.github.com> Co-authored-by: Linyu Yao <71672182+LinyuYao1021@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes:
Adding a cache to improve the performance of de-serializer by not parsing the schema and not re-creating the
DatumReaderfor every message.Testing
Profiled the code using AWS Code Guru and don't see
createDatumReaderandSchema.Parsemethod consuming CPU.Integration tests are successful locally.