source jackson changes that got us 80 mb per sec#55892
source jackson changes that got us 80 mb per sec#55892Subodh Kant Chaturvedi (subodh1810) wants to merge 1 commit intomasterfrom
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
| /** | ||
| * Parses command line args to a type safe config object for each command type. | ||
| */ | ||
| public class CliParse { |
There was a problem hiding this comment.
Ignore this class, this is just a copy paste (I was playing with some stuff)
| implementation 'com.fasterxml.jackson.core:jackson-annotations:2.18.3' | ||
|
|
||
| // Jackson Afterburner module for performance optimizations | ||
| implementation 'com.fasterxml.jackson.module:jackson-module-afterburner:2.18.3' |
There was a problem hiding this comment.
This AfterBurner module is important
| .configure(JsonGenerator.Feature.WRITE_BIGDECIMAL_AS_PLAIN, true) | ||
| .registerModule(new JavaTimeModule()) | ||
| .registerModule(new AfterburnerModule()); | ||
| } |
There was a problem hiding this comment.
Register the after burner module
| try (SequenceWriter seqWriter = DummyIterator.OBJECT_MAPPER | ||
| .writerFor(AirbyteMessage.class) | ||
| .with(new MinimalPrettyPrinter(System.lineSeparator())) | ||
| .writeValues(System.out)) { |
There was a problem hiding this comment.
- Use a SequenceWriter
- Use the writeValues method
- Write directly to the output stream (System.out)
- Specify the POJO class that the writer is handling (AirbyteMessage)
| AirbyteMessage message = iterator.next(); | ||
| seqWriter.write(message); | ||
| // Should we flush after each record?? | ||
| System.out.flush(); |
There was a problem hiding this comment.
Am not sure if we should flush after each method but since thats what we do with System.out.println, I just did it. I have not tested if there are any speed improvements if we dont flush after each record
|
We are closing this PR due to inactivity. We regularly close PRs that were created more than 6 months ago. If you'd like to keep working on it, please feel free reopen it. We'd be happy to keep working with you. |
3 participants
No description provided.