ARROW-2645: [Java] Refactor ArrowWriter to remove all ArrowFileWriter specifc logic #2090
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…er where they are used
Member
Author
Member
Author
|
@icexelloss, please review if you can, thanks! |
Contributor
|
@BryanCutler does this PR refactor out file format specific logic from ArrowWriter to ArrowFileWriter? |
icexelloss
reviewed
May 31, 2018
|
|
||
| private static final Logger LOGGER = LoggerFactory.getLogger(ArrowFileWriter.class); | ||
|
|
||
| private final List<ArrowBlock> dictionaryBlocks = new ArrayList<>(); |
Contributor
There was a problem hiding this comment.
Can you add a comment on the purpose of these two lists here?
Member
Author
Yes, specifically saving |
Contributor
|
Maybe it makes more sense to change the PR title to be "Refactor record batch and dictionary batch accumulating logic from ArrowWriter to ArrowFileWriter" (at least more understandable to me) but this is minor so your call @BryanCutler |
BryanCutler
changed the title
Member
Author
|
Thanks for reviewing @icexelloss ! |
Member
Author
|
merged to master |
pribor
pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
… specifc logic Related to apache#2079 , the DictionaryBatch `ArrowBlock`s were being accumulated in the base class and used by `ArrowFileWriter` but not `ArrowStreamWriter`. This refactors the `ArrowWriter` to move Lists of ArrowBlocks from the base class to only `ArrowFileWriter`. Moved tests counting ArrowBlocks written from ArrowStreamWriter tests to ArrowFileWriter. Author: Bryan Cutler <cutlerb@gmail.com> Closes apache#2090 from BryanCutler/java-ArrowStreamWriter-accum-DictionaryBlocks-ARROW-2645 and squashes the following commits: fc7f061 <Bryan Cutler> added comment about saving ArrowBlocks 5e4711a <Bryan Cutler> Moved lists of ArrowBlocks in ArrowWriter base class to ArrowFileWriter where they are used
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related to #2079 , the DictionaryBatch
ArrowBlocks were being accumulated in the base class and used byArrowFileWriterbut notArrowStreamWriter. This refactors theArrowWriterto move Lists of ArrowBlocks from the base class to onlyArrowFileWriter.Moved tests counting ArrowBlocks written from ArrowStreamWriter tests to ArrowFileWriter.