Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Java] Allow passing pre-allocated buffers to VarCharVector #38254

Closed
jduo opened this issue Oct 12, 2023 · 1 comment · Fixed by #38266
Closed

[Java] Allow passing pre-allocated buffers to VarCharVector #38254

jduo opened this issue Oct 12, 2023 · 1 comment · Fixed by #38266
Assignees
@jduo
Labels
Breaking Change Includes a breaking change to the API Component: Java Type: bug
Milestone
15.0.0

Comments

@jduo
Copy link
Member

jduo commented Oct 12, 2023

Describe the bug, including details regarding any error messages, version, and platform.

Allow VarCharVector and LargeVarCharVector to let the caller take in a pre-allocated buffer. The Text class was originally designed for this purpose but VarCharVector does not currently let the caller supply their own Text object.

The purpose of this change would be to allow a caller iterating through a VarCharVector to re-use a buffer without having to do extra allocations.

Component(s)

Java
@jduo jduo added the Type: bug label Oct 12, 2023
@jduo
Copy link
Member Author

jduo commented Oct 13, 2023

take

jduo added a commit to jduo/arrow that referenced this issue Oct 14, 2023
@jduo
apacheGH-38254: [Java] Add reusable buffer getters to char/binary vec…
fcdf8cf 
…tors

Add a reusable buffer interface that can be populated by
character and binary vectors to avoid allocations when consuming
vector content.

Optimize getObject() on VarCharVector/LargeVarCharVector to
avoid an extra allocation of a byte array (copy from ArrowBuf
directly to the resulting Text).
@github-actions github-actions bot mentioned this issue Oct 14, 2023
Merged
jduo added a commit to jduo/arrow that referenced this issue Oct 16, 2023
@jduo
apacheGH-38254: [Java] Add reusable buffer getters to char/binary vec…
0561808 
…tors

Add a reusable buffer interface that can be populated by
character and binary vectors to avoid allocations when consuming
vector content.

Optimize getObject() on VarCharVector/LargeVarCharVector to
avoid an extra allocation of a byte array (copy from ArrowBuf
directly to the resulting Text).
jduo added a commit to jduo/arrow that referenced this issue Oct 16, 2023
@jduo
apacheGH-38254: [Java] Add reusable buffer getters to char/binary vec…
519258f 
…tors

Add a reusable buffer interface that can be populated by
character and binary vectors to avoid allocations when consuming
vector content.

Optimize getObject() on VarCharVector/LargeVarCharVector to
avoid an extra allocation of a byte array (copy from ArrowBuf
directly to the resulting Text).
jduo added a commit to jduo/arrow that referenced this issue Oct 20, 2023
@jduo
apacheGH-38254: [Java] Add reusable buffer getters to char/binary vec…
cc90fb1 
…tors

Add a reusable buffer interface that can be populated by
character and binary vectors to avoid allocations when consuming
vector content.

Optimize getObject() on VarCharVector/LargeVarCharVector to
avoid an extra allocation of a byte array (copy from ArrowBuf
directly to the resulting Text).
@lidavidm lidavidm changed the title Allow passing pre-allocated buffers to VarCharVector [Java] Allow passing pre-allocated buffers to VarCharVector Oct 20, 2023
@lidavidm lidavidm added the Breaking Change Includes a breaking change to the API label Oct 20, 2023
jduo added a commit to jduo/arrow that referenced this issue Oct 20, 2023
@jduo
apacheGH-38254: [Java] Add reusable buffer getters to char/binary vec…
6540883 
…tors

Add a reusable buffer interface that can be populated by
character and binary vectors to avoid allocations when consuming
vector content.

Optimize getObject() on VarCharVector/LargeVarCharVector to
avoid an extra allocation of a byte array (copy from ArrowBuf
directly to the resulting Text).
jduo added a commit to jduo/arrow that referenced this issue Oct 20, 2023
@jduo
apacheGH-38254: [Java] Add reusable buffer getters to char/binary vec…
d1471b4 
…tors

Add a reusable buffer interface that can be populated by
character and binary vectors to avoid allocations when consuming
vector content.

Optimize getObject() on VarCharVector/LargeVarCharVector to
avoid an extra allocation of a byte array (copy from ArrowBuf
directly to the resulting Text).
jduo added a commit to jduo/arrow that referenced this issue Oct 20, 2023
@jduo
apacheGH-38254: [Java] Add reusable buffer getters to char/binary vec…
025ac75 
…tors

Add a reusable buffer interface that can be populated by
character and binary vectors to avoid allocations when consuming
vector content.

Optimize getObject() on VarCharVector/LargeVarCharVector to
avoid an extra allocation of a byte array (copy from ArrowBuf
directly to the resulting Text).
jduo added a commit to jduo/arrow that referenced this issue Oct 23, 2023
@jduo
apacheGH-38254: [Java] Add reusable buffer getters to char/binary vec…
fb64d2c 
…tors

Add a reusable buffer interface that can be populated by
character and binary vectors to avoid allocations when consuming
vector content.

Optimize getObject() on VarCharVector/LargeVarCharVector to
avoid an extra allocation of a byte array (copy from ArrowBuf
directly to the resulting Text).
lidavidm pushed a commit that referenced this issue Oct 23, 2023
@jduo
GH-38254: [Java] Add reusable buffer getters to char/binary vectors (#…
6f8f34b 
…38266)

### Rationale for this change
Provide a way for a user to reuse a buffer when iterating over byte-array-based ValueVectors to avoid excessive
reallocations.

### What changes are included in this PR?
Add a reusable buffer interface that can be populated by character and binary vectors to avoid allocations when consuming vector content.

Optimize getObject() on VarCharVector/LargeVarCharVector to avoid an extra allocation of a byte array (copy from ArrowBuf directly to the resulting Text).

### Are these changes tested?

### Are there any user-facing changes?

Yes.

**This PR includes breaking changes to public APIs.**

* Closes: #38254

Authored-by: James Duong <duong.james@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
@lidavidm lidavidm added this to the 15.0.0 milestone Oct 23, 2023
JerAguilon pushed a commit to JerAguilon/arrow that referenced this issue Oct 25, 2023
@jduo @JerAguilon
apacheGH-38254: [Java] Add reusable buffer getters to char/binary vec…
b988d32 
…tors (apache#38266)

### Rationale for this change
Provide a way for a user to reuse a buffer when iterating over byte-array-based ValueVectors to avoid excessive
reallocations.

### What changes are included in this PR?
Add a reusable buffer interface that can be populated by character and binary vectors to avoid allocations when consuming vector content.

Optimize getObject() on VarCharVector/LargeVarCharVector to avoid an extra allocation of a byte array (copy from ArrowBuf directly to the resulting Text).

### Are these changes tested?

### Are there any user-facing changes?

Yes.

**This PR includes breaking changes to public APIs.**

* Closes: apache#38254

Authored-by: James Duong <duong.james@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
loicalleyne pushed a commit to loicalleyne/arrow that referenced this issue Nov 13, 2023
@jduo @loicalleyne
apacheGH-38254: [Java] Add reusable buffer getters to char/binary vec…
b3c9008 
…tors (apache#38266)

### Rationale for this change
Provide a way for a user to reuse a buffer when iterating over byte-array-based ValueVectors to avoid excessive
reallocations.

### What changes are included in this PR?
Add a reusable buffer interface that can be populated by character and binary vectors to avoid allocations when consuming vector content.

Optimize getObject() on VarCharVector/LargeVarCharVector to avoid an extra allocation of a byte array (copy from ArrowBuf directly to the resulting Text).

### Are these changes tested?

### Are there any user-facing changes?

Yes.

**This PR includes breaking changes to public APIs.**

* Closes: apache#38254

Authored-by: James Duong <duong.james@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
dgreiss pushed a commit to dgreiss/arrow that referenced this issue Feb 19, 2024
@jduo @dgreiss
apacheGH-38254: [Java] Add reusable buffer getters to char/binary vec…
e1024ab 
…tors (apache#38266)

### Rationale for this change
Provide a way for a user to reuse a buffer when iterating over byte-array-based ValueVectors to avoid excessive
reallocations.

### What changes are included in this PR?
Add a reusable buffer interface that can be populated by character and binary vectors to avoid allocations when consuming vector content.

Optimize getObject() on VarCharVector/LargeVarCharVector to avoid an extra allocation of a byte array (copy from ArrowBuf directly to the resulting Text).

### Are these changes tested?

### Are there any user-facing changes?

Yes.

**This PR includes breaking changes to public APIs.**

* Closes: apache#38254

Authored-by: James Duong <duong.james@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jduo jduo

Labels
Breaking Change Includes a breaking change to the API Component: Java Type: bug
Projects
None yet
Milestone
15.0.0
Development

Successfully merging a pull request may close this issue.

GH-38254: [Java] Add reusable buffer getters to char/binary vectors jduo/arrow
2 participants
@lidavidm @jduo