Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-20566][SQL][BRANCH-2.2] ColumnVector should support appendFloats for array #18557

Closed
wants to merge 1 commit into from
Closed

[SPARK-20566][SQL][BRANCH-2.2] ColumnVector should support appendFloats for array #18557

wants to merge 1 commit into from

Conversation

dongjoon-hyun
Copy link
Member

What changes were proposed in this pull request?

This PR aims to add a missing appendFloats API for array into ColumnVector class. For double type, there is appendDoubles for array here.

How was this patch tested?

Pass the Jenkins with a newly added test case.

…ats` for array

This PR aims to add a missing `appendFloats` API for array into **ColumnVector** class. For double type, there is `appendDoubles` for array [here](https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVector.java#L818-L824).

Pass the Jenkins with a newly added test case.

Author: Dongjoon Hyun <dongjoon@apache.org>

Closes #17836 from dongjoon-hyun/SPARK-20566.
@SparkQA
Copy link

SparkQA commented Jul 7, 2017

Test build #79306 has finished for PR 18557 at commit 39839bf.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member Author

Hi, @cloud-fan .
This is the backport for #17836 .

@kiszk
Copy link
Member

kiszk commented Jul 7, 2017

@dongjoon-hyun Is there any reason to backport this to previous versions? This is because we had such a discussion.
Obviously, it makes sense to support the latest one.

@dongjoon-hyun
Copy link
Member Author

Hi, @kiszk .
I think this is a bug fix of ColumnVector as described in SPARK-20566.

@kiszk
Copy link
Member

kiszk commented Jul 7, 2017

We have not seen any failure in test suites. And, there is no usage of this API in Spark 2.2.

Does this missing cause any failure of test or application program? If so, it is good to put a sample program in this PR.

@cloud-fan
Copy link
Contributor

I've changed the ticket type from bug to improvement, adding a new API is not fixing a bug.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Jul 7, 2017

I know that 'there is no usage of this API internally in Spark 2.2', but it's only for 2.2.0.
My reason was any 3rd party Spark library cannot use ColumnVector for float type in Spark 2.2.1+. It's performance issue.

Anyway, @cloud-fan changes the issue type. If that means backporting is not allowed for this patch, I have no objection for the community decision. I agree with that, too.

So, @kiszk and @cloud-fan . Given that, may I close this PR?

@dongjoon-hyun
Copy link
Member Author

BTW, thank you for swift reviews and feedbacks on my PR. :)

@cloud-fan
Copy link
Contributor

cloud-fan commented Jul 7, 2017

ColumnVector is totally internal in Spark 2.2, so there won't be 3rd party Spark library issue.

@dongjoon-hyun
Copy link
Member Author

Yep. It's totally internal officially.

What I meant with performance issue is 3rd party can still use it and there might be a performance gap between float and double.

I'll close this PR. Thank you again.

@dongjoon-hyun dongjoon-hyun deleted the SPARK-20566-BRANCH-2.2 branch July 7, 2017 05:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants