Skip to content

[SPARK-45757][ML] Avoid re-computation of NNZ in Binarizer#43619

Closed
zhengruifeng wants to merge 1 commit intoapache:masterfrom
zhengruifeng:ml_binarizer_nnz
Closed

[SPARK-45757][ML] Avoid re-computation of NNZ in Binarizer#43619
zhengruifeng wants to merge 1 commit intoapache:masterfrom
zhengruifeng:ml_binarizer_nnz

Conversation

@zhengruifeng
Copy link
Contributor

@zhengruifeng zhengruifeng commented Nov 1, 2023

What changes were proposed in this pull request?

1, compress vectors with given nnz in Binarizer;

2, rename internal function def compressed(nnz: Int): Vector to avoid ambiguous reference issue (vec.compressed.apply(nnz)) when there is no type hint

[error] /Users/ruifeng.zheng/Dev/spark/mllib/src/main/scala/org/apache/spark/ml/feature/Binarizer.scala:132:61: ambiguous reference to overloaded definition,
[error] both method compressed in trait Vector of type (nnz: Int): org.apache.spark.ml.linalg.Vector
[error] and  method compressed in trait Vector of type org.apache.spark.ml.linalg.Vector

Why are the changes needed?

nnz is known before compression

Does this PR introduce any user-facing change?

no

How was this patch tested?

ci

Was this patch authored or co-authored using generative AI tooling?

no

Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM.

@dongjoon-hyun
Copy link
Member

Merged to master for Apache Spark 4.0.0.

@zhengruifeng zhengruifeng deleted the ml_binarizer_nnz branch November 2, 2023 23:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants