Skip to content

[SPARK-59173][ML] Use isEmpty/nonEmpty instead of size comparisons in MLlib - #58471

Open
uros-b wants to merge 2 commits into
apache:masterfrom
uros-b:mllib-isempty
Open

[SPARK-59173][ML] Use isEmpty/nonEmpty instead of size comparisons in MLlib#58471
uros-b wants to merge 2 commits into
apache:masterfrom
uros-b:mllib-isempty

Conversation

@uros-b

@uros-b uros-b commented Sep 2, 2026

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This replaces size-vs-zero comparisons with the direct emptiness / non-emptiness predicates in the mllib module:

  • Emptiness (x.size == 0 -> x.isEmpty): TreeEnsembleModel.totalImportances (an OpenHashMap, which extends Iterable) and two Seq[ShuffleDependency] checks in ALSSuite.
  • Non-emptiness (x.size > 0 -> x.nonEmpty): DecisionTreeMetadata.featureArity and Strategy.categoricalFeaturesInfo, both Scala Map[Int, Int].

There are no Java-collection cases in this module.

Deliberately left unchanged, because they are not collection-emptiness checks -- they are mllib.linalg.Vector dimensions or numeric sizes/thresholds, and Vector has no isEmpty/nonEmpty:

  • ChiSqTest (expected / observed are Vector dimensions, compared with != 0 and == 0.0);
  • BinaryClassificationPMMLModelExport and GeneralizedLinearPMMLModelExport (model.weights, a Vector);
  • MultivariateOnlineSummarizer (instance, a Vector -- the message reads "Vector should have dimension larger than zero");
  • BlockMatrix (a matrix row Vector);
  • ALS (RatingBlockBuilder.size, a custom builder with no nonEmpty);
  • SpearmanCorrelation (cachedUids.size >= 10000000, a size threshold, not an emptiness check).

Why are the changes needed?

isEmpty / nonEmpty state the intent directly. Behavior is unchanged: every converted receiver (OpenHashMap, Seq, Scala Map) defines the predicate as exactly equivalent to the original comparison.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing tests. This is a behavior-preserving refactor; the mllib module compiles cleanly. Counterpart of SPARK-59149 (SQL) and SPARK-59172 (core).

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.8)

@uros-b uros-b changed the title [SPARK-59173][ML] Use isEmpty instead of size == 0 in MLlib [SPARK-59173][ML] Use isEmpty/nonEmpty instead of size comparisons in MLlib Sep 2, 2026

@uros-b uros-b left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Waiting for CI.

@uros-b

uros-b commented Sep 2, 2026

Copy link
Copy Markdown
Member Author

cc @dongjoon-hyun @HyukjinKwon here, regarding #58449.

@uros-b
uros-b requested a review from HyukjinKwon September 2, 2026 12:30
@uros-b

uros-b commented Sep 2, 2026

Copy link
Copy Markdown
Member Author

added @HyukjinKwon for ML

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants