Skip to content

feat: Add support for Spark NaNvl math expression#3790

Closed
rafafrdz wants to merge 1 commit intoapache:mainfrom
rafafrdz:feat/add-nanvl-function
Closed

feat: Add support for Spark NaNvl math expression#3790
rafafrdz wants to merge 1 commit intoapache:mainfrom
rafafrdz:feat/add-nanvl-function

Conversation

@rafafrdz
Copy link
Copy Markdown
Contributor

Summary

  • Add support for Spark's NaNvl expression (returns first arg if not NaN, otherwise second arg)
  • Delegates to DataFusion's built-in nanvl function via CometScalarFunction
  • No native Rust code needed — DataFusion's function registry handles execution
  • SQL test added covering NaN, normal values, NULL, and mixed inputs

Test plan

  • SQL test added: nanvl.sql
  • Run CometSqlFileTestSuite to verify results match Spark

Related to #2084

Add NaN value handling function by delegating to DataFusion's built-in
nanvl implementation via CometScalarFunction.
Includes SQL test for NaN, normal values, and NULL inputs.
@rafafrdz
Copy link
Copy Markdown
Contributor Author

Closing this PR — DataFusion's nanvl has different NULL handling semantics than Spark's NaNvl. For the row nanvl(4.0, NULL), DataFusion returns NULL while Spark returns 4.0. This function needs a custom Comet implementation instead of simple delegation.

@rafafrdz rafafrdz closed this Mar 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant