Skip to content

Conversation

@Dharin-shah
Copy link
Contributor

@Dharin-shah Dharin-shah commented Nov 15, 2025

This PR fixes 3 small bugs:

  1. When you call asin(2.0) or acos(5.0) (values outside the valid range), PySpark is supposed to return NaN. DuckDB
    was returning NULL instead, which breaks code that expects PySpark behavior. Fixed by using a SQL workaround since
    the internal C++ layer converts NaN to NULL by design.

  2. The spark.read.json() method had all the code written, but a raise NotImplementedError at the end blocked it
    from ever returning. One line removal fixed it.

  3. Instead of hitting a generic NotImplementedError when encountering DuckDB union types (which PySpark doesn't
    support), it now throws a clear ContributionsAcceptedError explaining the limitation.

@Dharin-shah Dharin-shah changed the title Minor bugfixes [minor][spark] Minor bugfixes Nov 15, 2025
@Dharin-shah Dharin-shah marked this pull request as ready for review December 2, 2025 22:20
@evertlammerts evertlammerts changed the base branch from main to v1.4-andium December 8, 2025 07:55
@evertlammerts
Copy link
Collaborator

Thanks! I rebased onto v1.4-andium. When the tests pass I'll do a review.

@evertlammerts evertlammerts merged commit 030a76a into duckdb:v1.4-andium Dec 8, 2025
14 of 15 checks passed
@Dharin-shah Dharin-shah deleted the minor-bugfixes branch December 8, 2025 21:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants