Skip to content

[Bug] pmod rejects string arguments that Spark implicitly casts to a numeric type #23896

Description

@andygrove

Describe the bug

Spark's Pmod declares inputType = NumericType and relies on the analyzer's
string promotion rule, so a string argument is implicitly cast. DataFusion's
SparkPmod uses Signature::numeric(2, Volatility::Immutable), which rejects
strings at planning time.

To Reproduce

Spark 4.2.0 (verified with pyspark==4.2.0), and covered by Spark's own golden
file sql/core/src/test/resources/sql-tests/results/typeCoercion/native/promoteStrings.sql.out:

SELECT pmod('1', CAST(1 AS TINYINT));  -- 0, result type bigint
SELECT pmod('10', CAST(3 AS INT));     -- ansi on: 1 (int); ansi off: 1.0 (double)
SELECT pmod('1', '1');                 -- DATATYPE_MISMATCH.BINARY_OP_WRONG_TYPE

Note that Spark's promotion target differs by mode: with
spark.sql.ansi.enabled = true the string is cast to the other argument's type,
and with it disabled the pair is promoted to double. Two strings are rejected
in both modes.

DataFusion:

SELECT pmod('10'::string, 3::int);
-- Error during planning: For function 'pmod' Utf8 and Int32 are not coercible
-- to a common numeric type.

Expected behavior

A string argument is accepted and coerced the way Spark coerces it.

Scope

This is not specific to pmod. Every datafusion-spark function that uses
Signature::numeric inherits it, and reproducing Spark's mode-dependent
promotion target needs a decision about how far datafusion-spark should go in
emulating Spark's analyzer. Filing it so the gap is visible rather than
proposing a pmod-local workaround.

A commented-out query recording the Spark 4.2.0 result is in
datafusion/sqllogictest/test_files/spark/math/pmod.slt.

Relevant code

datafusion/spark/src/function/math/modulus.rs, SparkPmod::new.

Surfaced by the audit-datafusion-spark-expression skill.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingspark

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions