Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Cast String to Decimal could return null incorrectly when scale = precision #10890

Open
thirtiseven opened this issue May 24, 2024 · 0 comments
Assignees
Labels
bug Something isn't working

Comments

@thirtiseven
Copy link
Collaborator

Describe the bug
Cast string to decimal could return null on GPU while a valid value on CPU in some edge cases when scale - precision.

An example: -9.72792462805176E-15 will return null when cast to decimal(15,15) on GPU but -1.0E-14 on CPU.

It seems very edge.

Steps/Code to reproduce bug

scala> import org.apache.spark.sql.functions._
import org.apache.spark.sql.functions._

scala> import org.apache.spark.sql.types._
import org.apache.spark.sql.types._

scala> val data = Seq("-9.72792462805176E-15").toDF
data: org.apache.spark.sql.DataFrame = [value: string]

scala> data.write.mode("OVERWRITE").parquet("TEMP")
24/05/24 08:07:55 WARN GpuOverrides:
*Exec <DataWritingCommandExec> will run on GPU
  *Output <InsertIntoHadoopFsRelationCommand> will run on GPU
  ! <LocalTableScanExec> cannot run on GPU because GPU does not currently support the operator class org.apache.spark.sql.execution.LocalTableScanExec
    @Expression <AttributeReference> value#1 could run on GPU


scala> val df = spark.read.parquet("TEMP")
df: org.apache.spark.sql.DataFrame = [value: string]

scala> df.select(col("value").cast(DecimalType(15, 15))).show()
24/05/24 08:08:13 WARN GpuOverrides:
!Exec <CollectLimitExec> cannot run on GPU because the Exec CollectLimitExec has been disabled, and is disabled by default because Collect Limit replacement can be slower on the GPU, if huge number of rows in a batch it could help by limiting the number of rows transferred from GPU to CPU. Set spark.rapids.sql.exec.CollectLimitExec to true if you wish to enable it
  @Partitioning <SinglePartition$> could run on GPU
  *Exec <ProjectExec> will run on GPU
    *Expression <Alias> cast(cast(value#6 as decimal(15,15)) as string) AS value#12 will run on GPU
      *Expression <Cast> cast(cast(value#6 as decimal(15,15)) as string) will run on GPU
        *Expression <Cast> cast(value#6 as decimal(15,15)) will run on GPU
    *Exec <FileSourceScanExec> will run on GPU

+-----+
|value|
+-----+
| null|
+-----+


scala> spark.conf.set("spark.rapids.sql.enabled", "false")

scala> df.select(col("value").cast(DecimalType(15, 15))).show()
+--------+
|   value|
+--------+
|-1.0E-14|
+--------+

A test cases:

def test_cast_double_to_string_to_decimal():
    assert_gpu_and_cpu_are_equal_collect(
        lambda spark : unary_op_df(spark, double_gen, length=100000).selectExpr(
            "cast(cast(a as string) as decimal(15, 15))"),
            conf = {'spark.rapids.sql.castFloatToDecimal.enabled': 'true',
                    'spark.rapids.sql.castDecimalToFloat.enabled': 'true',
                    'spark.rapids.sql.castFloatToString.enabled': 'true'})

Expected behavior
CPU and GPU results should match.

Environment details (please complete the following information)
latest code, jdk 8, tested spark 330 & 341

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants