Skip to content

Native engine panics on all-scalar inputs for hour, minute, second, unix_timestamp #3336

@andygrove

Description

@andygrove

Description

When Spark's ConstantFolding optimizer rule is disabled, all-scalar (literal-only) expressions reach Comet's native engine instead of being folded at plan time. The native engine panics with messages like:

  • hour(scalar) should be fold in Spark JVM side.
  • minute(scalar) should be fold in Spark JVM side.
  • second(scalar) should be fold in Spark JVM side.
  • unix_timestamp(scalar) should be fold in Spark JVM side.

How to Reproduce

Disable constant folding and run an all-literal datetime extraction query:

SET spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.ConstantFolding;
SELECT hour(timestamp('2024-01-15 12:30:45'));

Expected Behavior

The native engine should handle scalar inputs gracefully — either by computing the correct result or by falling back to Spark, not by panicking.

Affected Expressions

  • hour()
  • minute()
  • second()
  • unix_timestamp()

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions