Skip to content

feat: Wire DataFusion function Claude skill and csc #4337

Open
comphead wants to merge 1 commit into
apache:mainfrom
comphead:df_wire
Open

feat: Wire DataFusion function Claude skill and csc #4337
comphead wants to merge 1 commit into
apache:mainfrom
comphead:df_wire

Conversation

@comphead
Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part #4150

Rationale for this change

Spark csc(x) = 1 / sin(x) was unsupported in Comet, falling back to Spark. The upstream datafusion-spark crate (53.1.0) ships a Spark-compatible SparkCsc implementation, so we can wire it
directly with no native Rust written in this repo.

This PR also adds a new Claude Code skill that codifies the wiring workflow used here, so future contributors can apply the same recipe consistently.

What changes are included in this PR?

Wire Spark Csc to datafusion-spark:

  • spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala — register classOf[Csc] -> CometScalarFunction("csc") in mathExpressions.
  • native/core/src/execution/jni_api.rs — register SparkCsc in register_datafusion_spark_function.
  • docs/source/user-guide/latest/expressions.md — add Csc to the Math Expressions table.
  • docs/source/contributor-guide/spark_expressions_support.md — flip csc checkbox to supported.

Upstream function: datafusion_spark::function::math::trigonometry::SparkCsc. Wiring follows Pattern B (one-line Scala passthrough + one-line UDF registration).

New Claude Code skill wire-datafusion-function:

  • .claude/skills/wire-datafusion-function/SKILL.md — project-level skill that walks contributors through wiring an existing DataFusion or datafusion-spark function into Comet. Documents the three
    wiring patterns (passthrough, UDF registration, custom serde with restrictions), the SQL-test coverage requirements (parquet-backed inputs, NULLs, FP edge cases, integer/decimal min/max), and the
    hand-curated docs that must be updated. Complements the existing implement-comet-expression skill — used here to scaffold the csc wiring.

@comphead comphead requested a review from andygrove May 15, 2026 00:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant