Skip to content

[spark] Remove upsert-key support - #9093

Merged
JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/remove-upsert-key
Aug 7, 2026
Merged

[spark] Remove upsert-key support#9093
JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/remove-upsert-key

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Purpose

Remove the Spark upsert-key table property and its automatic INSERT INTO to MERGE INTO rewrite. It's just syntactic sugar; we shouldn't provide it because it has a lot of ambiguity with primary key tables.

This change removes the option and accessor from CoreOptions, the associated schema validation, the Spark analyzer rule and registration, and the Spark 3.2 through 4.1 tests. It also removes the SQL Upsert documentation, navigation entry, redirect, and generated configuration entry.

Existing tables that specify upsert-key will no longer receive the automatic Spark upsert rewrite. The independent PyPaimon Data Evolution upsert_by_key API is not affected.

Tests

  • mvn -pl paimon-core -Pfast-build -DwildcardSuites=none -Dtest=SchemaValidationTest test
  • mvn -pl paimon-spark/paimon-spark-ut -am -Pfast-build -Pspark3 -DskipTests compile
  • JAVA_HOME=<jdk17> mvn -pl paimon-spark/paimon-spark-ut -am -Pfast-build -Pspark4 -DskipTests compile
  • mvn -pl paimon-api,paimon-core,paimon-spark/paimon-spark-common -Pspark3 -DskipTests spotless:check

@JingsongLi
JingsongLi marked this pull request as ready for review August 7, 2026 07:30

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. I verified the complete removal of the Spark upsert-key option, analyzer rules, version-specific tests, and documentation references. SchemaValidationTest and Spark 3/Spark 4 compilation passed locally.

@JingsongLi
JingsongLi merged commit a3fc1b7 into apache:master Aug 7, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants