Skip to content

[SPARK-55731][Streaming] Assign error class for multiple event time columns#55745

Open
XdithyX wants to merge 1 commit intoapache:masterfrom
XdithyX:SPARK-55731
Open

[SPARK-55731][Streaming] Assign error class for multiple event time columns#55745
XdithyX wants to merge 1 commit intoapache:masterfrom
XdithyX:SPARK-55731

Conversation

@XdithyX
Copy link
Copy Markdown
Contributor

@XdithyX XdithyX commented May 7, 2026

What changes were proposed in this pull request?

This PR replaces _LEGACY_ERROR_TEMP_3077 with a proper error class, MULTIPLE_EVENT_TIME_COLUMNS(SPARK-55731).

The error is thrown when a stateful streaming operator receives an input DataFrame with more than one distinct event-time column while multiple event-time columns are not allowed.

This PR:

  • Adds MULTIPLE_EVENT_TIME_COLUMNS to error-conditions.json
  • Assigns SQLSTATE 42K09, matching nearby event-time analysis errors
  • Updates both throw sites in WatermarkSupport
  • Updates the existing EventTimeWatermarkSuite test to verify the new error condition

Why are the changes needed?

_LEGACY_ERROR_TEMP_3077 is a temporary legacy error class and should be replaced with a stable, descriptive error class.

The query behavior is already correct. This change improves the structured error reporting by assigning a real error condition for the multiple event-time columns case.

Does this PR introduce any user-facing change?

Yes.
For this error case, the error condition changes from _LEGACY_ERROR_TEMP_3077 to MULTIPLE_EVENT_TIME_COLUMNS.
The error message remains the same, and the query behavior is unchanged. The error now also has SQLSTATE 42K09.

How was this patch tested?

Tested by running
build/sbt 'sql / Test / testOnly org.apache.spark.sql.streaming.EventTimeWatermarkSuite -- -z "multiple event time columns in an input DataFrame for stateful operator is not allowed"'

Was this patch authored or co-authored using generative AI tooling?

Yes.

@XdithyX
Copy link
Copy Markdown
Contributor Author

XdithyX commented May 7, 2026

Hi @HeartSaVioR Can you please have a look? Thanks!

@XdithyX
Copy link
Copy Markdown
Contributor Author

XdithyX commented May 8, 2026

Hi @cloud-fan Can you please have a look? CI is now green.
Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant