[SPARK-38732][SQL][TESTS] Test the error class: INCOMPARABLE_PIVOT_COLUMN#36320
Closed
lvshaokang wants to merge 6 commits intoapache:masterfrom
Closed
[SPARK-38732][SQL][TESTS] Test the error class: INCOMPARABLE_PIVOT_COLUMN#36320lvshaokang wants to merge 6 commits intoapache:masterfrom
lvshaokang wants to merge 6 commits intoapache:masterfrom
Conversation
### What changes were proposed in this pull request?
In the PR, I propose to prepend the error class tag to error messages. The `SparkThrowableHelper.getMessage()` method was modified for this purpose:
```scala
def getMessage(errorClass: String, messageParameters: Array[String]): String = {
...
"[" + errorClass + "] " + String.format ...
}
```
Other changes are related to tests where the expected error messages were extended by error class tags. Also I introduced new base trait `QueryErrorsSuiteBase` for test suites w/ two helper methods:
```scala
def checkErrorClass(
exception: Exception with SparkThrowable,
errorClass: String,
msg: String,
sqlState: Option[String] = None,
matchMsg: Boolean = false): Unit
def validateParsingError(
sqlText: String,
errorClass: String,
sqlState: String,
message: String): Unit
```
The methods are supposed to be used in `QueryParsingErrorsSuite`, `QueryCompilationErrorsSuite`, `QueryExecutionErrorsSuite` and so on, to check error classes, messages, and sqlStates.
### Why are the changes needed?
The error class tags will be used as unique identifiers to google issues, and refer to Spark docs.
### Does this PR introduce _any_ user-facing change?
Yes, the PR changes user-facing error messages.
### How was this patch tested?
By running the modified test suites:
```
$ build/sbt "sql/testOnly *QueryExecutionErrorsSuite"
$ build/sbt "test:testOnly *QueryCompilationErrorsSuite"
$ build/sbt "test:testOnly *QueryCompilationErrorsDSv2Suite"
$ build/sbt "testOnly *QueryParsingErrorsSuite"
$ build/sbt "sql/testOnly *SQLQueryTestSuite"
```
Closes apache#36287 from MaxGekk/output-error-class-tag.
Authored-by: Max Gekk <max.gekk@gmail.com>
Signed-off-by: Max Gekk <max.gekk@gmail.com>
### What changes were proposed in this pull request? This PR proposes to add an example of `StreamingQueryListener` in Python addd in SPARK-38759. ### Why are the changes needed? To guide users about the new feature `StreamingQueryListener`. ### Does this PR introduce _any_ user-facing change? Yes, new example in the documentation: <img width="947" alt="Screen Shot 2022-04-22 at 2 01 36 PM" src="https://user-images.githubusercontent.com/6477701/164606564-2aaeb912-7c09-463d-a567-d4d52cab8c97.png"> ### How was this patch tested? Manually built the docs and checked the output. Closes apache#36318 from HyukjinKwon/SPARK-38994. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
MaxGekk
requested changes
Apr 22, 2022
Comment on lines
+395
to
+398
| assert(e.getErrorClass === "INCOMPARABLE_PIVOT_COLUMN") | ||
| assert(e.getSqlState === "42000") | ||
| assert(e.getMessage.matches("Invalid pivot column 'map.*\\'. " + | ||
| "Pivot columns must be comparable.")) |
Member
There was a problem hiding this comment.
Could you use the checkErrorClass method added by #36287. BTW, the method checks the tag in error messages.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
I add a test case for the error class INCOMPARABLE_PIVOT_COLUMN in the QueryExecutionErrorsSuite.
Why are the changes needed?
To improve test coverage, and record expected error messages in tests.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
By existing tests.