Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-36943][SQL] Improve readability of missing column error message #34202

Closed

Conversation

karenfeng
Copy link
Contributor

What changes were proposed in this pull request?

Improves the quality of the error message encountered by users when they attempt to access a column that does not exist.
Removes the lingo term "resolve" and sorts the suggestions by probability.

Why are the changes needed?

Improves the user experience

Does this PR introduce any user-facing change?

Yes:

Before:

cannot resolve 'foo' given input columns [bar, baz, froo]

After:

Column 'foo' does not exist. Did you mean one of the following? [bar, baz, froo]

How was this patch tested?

Unit tests

…seable grammar

Signed-off-by: Karen Feng <karen.feng@databricks.com>
@karenfeng karenfeng changed the title [SPARK-36943] Improve readability of missing column error message [SPARK-36943][SQL] Improve readability of missing column error message Oct 7, 2021
@SparkQA
Copy link

SparkQA commented Oct 7, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48409/

@SparkQA
Copy link

SparkQA commented Oct 7, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48409/

@SparkQA
Copy link

SparkQA commented Oct 7, 2021

Test build #143897 has finished for PR 34202 at commit c089166.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 7, 2021

Test build #143932 has started for PR 34202 at commit c089166.

@SparkQA
Copy link

SparkQA commented Oct 7, 2021

Test build #143958 has finished for PR 34202 at commit c089166.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 7, 2021

Test build #143987 has finished for PR 34202 at commit c089166.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Signed-off-by: Karen Feng <karen.feng@databricks.com>
@karenfeng
Copy link
Contributor Author

@cloud-fan and @HyukjinKwon, can you help take a look?

@SparkQA
Copy link

SparkQA commented Oct 7, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48478/

@SparkQA
Copy link

SparkQA commented Oct 7, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48478/

@SparkQA
Copy link

SparkQA commented Oct 7, 2021

Test build #144001 has finished for PR 34202 at commit 8bbcc15.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@@ -169,6 +169,37 @@ trait AnalysisTest extends PlanTest {
}
}

protected def assertAnalysisErrorClass(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

qq why does this have Class in the end? Maybe we should just name it something like assertAnalysisErrors vs assertAnalysisError

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's testing error-class ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's why. I thought about changing assertAnalysisError to assertAnalysisErrorMessage, but I think I've altered more than enough code in this PR already.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ohhhh gotya!

private[spark] def orderStringsBySimilarity(
baseString: String,
testStrings: Seq[String]): Seq[String] = {
testStrings.sortWith { (a, b) =>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: testStrings.sortBy(ACLStringUtils.getLevenshteinDistance(_, baseString))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, thanks @cloud-fan!

Signed-off-by: Karen Feng <karen.feng@databricks.com>
@SparkQA
Copy link

SparkQA commented Oct 8, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48512/

@SparkQA
Copy link

SparkQA commented Oct 8, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48512/

@SparkQA
Copy link

SparkQA commented Oct 8, 2021

Test build #144035 has finished for PR 34202 at commit 796edd3.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

thanks, merging to master!

@cloud-fan cloud-fan closed this in d83abfc Oct 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants