Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-13101][SQL][branch-1.6] nullability of array type element should not fail analysis of encoder #11042

Closed
wants to merge 1 commit into from

Conversation

cloud-fan
Copy link
Contributor

nullability should only be considered as an optimization rather than part of the type system, so instead of failing analysis for mismatch nullability, we should pass analysis and add runtime null check.

backport #11035 to 1.6

@SparkQA
Copy link

SparkQA commented Feb 3, 2016

Test build #50635 has finished for PR 11042 at commit c2bc287.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Feb 3, 2016

Test build #50639 has finished for PR 11042 at commit 9f8c41e.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan cloud-fan force-pushed the branch-1.6 branch 2 times, most recently from 9845bed to c3f659b Compare February 3, 2016 05:53
@cloud-fan
Copy link
Contributor Author

The error message looks like:

[info]   Cause: java.lang.RuntimeException: Null value appeared in non-nullable field:
[info] - field (class: "scala.Int", name: "b")
[info] - field (class: "org.apache.spark.sql.ClassData", name: "f")
[info] - root class: "org.apache.spark.sql.NestedStruct"
[info] If the schema is inferred from a Scala tuple/case class, or a Java bean, please try to use scala.Option[_] or other nullable types (e.g. java.lang.Integer instead of int/scala.Int).

@liancheng
Copy link
Contributor

Memo: CodegenContext.references can only hold Expression objects in branch-1.6. That's why the error message can't be moved to codegen references like what #11035 does.

@SparkQA
Copy link

SparkQA commented Feb 3, 2016

Test build #50648 has finished for PR 11042 at commit 6b5a100.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@liancheng
Copy link
Contributor

LGTM

@marmbrus
Copy link
Contributor

marmbrus commented Feb 4, 2016

Thanks, merging.

asfgit pushed a commit that referenced this pull request Feb 4, 2016
…ld not fail analysis of encoder

nullability should only be considered as an optimization rather than part of the type system, so instead of failing analysis for mismatch nullability, we should pass analysis and add runtime null check.

backport #11035 to 1.6

Author: Wenchen Fan <wenchen@databricks.com>

Closes #11042 from cloud-fan/branch-1.6.
@cloud-fan cloud-fan closed this Feb 5, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants