[SPARK-13101][SQL] nullability of array type element should not fail analysis of encoder #11035
Conversation
|
|
||
| { | ||
| val attrs = Seq('a.string, 'b.int) |
There was a problem hiding this comment.
The previous tests are very verbose and hard to maintain, every time we change some unrelated analysis behaviours, these tests need to be updated. Actually for these tests, we only care about if analysis passes, runtime execution successes. So I simplified these tests to only ensure we don't fail analysis and execution.
|
Test build #50611 has finished for PR 11035 at commit
|
|
retest this please |
|
Test build #50617 has finished for PR 11035 at commit
|
|
retest this please |
|
I'm investigating the failed flaky tests, and @davies is marking them as ignored temporarily to bring back the PR builder. |
|
Test build #50620 has finished for PR 11035 at commit
|
|
retest it please |
|
LGTM |
| @@ -249,6 +249,8 @@ object ScalaReflection extends ScalaReflection { | |||
|
|
|||
| case t if t <:< localTypeOf[Array[_]] => | |||
| val TypeRef(_, _, Seq(elementType)) = t | |||
|
|
|||
| // TODO: add runtime null check for primitive array | |||
There was a problem hiding this comment.
So this does still silently corrupt values.
scala> Seq(("a", Seq(null, new Integer(1)))).toDS().as[(String, Array[Int])].collect()
res5: Array[(String, Array[Int])] = Array((a,Array(0, 1)))Since this isn't a regression its probably okay as long as we open another JIRA.
There was a problem hiding this comment.
|
Test build #2507 has finished for PR 11035 at commit
|
|
retest this please |
…ld not fail analysis of encoder nullability should only be considered as an optimization rather than part of the type system, so instead of failing analysis for mismatch nullability, we should pass analysis and add runtime null check. backport #11035 to 1.6 Author: Wenchen Fan <wenchen@databricks.com> Closes #11042 from cloud-fan/branch-1.6.
|
retest this please |
|
Test build #50860 has finished for PR 11035 at commit
|
|
retest this please |
|
Test build #50861 has finished for PR 11035 at commit
|
|
retest this please |
|
Test build #50864 has finished for PR 11035 at commit
|
|
retest this please |
|
Test build #50871 has finished for PR 11035 at commit
|
|
retest this please |
|
Test build #50886 has finished for PR 11035 at commit
|
|
cc @marmbrus |
|
Thanks, merging to master. |
nullability should only be considered as an optimization rather than part of the type system, so instead of failing analysis for mismatch nullability, we should pass analysis and add runtime null check.