[SPARK-8993][SQL] More comprehensive type checking in expressions. #7348

rxin · 2015-07-10T22:57:11Z

This patch makes the following changes:

ExpectsInputTypes only defines expected input types, but does not perform any implicit type casting.
ImplicitCastInputTypes is a new trait that defines both expected input types, as well as performs implicit type casting.
BinaryOperator has a new abstract function "inputType", which defines the expected input type for both left/right. Concrete BinaryOperator expressions no longer perform any implicit type casting.
For BinaryOperators, convert NullType (i.e. null literals) into some accepted type so BinaryOperators don't need to handle NullTypes.

TODOs needed: fix unit tests for error reporting.

I'm intentionally not changing anything in aggregate expressions because @yhuai is doing a big refactoring on that right now.

rxin · 2015-07-10T22:57:52Z

cc @cloud-fan

I had this change on my laptop I didn't commit, which I believe actually encompasses your change. #7338

rxin · 2015-07-10T22:58:25Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala

+  override def checkInputDataTypes(): TypeCheckResult = {
+    // First call the checker for ExpectsInputTypes, and then check whether left and right have
+    // the same type.
+    super.checkInputDataTypes() match {


note that this will use expects input types to do the type check, if available.

SparkQA · 2015-07-11T00:24:39Z

Test build #37063 has finished for PR 7348 at commit d915899.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnaryMinus(child: Expression) extends UnaryArithmetic with ExpectsInputTypes
- case class Abs(child: Expression) extends UnaryArithmetic with ExpectsInputTypes
- case class Add(left: Expression, right: Expression)
- case class Subtract(left: Expression, right: Expression)
- case class Multiply(left: Expression, right: Expression)
- case class Divide(left: Expression, right: Expression)
- case class Remainder(left: Expression, right: Expression)
- case class BitwiseAnd(left: Expression, right: Expression)
- case class BitwiseOr(left: Expression, right: Expression)
- case class BitwiseXor(left: Expression, right: Expression)
- case class BitwiseNot(child: Expression) extends UnaryArithmetic with ExpectsInputTypes

cloud-fan · 2015-07-11T02:51:47Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala

+        } else {
+          TypeCheckResult.TypeCheckSuccess
+        }
+      case TypeCheckResult.TypeCheckFailure(msg) => TypeCheckResult.TypeCheckFailure(msg)


Can we simplify this to case fail => fail?

that's a good idea

JoshRosen · 2015-07-11T23:52:40Z

I've started writing a very basic fuzz-testing suite to check that expression code generation does not fail if an expression is initialized with NullType-d expressions as children and passes type checks after type coercion is applied: master...JoshRosen:fuzz-test

Running that suite against this patch shows that it fixes 9 of my test's failures. I wouldn't read too much into this for now, since my suite needs some work to be more useful, but just thought I'd share since it's a neat testing technique and uncovered a few bugs that I had previously found via other methods (such as sum(null) with a null literal).

JoshRosen · 2015-07-11T23:56:52Z

It's worth noting that a bunch of the errors in that fuzzing suite are caused by the fact that we skip type checks for certain internal expressions, such as UnscaledValue or AddItemToSet. Is there a good reason why we shouldn't add type checking to these expressions, if only to act as an internal sanity check?

rxin · 2015-07-12T05:14:20Z

Those two expressions should / will just go away.

chenghao-intel · 2015-07-13T03:04:42Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala

+ *
+ * 1. The string representation is "x symbol y", rather than "funcName(x, y)".
+ * 2. Two inputs are expected to the be same type. If the two inputs have different types,
+ *    the analyzer will find the tightest common type and do the proper type casting.
 */
 abstract class BinaryOperator extends BinaryExpression {
  self: Product =>

  def symbol: String


Should we remove this function if we removed the function toString. Let the subclass define the symbol as needed.

that's a mistake actually. i'm adding tostirng back

cloud-fan · 2015-07-14T06:27:59Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala

@@ -48,6 +49,9 @@ case class UnaryMinus(child: Expression) extends UnaryArithmetic {
 case class UnaryPositive(child: Expression) extends UnaryArithmetic {
  override def prettyName: String = "positive"

+  override def checkInputDataTypes(): TypeCheckResult =
+    TypeUtils.checkForNumericExpr(child.dataType, "operator -")


should be "operator +"?

SparkQA · 2015-07-14T06:40:49Z

Test build #37202 has finished for PR 7348 at commit e4727cc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class And(left: Expression, right: Expression) extends BinaryOperator with Predicate
- case class Or(left: Expression, right: Expression) extends BinaryOperator with Predicate

cloud-fan · 2015-07-14T06:42:20Z

sql/catalyst/src/main/scala/org/apache/spark/sql/types/AbstractDataType.scala

+   * Types that can be used in bitwise operations.
+   */
+  val Bitwise = TypeCollection(
+    BooleanType,


Does boolean support bitwise?

Does boolean support bitwise operations?

SparkQA · 2015-07-14T07:00:05Z

Test build #37210 has finished for PR 7348 at commit 4932d57.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait ImplicitCastInputTypes extends ExpectsInputTypes
- abstract class BinaryOperator extends BinaryExpression with ExpectsInputTypes
- case class UnaryMinus(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class UnaryPositive(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Abs(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class BitwiseNot(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Factorial(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Hex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Unhex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Md5(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Sha1(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Crc32(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Not(child: Expression)
- case class And(left: Expression, right: Expression) extends BinaryOperator with Predicate
- case class Or(left: Expression, right: Expression) extends BinaryOperator with Predicate
- trait StringRegexExpression extends ImplicitCastInputTypes
- trait String2StringExpression extends ImplicitCastInputTypes
- trait StringComparison extends ImplicitCastInputTypes
- case class StringSpace(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class StringLength(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Ascii(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Base64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class UnBase64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes

rxin · 2015-07-14T18:53:45Z

Jenkins, retest this please.

SparkQA · 2015-07-14T19:16:21Z

Test build #37248 has finished for PR 7348 at commit 4932d57.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait ImplicitCastInputTypes extends ExpectsInputTypes
- abstract class BinaryOperator extends BinaryExpression with ExpectsInputTypes
- case class UnaryMinus(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class UnaryPositive(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Abs(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class BitwiseNot(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Factorial(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Hex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Unhex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Md5(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Sha1(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Crc32(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Not(child: Expression)
- case class And(left: Expression, right: Expression) extends BinaryOperator with Predicate
- case class Or(left: Expression, right: Expression) extends BinaryOperator with Predicate
- trait StringRegexExpression extends ImplicitCastInputTypes
- trait String2StringExpression extends ImplicitCastInputTypes
- trait StringComparison extends ImplicitCastInputTypes
- case class StringSpace(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class StringLength(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Ascii(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Base64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class UnBase64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes

SparkQA · 2015-07-14T21:40:22Z

Test build #37257 has finished for PR 7348 at commit 2e22330.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait ImplicitCastInputTypes extends ExpectsInputTypes
- abstract class BinaryOperator extends BinaryExpression with ExpectsInputTypes
- case class UnaryMinus(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class UnaryPositive(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Abs(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class BitwiseNot(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Factorial(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Hex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Unhex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Md5(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Sha1(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Crc32(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Not(child: Expression)
- case class And(left: Expression, right: Expression) extends BinaryOperator with Predicate
- case class Or(left: Expression, right: Expression) extends BinaryOperator with Predicate
- trait StringRegexExpression extends ImplicitCastInputTypes
- trait String2StringExpression extends ImplicitCastInputTypes
- trait StringComparison extends ImplicitCastInputTypes
- case class StringSpace(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class StringLength(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Ascii(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Base64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class UnBase64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes

SparkQA · 2015-07-14T23:29:53Z

Test build #37273 has finished for PR 7348 at commit fb66657.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait ImplicitCastInputTypes extends ExpectsInputTypes
- abstract class BinaryOperator extends BinaryExpression with ExpectsInputTypes
- case class UnaryMinus(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class UnaryPositive(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Abs(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class BitwiseNot(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Factorial(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Hex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Unhex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Md5(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Sha1(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Crc32(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Not(child: Expression)
- case class And(left: Expression, right: Expression) extends BinaryOperator with Predicate
- case class Or(left: Expression, right: Expression) extends BinaryOperator with Predicate
- trait StringRegexExpression extends ImplicitCastInputTypes
- trait String2StringExpression extends ImplicitCastInputTypes
- trait StringComparison extends ImplicitCastInputTypes
- case class StringSpace(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class StringLength(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Ascii(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Base64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class UnBase64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes

SparkQA · 2015-07-15T00:56:29Z

Test build #37280 has finished for PR 7348 at commit 360d124.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait ImplicitCastInputTypes extends ExpectsInputTypes
- abstract class BinaryOperator extends BinaryExpression with ExpectsInputTypes
- case class UnaryMinus(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class UnaryPositive(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Abs(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class BitwiseNot(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Factorial(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Hex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Unhex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Md5(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Sha1(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Crc32(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Not(child: Expression)
- case class And(left: Expression, right: Expression) extends BinaryOperator with Predicate
- case class Or(left: Expression, right: Expression) extends BinaryOperator with Predicate
- trait StringRegexExpression extends ImplicitCastInputTypes
- trait String2StringExpression extends ImplicitCastInputTypes
- trait StringComparison extends ImplicitCastInputTypes
- case class StringSpace(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class StringLength(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Ascii(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Base64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class UnBase64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes

SparkQA · 2015-07-15T01:32:40Z

Test build #37294 has finished for PR 7348 at commit 438ea07.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait ImplicitCastInputTypes extends ExpectsInputTypes
- abstract class BinaryOperator extends BinaryExpression with ExpectsInputTypes
- case class UnaryMinus(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class UnaryPositive(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Abs(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class BitwiseNot(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Factorial(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Hex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Unhex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Md5(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Sha1(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Crc32(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Not(child: Expression)
- case class And(left: Expression, right: Expression) extends BinaryOperator with Predicate
- case class Or(left: Expression, right: Expression) extends BinaryOperator with Predicate
- trait StringRegexExpression extends ImplicitCastInputTypes
- trait String2StringExpression extends ImplicitCastInputTypes
- trait StringComparison extends ImplicitCastInputTypes
- case class StringSpace(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class StringLength(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Ascii(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Base64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class UnBase64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes

SparkQA · 2015-07-15T02:15:28Z

Test build #37296 has finished for PR 7348 at commit 3bb63e7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait ImplicitCastInputTypes extends ExpectsInputTypes
- abstract class BinaryOperator extends BinaryExpression with ExpectsInputTypes
- case class UnaryMinus(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class UnaryPositive(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Abs(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class BitwiseNot(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Factorial(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Hex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Unhex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Md5(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Sha1(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Crc32(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Not(child: Expression)
- case class And(left: Expression, right: Expression) extends BinaryOperator with Predicate
- case class Or(left: Expression, right: Expression) extends BinaryOperator with Predicate
- trait StringRegexExpression extends ImplicitCastInputTypes
- trait String2StringExpression extends ImplicitCastInputTypes
- trait StringComparison extends ImplicitCastInputTypes
- case class StringSpace(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class StringLength(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Ascii(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Base64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class UnBase64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes

SparkQA · 2015-07-15T03:48:04Z

Test build #37302 has finished for PR 7348 at commit 8fcf814.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait ImplicitCastInputTypes extends ExpectsInputTypes
- abstract class BinaryOperator extends BinaryExpression with ExpectsInputTypes
- case class UnaryMinus(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class UnaryPositive(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Abs(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class BitwiseNot(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Factorial(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Hex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Unhex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Md5(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Sha1(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Crc32(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Not(child: Expression)
- case class And(left: Expression, right: Expression) extends BinaryOperator with Predicate
- case class Or(left: Expression, right: Expression) extends BinaryOperator with Predicate
- trait StringRegexExpression extends ImplicitCastInputTypes
- trait String2StringExpression extends ImplicitCastInputTypes
- trait StringComparison extends ImplicitCastInputTypes
- case class StringSpace(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class StringLength(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Ascii(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Base64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class UnBase64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes

rxin · 2015-07-15T05:35:33Z

Jenkins, retest this please.

rxin · 2015-07-15T05:52:32Z

The test failure looks unrelated. I'm going to merge this and fix any problems later, since it changes a lot of code.

rxin · 2015-07-15T05:53:17Z

@cloud-fan can you review this and help me fix the ignored test cases? Thanks.

chenghao-intel · 2015-07-15T06:16:11Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala

+        val children: Seq[Expression] = e.children.zip(e.inputTypes).map { case (in, expected) =>
+          if (in.dataType == NullType && !expected.acceptsType(NullType)) {
+            Cast(in, expected.defaultConcreteType)
+          } else {


What if expected.acceptsType(in.dateType) == false, probably we'd better to raise a TypeChecking exception?

that happens during CheckAnalysis when we report errors.

SparkQA · 2015-07-15T07:05:07Z

Test build #37320 has finished for PR 7348 at commit 8fcf814.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait ImplicitCastInputTypes extends ExpectsInputTypes
- abstract class BinaryOperator extends BinaryExpression with ExpectsInputTypes
- case class UnaryMinus(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class UnaryPositive(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Abs(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class BitwiseNot(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Factorial(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Hex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Unhex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Md5(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Sha1(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Crc32(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Not(child: Expression)
- case class And(left: Expression, right: Expression) extends BinaryOperator with Predicate
- case class Or(left: Expression, right: Expression) extends BinaryOperator with Predicate
- trait StringRegexExpression extends ImplicitCastInputTypes
- trait String2StringExpression extends ImplicitCastInputTypes
- trait StringComparison extends ImplicitCastInputTypes
- case class StringSpace(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class StringLength(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Ascii(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Base64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class UnBase64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes

based on #7348 Author: Wenchen Fan <cloud0fan@outlook.com> Closes #7420 from cloud-fan/type-check and squashes the following commits: 7633fa9 [Wenchen Fan] revert fe169b0 [Wenchen Fan] improve test 03b70da [Wenchen Fan] enhance implicit type cast

rxin reviewed Jul 10, 2015
View reviewed changes

rxin changed the title ~~[WIP] Improve expression type checking.~~ [SPARK-8993][SQL] Leverage ExpectsInputTypes in as many expressions as possible Jul 10, 2015

rxin changed the title ~~[SPARK-8993][SQL] Leverage ExpectsInputTypes in as many expressions as possible~~ [SPARK-8993][SQL] [WIP] Leverage ExpectsInputTypes in as many expressions as possible Jul 10, 2015

cloud-fan reviewed Jul 11, 2015
View reviewed changes

cloud-fan mentioned this pull request Jul 11, 2015

[SPARK-8969][SQL] move type-check from BinaryArithmetic and BinaryComparison to BinaryOperator #7338

Closed

Improve expression type checking.

d017861

chenghao-intel reviewed Jul 13, 2015
View reviewed changes

BinaryOperator should not be doing implicit cast.

e4727cc

rxin force-pushed the typecheck branch from d915899 to e4727cc Compare July 14, 2015 06:18

cloud-fan reviewed Jul 14, 2015
View reviewed changes

Rename existing ExpectsInputTypes -> ImplicitCastInputTypes.

d061691

Style fix.

4932d57

rxin changed the title ~~[SPARK-8993][SQL] [WIP] Leverage ExpectsInputTypes in as many expressions as possible~~ [SPARK-8993][SQL] More comprehensive type checking in expressions. Jul 14, 2015

cloud-fan reviewed Jul 14, 2015
View reviewed changes

Fixed unit tests.

2e22330

rxin added 2 commits July 14, 2015 14:52

Convert NullType into some accepted type for BinaryOperators.

fb66657

Fixed the rule.

360d124

rxin added 2 commits July 14, 2015 18:22

Removes NullTypes.

d55c9e5

space

438ea07

rxin added 3 commits July 14, 2015 18:47

Moved RemoveNullTypes into ImplicitTypeCasts.

aa7790e

Comment update.

f45408f

Style fix.

3bb63e7

Fixed ordering of cases.

8fcf814

rxin mentioned this pull request Jul 15, 2015

[SPARK-8945][SQL] Add add and subtract expressions for IntervalType #7398

Closed

asfgit closed this in f23a721 Jul 15, 2015

chenghao-intel reviewed Jul 15, 2015
View reviewed changes

rxin mentioned this pull request Jul 15, 2015

[SPARK-8221][SQL]Add pmod function #6783

Closed

cloud-fan mentioned this pull request Jul 15, 2015

[SPARK-9068][SQL] refactor the implicit type cast code #7420

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-8993][SQL] More comprehensive type checking in expressions. #7348

[SPARK-8993][SQL] More comprehensive type checking in expressions. #7348

rxin commented Jul 10, 2015

rxin commented Jul 10, 2015

rxin Jul 10, 2015

SparkQA commented Jul 11, 2015

cloud-fan Jul 11, 2015

rxin Jul 14, 2015

JoshRosen commented Jul 11, 2015

JoshRosen commented Jul 11, 2015

rxin commented Jul 12, 2015

chenghao-intel Jul 13, 2015

rxin Jul 13, 2015

cloud-fan Jul 14, 2015

SparkQA commented Jul 14, 2015

cloud-fan Jul 14, 2015

cloud-fan Jul 14, 2015

rxin Jul 14, 2015

SparkQA commented Jul 14, 2015

rxin commented Jul 14, 2015

SparkQA commented Jul 14, 2015

SparkQA commented Jul 14, 2015

SparkQA commented Jul 14, 2015

SparkQA commented Jul 15, 2015

SparkQA commented Jul 15, 2015

SparkQA commented Jul 15, 2015

SparkQA commented Jul 15, 2015

rxin commented Jul 15, 2015

rxin commented Jul 15, 2015

rxin commented Jul 15, 2015

chenghao-intel Jul 15, 2015

rxin Jul 15, 2015

SparkQA commented Jul 15, 2015

[SPARK-8993][SQL] More comprehensive type checking in expressions. #7348

[SPARK-8993][SQL] More comprehensive type checking in expressions. #7348

Conversation

rxin commented Jul 10, 2015

rxin commented Jul 10, 2015

Choose a reason for hiding this comment

SparkQA commented Jul 11, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoshRosen commented Jul 11, 2015

JoshRosen commented Jul 11, 2015

rxin commented Jul 12, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jul 14, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jul 14, 2015

rxin commented Jul 14, 2015

SparkQA commented Jul 14, 2015

SparkQA commented Jul 14, 2015

SparkQA commented Jul 14, 2015

SparkQA commented Jul 15, 2015

SparkQA commented Jul 15, 2015

SparkQA commented Jul 15, 2015

SparkQA commented Jul 15, 2015

rxin commented Jul 15, 2015

rxin commented Jul 15, 2015

rxin commented Jul 15, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jul 15, 2015