Skip to content

Commit

Permalink
[SPARK-24762][SQL][FOLLOWUP] Enable Option of Product encoders
Browse files Browse the repository at this point in the history
## What changes were proposed in this pull request?

This is follow-up of #21732. This patch inlines `isOptionType` method.

## How was this patch tested?

Existing tests.

Closes #23143 from viirya/SPARK-24762-followup.

Authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
  • Loading branch information
viirya authored and cloud-fan committed Nov 27, 2018
1 parent c995e07 commit 1c487f7
Showing 1 changed file with 3 additions and 6 deletions.
Expand Up @@ -251,19 +251,16 @@ case class ExpressionEncoder[T](
*/
def isSerializedAsStruct: Boolean = objSerializer.dataType.isInstanceOf[StructType]

/**
* Returns true if the type `T` is an `Option` type.
*/
def isOptionType: Boolean = classOf[Option[_]].isAssignableFrom(clsTag.runtimeClass)

/**
* If the type `T` is serialized as a struct, when it is encoded to a Spark SQL row, fields in
* the struct are naturally mapped to top-level columns in a row. In other words, the serialized
* struct is flattened to row. But in case of the `T` is also an `Option` type, it can't be
* flattened to top-level row, because in Spark SQL top-level row can't be null. This method
* returns true if `T` is serialized as struct and is not `Option` type.
*/
def isSerializedAsStructForTopLevel: Boolean = isSerializedAsStruct && !isOptionType
def isSerializedAsStructForTopLevel: Boolean = {
isSerializedAsStruct && !classOf[Option[_]].isAssignableFrom(clsTag.runtimeClass)
}

// serializer expressions are used to encode an object to a row, while the object is usually an
// intermediate value produced inside an operator, not from the output of the child operator. This
Expand Down

0 comments on commit 1c487f7

Please sign in to comment.