Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-20105][TESTS][R] Add tests for checkType and type string in structField in R #17439

Closed
wants to merge 2 commits into from

Conversation

HyukjinKwon
Copy link
Member

What changes were proposed in this pull request?

It seems checkType and the type string in structField are not being tested closely. This string format currently seems SparkR-specific (see

def getSQLDataType(dataType: String): DataType = {
dataType match {
case "byte" => org.apache.spark.sql.types.ByteType
case "integer" => org.apache.spark.sql.types.IntegerType
case "float" => org.apache.spark.sql.types.FloatType
case "double" => org.apache.spark.sql.types.DoubleType
case "numeric" => org.apache.spark.sql.types.DoubleType
case "character" => org.apache.spark.sql.types.StringType
case "string" => org.apache.spark.sql.types.StringType
case "binary" => org.apache.spark.sql.types.BinaryType
case "raw" => org.apache.spark.sql.types.BinaryType
case "logical" => org.apache.spark.sql.types.BooleanType
case "boolean" => org.apache.spark.sql.types.BooleanType
case "timestamp" => org.apache.spark.sql.types.TimestampType
case "date" => org.apache.spark.sql.types.DateType
case r"\Aarray<(.+)${elemType}>\Z" =>
org.apache.spark.sql.types.ArrayType(getSQLDataType(elemType))
case r"\Amap<(.+)${keyType},(.+)${valueType}>\Z" =>
if (keyType != "string" && keyType != "character") {
throw new IllegalArgumentException("Key type of a map must be string or character")
}
org.apache.spark.sql.types.MapType(getSQLDataType(keyType), getSQLDataType(valueType))
case r"\Astruct<(.+)${fieldsStr}>\Z" =>
if (fieldsStr(fieldsStr.length - 1) == ',') {
throw new IllegalArgumentException(s"Invalid type $dataType")
}
val fields = fieldsStr.split(",")
val structFields = fields.map { field =>
field match {
case r"\A(.+)${fieldName}:(.+)${fieldType}\Z" =>
createStructField(fieldName, fieldType, true)
case _ => throw new IllegalArgumentException(s"Invalid type $dataType")
}
}
createStructType(structFields)
case _ => throw new IllegalArgumentException(s"Invalid type $dataType")
}
}
) but resembles SQL type definition.

Therefore, it seems nicer if we test positive/negative cases in R side.

How was this patch tested?

Unit tests in test_sparkSQL.R.

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Mar 27, 2017

cc @felixcheung, this adds the tests for the current status. Probably, let me try to investigate and propose the changes about the type definition thing we talked about later. I guess the branch is going to be cut out soon.

This PR only adds the positive/negative test cases for both structField and checkType.

@HyukjinKwon HyukjinKwon changed the title [SPARK-20105][R] Add tests for checkType and type string in structField in R [SPARK-20105][TESTS][R] Add tests for checkType and type string in structField in R Mar 27, 2017
@SparkQA
Copy link

SparkQA commented Mar 27, 2017

Test build #75249 has finished for PR 17439 at commit b841ab3.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Mar 27, 2017

Test build #75251 has finished for PR 17439 at commit 32ba2e1.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@felixcheung
Copy link
Member

merged to master, thanks!

@asfgit asfgit closed this in 3fada2f Mar 27, 2017
@HyukjinKwon HyukjinKwon deleted the r-typestring-tests branch January 2, 2018 03:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants