Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Spark-2137][SQL] Timestamp UDFs broken #1081

Closed
wants to merge 2 commits into from
Closed

[Spark-2137][SQL] Timestamp UDFs broken #1081

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
205f17b
Make Hive UDF wrapper support Timestamp.
yhuai Jun 13, 2014
c04f910
Merge remote-tracking branch 'upstream/master' into SPARK-2137
yhuai Jun 13, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -250,7 +250,7 @@ class HiveContext(sc: SparkContext) extends SQLContext(sc) {

protected val primitiveTypes =
Seq(StringType, IntegerType, LongType, DoubleType, FloatType, BooleanType, ByteType,
ShortType, DecimalType)
ShortType, DecimalType, TimestampType)

protected def toHiveString(a: (Any, DataType)): String = a match {
case (struct: Row, StructType(fields)) =>
Expand Down
3 changes: 2 additions & 1 deletion sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUdfs.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,8 @@ private[hive] case class HiveSimpleUdf(name: String, children: Seq[Expression])
val primitiveClasses = Seq(
Integer.TYPE, classOf[java.lang.Integer], classOf[java.lang.String], java.lang.Double.TYPE,
classOf[java.lang.Double], java.lang.Long.TYPE, classOf[java.lang.Long],
classOf[HiveDecimal], java.lang.Byte.TYPE, classOf[java.lang.Byte]
classOf[HiveDecimal], java.lang.Byte.TYPE, classOf[java.lang.Byte],
classOf[java.sql.Timestamp]
)
val matchingConstructor = argClass.getConstructors.find { c =>
c.getParameterTypes.size == 1 && primitiveClasses.contains(c.getParameterTypes.head)
Expand Down
1 change: 1 addition & 0 deletions sql/hive/src/test/resources/golden/timestamp_udf-10-dbc23736a61d9482d13cacada02a7a09
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
2011-05-06 07:08:09.1234567 2011-05-06 02:08:09.1234567
1 change: 1 addition & 0 deletions sql/hive/src/test/resources/golden/timestamp_udf-11-442cf850a0cc1f1dcfdeaeffbffb2c35
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
2011-05-06 07:08:09.1234567 2011-05-06 02:08:09.1234567 2011-05-06 07:08:09.1234567 2011-05-06 02:08:09.1234567
1 change: 1 addition & 0 deletions sql/hive/src/test/resources/golden/timestamp_udf-12-51959036fd4ac4f1e24f4e06eb9b0b6
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
2011-05-06 07:08:09.1234567 2011-05-06 12:08:09.1234567
1 change: 1 addition & 0 deletions sql/hive/src/test/resources/golden/timestamp_udf-13-6ab3f356deaf807e8accc37e1f4849a
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
2011-05-06 07:08:09.1234567 2011-05-06 12:08:09.1234567 2011-05-06 07:08:09.1234567 2011-05-06 12:08:09.1234567
1 change: 1 addition & 0 deletions sql/hive/src/test/resources/golden/timestamp_udf-14-c745a1016461403526d44928a269c1de
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
1304690889 2011 5 6 6 18 7 8 9 2011-05-06
1 change: 1 addition & 0 deletions sql/hive/src/test/resources/golden/timestamp_udf-15-7ab76c4458c7f78038c8b1df0fdeafbe
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
2011-05-11 2011-04-26
1 change: 1 addition & 0 deletions sql/hive/src/test/resources/golden/timestamp_udf-16-b36e87e17ca24d82072220bff559c718
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0 3333 -3333
1 change: 1 addition & 0 deletions sql/hive/src/test/resources/golden/timestamp_udf-17-dad44d2d4a421286e9da080271bd2639
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
2011-05-06 02:08:09.1234567
1 change: 1 addition & 0 deletions sql/hive/src/test/resources/golden/timestamp_udf-18-cb033ecad964a2623bc633ac1d3f752a
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
2011-05-06 12:08:09.1234567
Empty file added sql/hive/src/test/resources/golden/timestamp_udf-19-79914c5347620c6e62a8e0b9a95984af
View file
Open in desktop
Empty file.
Empty file added sql/hive/src/test/resources/golden/timestamp_udf-20-59fc1842a23369235d42ed040d45fb3d
View file
Open in desktop
Empty file.
Empty file added sql/hive/src/test/resources/golden/timestamp_udf-4-80ce02ec84ee8abcb046367ca37279cc
View file
Open in desktop
Empty file.
1 change: 1 addition & 0 deletions sql/hive/src/test/resources/golden/timestamp_udf-5-1124399033bcadf3874fb48f593392d
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
1304690889 2011 5 6 6 18 7 8 9 2011-05-06
1 change: 1 addition & 0 deletions sql/hive/src/test/resources/golden/timestamp_udf-6-5810193ce35d38c23f4fc4b4979d60a4
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
2011-05-11 2011-04-26
1 change: 1 addition & 0 deletions sql/hive/src/test/resources/golden/timestamp_udf-7-250e640a6a818f989f3f3280b00f64f9
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0 3333 -3333
1 change: 1 addition & 0 deletions sql/hive/src/test/resources/golden/timestamp_udf-8-975df43df015d86422965af456f87a94
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
2011-05-06 02:08:09.1234567
1 change: 1 addition & 0 deletions sql/hive/src/test/resources/golden/timestamp_udf-9-287614364eaa3fb82aad08c6b62cc938
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
2011-05-06 12:08:09.1234567