[SPARK-8227][SQL]Add function unhex #7113

zhichao-li · 2015-06-30T06:22:15Z

cc @chenghao-intel @adrian-wang

AmplabJenkins · 2015-06-30T06:27:13Z

Merged build triggered.

AmplabJenkins · 2015-06-30T06:27:18Z

Merged build started.

SparkQA · 2015-06-30T06:30:20Z

Test build #36100 has started for PR 7113 at commit c852d46.

SparkQA · 2015-06-30T06:32:08Z

Test build #36100 has finished for PR 7113 at commit c852d46.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnHex(child: Expression)

AmplabJenkins · 2015-06-30T06:32:09Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-06-30T06:52:13Z

Merged build triggered.

AmplabJenkins · 2015-06-30T06:52:19Z

Merged build started.

SparkQA · 2015-06-30T06:52:48Z

Test build #36109 has started for PR 7113 at commit 11945c7.

SparkQA · 2015-06-30T06:56:48Z

Test build #36109 has finished for PR 7113 at commit 11945c7.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-06-30T06:56:50Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-06-30T07:07:11Z

Merged build triggered.

AmplabJenkins · 2015-06-30T07:07:20Z

Merged build started.

SparkQA · 2015-06-30T07:10:31Z

Test build #36111 has started for PR 7113 at commit cde73f5.

SparkQA · 2015-06-30T08:36:44Z

Test build #36111 has finished for PR 7113 at commit cde73f5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnHex(child: Expression)

AmplabJenkins · 2015-06-30T08:36:55Z

Merged build finished. Test FAILed.

davies · 2015-06-30T22:16:00Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/math.scala

+    if (num == null) {
+      null
+    } else {
+      unhex(num.asInstanceOf[UTF8String].toString)


Converting UTF8String to String is slow, please use getBytes

AmplabJenkins · 2015-07-01T02:53:11Z

Merged build triggered.

AmplabJenkins · 2015-07-01T02:53:18Z

Merged build started.

SparkQA · 2015-07-01T02:56:27Z

Test build #36223 has started for PR 7113 at commit bffd37f.

SparkQA · 2015-07-01T03:07:09Z

Test build #36223 has finished for PR 7113 at commit bffd37f.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnHex(child: Expression)

AmplabJenkins · 2015-07-01T03:07:14Z

Merged build finished. Test FAILed.

zhichao-li · 2015-07-01T03:11:11Z

reset this please

tarekbecker · 2015-07-01T04:00:27Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/math.scala

+case class UnHex(child: Expression)
+  extends UnaryExpression with AutoCastInputTypes with Serializable  {
+
+  override def expectedChildTypes: Seq[DataType] = Seq(StringType)


@davies this could be changed to ByteType, couldn't it? This avoids a cast, if someone calls this with a byte array.

hex() return StringType, so unhex() should take StringType, not BinaryType.

AmplabJenkins · 2015-07-01T13:18:11Z

Merged build triggered.

AmplabJenkins · 2015-07-01T13:18:20Z

Merged build started.

SparkQA · 2015-07-01T13:18:47Z

Test build #36262 has started for PR 7113 at commit a4ae6dc.

SparkQA · 2015-07-01T14:55:14Z

Test build #36262 has finished for PR 7113 at commit a4ae6dc.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnHex(child: Expression) extends UnaryExpression with Serializable

AmplabJenkins · 2015-07-01T14:55:47Z

Merged build finished. Test PASSed.

davies · 2015-07-01T15:20:55Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/math.scala

+  }
+
+  private def toDigit(b: Byte): Byte = {
+    val digit = hexDigits(b)


This should be a val of UnHex.

AmplabJenkins · 2015-07-02T01:33:11Z

Merged build triggered.

AmplabJenkins · 2015-07-02T01:33:21Z

Merged build started.

SparkQA · 2015-07-02T01:36:34Z

Test build #36320 has started for PR 7113 at commit 379356e.

SparkQA · 2015-07-02T03:12:06Z

Test build #36320 has finished for PR 7113 at commit 379356e.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnHex(child: Expression) extends UnaryExpression with Serializable

AmplabJenkins · 2015-07-02T03:12:54Z

Merged build finished. Test PASSed.

davies · 2015-07-02T05:19:22Z

LGTM, merging this into master, thanks!

rxin · 2015-07-02T05:23:09Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/math.scala

+ * Performs the inverse operation of HEX.
+ * Resulting characters are returned as a byte array.
+ */
+case class UnHex(child: Expression) extends UnaryExpression with Serializable {


this should just implement ExpectsInputTypes, and use BinaryType as the only type.

There will be a rule in #7175 to do implicit type casting from null type to expected type(s).

Add function unhex

c852d46

style

11945c7

update to use AutoCastInputTypes

cde73f5

davies reviewed Jun 30, 2015
View reviewed changes

change to use Hex in apache common package

bffd37f

tarekbecker reviewed Jul 1, 2015
View reviewed changes

zhichao-li added 3 commits June 30, 2015 23:48

use checkInputTypes

607d7a3

add todigit

fe5c14a

add udf_unhex to whitelist

a4ae6dc

zhichao-li changed the title ~~[SPARK-8227][SQL][WIP]Add function unhex~~ [SPARK-8227][SQL]Add function unhex Jul 1, 2015

davies reviewed Jul 1, 2015
View reviewed changes

remove exception checking

379356e

asfgit closed this in b285ac5 Jul 2, 2015

rxin reviewed Jul 2, 2015
View reviewed changes

[SPARK-8227][SQL]Add function unhex #7113

[SPARK-8227][SQL]Add function unhex #7113

Conversation

zhichao-li commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

SparkQA commented Jun 30, 2015

SparkQA commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

SparkQA commented Jun 30, 2015

SparkQA commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

SparkQA commented Jun 30, 2015

SparkQA commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

davies Jun 30, 2015

Choose a reason for hiding this comment

AmplabJenkins commented Jul 1, 2015

AmplabJenkins commented Jul 1, 2015

SparkQA commented Jul 1, 2015

SparkQA commented Jul 1, 2015

AmplabJenkins commented Jul 1, 2015

zhichao-li commented Jul 1, 2015

tarekbecker Jul 1, 2015

Choose a reason for hiding this comment

davies Jul 1, 2015

Choose a reason for hiding this comment

AmplabJenkins commented Jul 1, 2015

AmplabJenkins commented Jul 1, 2015

SparkQA commented Jul 1, 2015

SparkQA commented Jul 1, 2015

AmplabJenkins commented Jul 1, 2015

davies Jul 1, 2015

Choose a reason for hiding this comment

AmplabJenkins commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

SparkQA commented Jul 2, 2015

SparkQA commented Jul 2, 2015

AmplabJenkins commented Jul 2, 2015

davies commented Jul 2, 2015

rxin Jul 2, 2015

Choose a reason for hiding this comment