[SPARK-8271][SQL]string function: soundex #7115

hujy · 2015-06-30T07:49:47Z

Add soundex for SQL

It is written according to https://en.wikipedia.org/wiki/Soundex

Retain the first letter of the name and drop all other occurrences of a, e, i, o, u, y, h, w.
Replace consonants with digits as follows (after the first letter):
b, f, p, v → 1
c, g, j, k, q, s, x, z → 2
d, t → 3
l → 4
m, n → 5
r → 6
If two or more letters with the same number are adjacent in the original name (before step 1), only retain the first letter; also two letters with the same number separated by 'h' or 'w' are coded as a single number, whereas such letters separated by a vowel are coded twice. This rule also applies to the first letter.
Iterate the previous step until you have one letter and three numbers. If you have too few letters in your word that you can't assign three numbers, append with zeros until there are three numbers. If you have more than 3 letters, just retain the first 3 numbers.

Using this algorithm, both "Robert" and "Rupert" return the same string "R163" while "Rubin" yields "R150". "Ashcraft" and "Ashcroft" both yield "A261" and not "A226" (the chars 's' and 'c' in the name would receive a single number of 2 and not 22 since an 'h' lies in between them). "Tymczak" yields "T522" not "T520" (the chars 'z' and 'k' in the name are coded as 2 twice since a vowel lies in between them). "Pfister" yields "P236" not "P123" (the first two letters have the same number and are coded once as 'P').

chenghao-intel · 2015-06-30T07:53:17Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala

@@ -153,7 +153,8 @@ object FunctionRegistry {
    expression[Substring]("substr"),
    expression[Substring]("substring"),
    expression[Upper]("ucase"),
-    expression[Upper]("upper")
+    expression[Upper]("upper"),
+    expression[SoundEx]("SoundEx")


The registered function name should be in lower case, and sort the code in alphabet order.

chenghao-intel · 2015-07-06T02:04:08Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala

+    if (string == null) {
+      null
+    } else {
+      val str: String = string.toString


Nit: you don't need to declare the type explicitly. val str = string.toString will be OK.

Nit: Remove the : String.

chenghao-intel · 2015-07-06T02:04:43Z

@hujiayin can you rebase the code?

chenghao-intel · 2015-07-06T02:05:47Z

@rxin @davies @liancheng can you trigger the unit test?

SparkQA · 2015-07-06T03:11:05Z

Test build #999 has finished for PR 7115 at commit f88d320.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
- class Word2VecModel(JavaVectorTransformer, JavaSaveable, JavaLoader):
- case class CreateNamedStruct(children: Seq[Expression]) extends Expression
- case class Factorial(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class ShiftLeft(left: Expression, right: Expression) extends BinaryExpression
- case class ShiftRight(left: Expression, right: Expression) extends BinaryExpression
- case class Md5(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Sha1(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Crc32(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Not(child: Expression) extends UnaryExpression with Predicate with ExpectsInputTypes
- trait StringRegexExpression extends ExpectsInputTypes
- trait CaseConversionExpression extends ExpectsInputTypes
- trait StringComparison extends ExpectsInputTypes
- case class StringLength(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class SoundEx(child: Expression) extends UnaryExpression with ExpectsInputTypes
- protected[sql] abstract class AtomicType extends DataType
- abstract class NumericType extends AtomicType
- abstract class DataType extends AbstractDataType

rxin · 2015-07-06T06:02:31Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala

+      null
+    } else {
+      val str: String = string.toString
+      UTF8String.fromString(so.encode(str))


you need to return null if there is an exception

also can you copy the body of common codec's soundex function into UTF8String?

liancheng · 2015-07-06T08:26:34Z

ok to test

tarekbecker · 2015-07-06T09:16:36Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala

+    } else {
+      try {
+        val str: String = string.toString
+        UTF8String.fromString(so.encode(str))


@hujiayin I was talking about this line (in the other pr #7208). Even if str = "" Soundex gets created (because of so.encode), but as @chenghao-intel mentioned, it might be no big deal.

chenghao-intel · 2015-07-16T14:33:55Z

unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java

+        if(idxp >= 0 && idxp <= US_ENGLISH_MAPPING.length
+                && US_ENGLISH_MAPPING[idxp] - '0' == 0
+                && ch - data[j-1] == 0)
+        {


Nit: move the { to the end of last line.

chenghao-intel · 2015-07-16T14:42:08Z

Looks much better now, except some minor issues. And I didn't check the logic of soundex, it will be great if you can add more unit test for soundex.
Some examples can be found at https://issues.apache.org/jira/browse/HIVE-9738

SparkQA · 2015-07-17T04:01:10Z

Test build #37565 has finished for PR 7115 at commit 9a35ada.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class SoundEx(child: Expression) extends UnaryExpression with ExpectsInputTypes

SparkQA · 2015-07-17T06:48:49Z

Test build #37581 has finished for PR 7115 at commit a2d6313.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class SoundEx(child: Expression) extends UnaryExpression with ExpectsInputTypes

rxin · 2015-07-18T01:39:32Z

cc @davies for review too

davies · 2015-07-18T06:30:53Z

unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java

+      if (j > 3) break;
+      if (getByte(i) == getByte(i - 1)
+              || (getByte(i) - 32 == getByte(i - 1)
+              || (getByte(i) == getByte(i - 1) - 32))) {


This is not correct, for example, A = ! + 32

yes, the check was added

SparkQA · 2015-07-20T03:28:42Z

Test build #37785 has finished for PR 7115 at commit 8302b94.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
- case class SoundEx(child: Expression) extends UnaryExpression with ExpectsInputTypes

chenghao-intel · 2015-07-22T14:19:34Z

unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java

+        }
+        if (ch - '0' > 0) {
+          tmp = Character.toString(ch);
+          System.arraycopy(tmp.getBytes(), 0, data, j, 1);


This seems too heavy, can we update the data directly?

I've just tested.

scala> val c = 'A' c: Char = A scala> Character.toString(c) res0: String = A scala> val c = '0' c: Char = 0 scala> Character.toString(c) res1: String = 0

As we are sure the ch will be the character defined in US_ENGLISH_MAPPING, only a single byte will be copied, then we can simplify the code like:

data[j] = (byte)ch;

ch is just one byte and the data is only 4 bytes. what about the type conversion failed, from char to byte : (

davies · 2015-07-30T23:17:02Z

@hujiayin Thanks for working on this, do you mind me to take over it?

hujy · 2015-07-31T00:34:09Z

ok : ) the unit test is ok from my side.

This PR brings SQL function soundex(), see https://issues.apache.org/jira/browse/HIVE-9738 It's based on #7115 , thanks to HuJiayin Author: HuJiayin <jiayin.hu@intel.com> Author: Davies Liu <davies@databricks.com> Closes #7812 from davies/soundex and squashes the following commits: fa75941 [Davies Liu] Merge branch 'master' of github.com:apache/spark into soundex a4bd6d8 [Davies Liu] fix soundex 2538908 [HuJiayin] add codegen soundex d15d329 [HuJiayin] add back ut ded1a14 [HuJiayin] Merge branch 'master' of https://github.com/apache/spark e2dec2c [HuJiayin] support soundex rebase code

chenghao-intel reviewed Jun 30, 2015
View reviewed changes

tarekbecker mentioned this pull request Jul 3, 2015

[SPARK-8269][SQL]string function: initcap #7208

Closed

chenghao-intel reviewed Jul 6, 2015
View reviewed changes

hujy force-pushed the master branch from 43944d7 to 07fd066 Compare July 6, 2015 05:52

rxin reviewed Jul 6, 2015
View reviewed changes

tarekbecker reviewed Jul 6, 2015
View reviewed changes

chenghao-intel reviewed Jul 16, 2015
View reviewed changes

hujy force-pushed the master branch from 3bc573c to 9a35ada Compare July 17, 2015 02:04

davies reviewed Jul 18, 2015
View reviewed changes

hujy force-pushed the master branch from a2d6313 to 8302b94 Compare July 20, 2015 01:10

hujy closed this Jul 20, 2015

hujy force-pushed the master branch from 8302b94 to 163e3f1 Compare July 20, 2015 01:51

hujy reopened this Jul 20, 2015

hujy force-pushed the master branch from dbbc201 to 7c667ba Compare July 20, 2015 03:12

hujy force-pushed the master branch from 7c667ba to e2dec2c Compare July 22, 2015 01:17

support soundex rebase code

e2dec2c

chenghao-intel reviewed Jul 22, 2015
View reviewed changes

hujy added 3 commits July 23, 2015 12:49

Merge branch 'master' of https://github.com/apache/spark

ded1a14

add back ut

d15d329

add codegen soundex

2538908

davies mentioned this pull request Jul 31, 2015

[SPARK-8271][SQL]string function: soundex #7812

Closed

hujy closed this Aug 3, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-8271][SQL]string function: soundex #7115

[SPARK-8271][SQL]string function: soundex #7115

hujy commented Jun 30, 2015

chenghao-intel Jun 30, 2015

chenghao-intel Jul 6, 2015

chenghao-intel Jul 6, 2015

chenghao-intel commented Jul 6, 2015

chenghao-intel commented Jul 6, 2015

SparkQA commented Jul 6, 2015

rxin Jul 6, 2015

rxin Jul 6, 2015

liancheng commented Jul 6, 2015

tarekbecker Jul 6, 2015

chenghao-intel Jul 16, 2015

chenghao-intel commented Jul 16, 2015

SparkQA commented Jul 17, 2015

SparkQA commented Jul 17, 2015

rxin commented Jul 18, 2015

davies Jul 18, 2015

hujy Jul 20, 2015

SparkQA commented Jul 20, 2015

chenghao-intel Jul 22, 2015

chenghao-intel Jul 23, 2015

hujy Jul 23, 2015

davies commented Jul 30, 2015

hujy commented Jul 31, 2015

[SPARK-8271][SQL]string function: soundex #7115

[SPARK-8271][SQL]string function: soundex #7115

Conversation

hujy commented Jun 30, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chenghao-intel commented Jul 6, 2015

chenghao-intel commented Jul 6, 2015

SparkQA commented Jul 6, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liancheng commented Jul 6, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chenghao-intel commented Jul 16, 2015

SparkQA commented Jul 17, 2015

SparkQA commented Jul 17, 2015

rxin commented Jul 18, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jul 20, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davies commented Jul 30, 2015

hujy commented Jul 31, 2015