[SPARK-8264][SQL]add substring_index function #7533

zhichao-li · 2015-07-20T08:14:43Z

No description provided.

SparkQA · 2015-07-20T15:39:54Z

Test build #37824 has finished for PR 7533 at commit e52b379.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Substring_index(strExpr: Expression, delimExpr: Expression, countExpr: Expression)

SparkQA · 2015-07-21T10:26:43Z

Test build #37941 has finished for PR 7533 at commit 4fe5df5.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
- case class Substring_index(strExpr: Expression, delimExpr: Expression, countExpr: Expression)

SparkQA · 2015-07-22T03:46:30Z

Test build #38006 has finished for PR 7533 at commit 12e108f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Substring_index(strExpr: Expression, delimExpr: Expression, countExpr: Expression)

zhichao-li · 2015-07-22T04:04:34Z

cc @chenghao-intel @rxin

chenghao-intel · 2015-07-22T04:48:59Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala

+
+  override def dataType: DataType = StringType
+  override def inputTypes: Seq[DataType] = Seq(StringType, StringType, IntegerType)
+  override def nullable: Boolean = strExpr.nullable || delimExpr.nullable || countExpr.nullable


Need to override the nullable method, as the default value is false.

I guess it's abstract in Expression which Substring_index inherited from?

Sorry, my bad, actually I mean the function foldable, not the nullable.

zhichao-li · 2015-07-22T08:01:33Z

@chenghao-intel just refactor the code to remedy the problem of invoking numChars to much, but I still think we should add numChars as a field to UTF8String since it's quite an useful function.

zhichao-li · 2015-07-22T08:04:01Z

@rxin @chenghao-intel should I turn these functions(ordinalIndexOf, lastOrdinalIndexOf and subStringIndex) to be public in UTF8String? I guess they would be useful but on one use them except for substring_index UDF.

SparkQA · 2015-07-22T08:06:53Z

Test build #38058 has finished for PR 7533 at commit ac863e9.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Substring_index(strExpr: Expression, delimExpr: Expression, countExpr: Expression)

zhichao-li · 2015-07-22T08:49:36Z

retest this please.

SparkQA · 2015-07-22T08:51:29Z

Test build #55 has finished for PR 7533 at commit ac863e9.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Substring_index(strExpr: Expression, delimExpr: Expression, countExpr: Expression)

SparkQA · 2015-07-22T08:56:09Z

Test build #38063 has finished for PR 7533 at commit ac863e9.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Substring_index(strExpr: Expression, delimExpr: Expression, countExpr: Expression)

chenghao-intel · 2015-07-22T13:14:53Z

sql/core/src/main/scala/org/apache/spark/sql/functions.scala

+   * @group string_funcs
+   * @since 1.5.0
+   */
+  def substring_index(str: String, delim: String, count: Int): Column =


Let's remove this version of API. @rxin actually made some clean up, and removed the string (the column name) version API.

SparkQA · 2015-07-23T07:34:40Z

Test build #38186 has finished for PR 7533 at commit b19b013.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Substring_index(strExpr: Expression, delimExpr: Expression, countExpr: Expression)

chenghao-intel · 2015-07-23T15:00:52Z

unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java

+   * @return the n-th last index of the search String,
+   *  <code>-1</code> if no match or <code>null</code> string input
+   */
+  public static int lastOrdinalIndexOf(


We'd better not make it the static function, like the indexOf, it mean we are locating the substring from CURRENT string.

chenghao-intel · 2015-07-23T15:10:10Z

unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java

+      }
+      bytePos--;
+    }
+    throw new RuntimeException("Invalid UTF8 string");


Give more verbose info, like this.toString()?

SparkQA · 2015-07-24T02:43:01Z

Test build #38294 has finished for PR 7533 at commit 67c253a.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Substring_index(strExpr: Expression, delimExpr: Expression, countExpr: Expression)

SparkQA · 2015-07-24T04:45:04Z

Test build #38298 has finished for PR 7533 at commit 9546991.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Substring_index(strExpr: Expression, delimExpr: Expression, countExpr: Expression)

chenghao-intel · 2015-07-24T05:05:48Z

unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java

+   * right) is returned. substring_index performs a case-sensitive match when searching for delim.
+   */
+  public UTF8String subStringIndex(UTF8String delim, int count) {
+    if (delim == null) {


We dont' need to check the null value here, as it's done in the expression side.

SparkQA · 2015-07-24T08:21:41Z

Test build #38317 has finished for PR 7533 at commit 515519b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Substring_index(strExpr: Expression, delimExpr: Expression, countExpr: Expression)

zhichao-li · 2015-07-27T00:29:08Z

@rxin it looks a few better now. could you take a look at this?

chenghao-intel · 2015-07-31T02:32:07Z

unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java

+      }
+      bytePos--;
+    }
+    throw new RuntimeException("Invalid UTF8 string: " + toString());


Nit: throws more concrete exception like IllegalArgumentException or IllegalCharacterException etc.

chenghao-intel · 2015-07-31T02:32:52Z

LGTM in general, @davies @rxin

davies · 2015-07-31T20:45:45Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala

+ * right) is returned. substring_index performs a case-sensitive match when searching for delim.
+ */
+case class Substring_index(strExpr: Expression, delimExpr: Expression, countExpr: Expression)
+ extends Expression with ImplicitCastInputTypes {


Could you use TernaryExpression ?

davies · 2015-07-31T21:28:05Z

@zhichao-li Do you mind me to take over this one?

zhichao-li added 3 commits July 22, 2015 08:43

add substring_index function

52d7b03

add lastIndexOf

d92951b

refine unittest

12e108f

zhichao-li force-pushed the substrindex branch from 4fe5df5 to 12e108f Compare July 22, 2015 01:26

chenghao-intel reviewed Jul 22, 2015
View reviewed changes

reduce the calling of numChars

ac863e9

chenghao-intel reviewed Jul 22, 2015
View reviewed changes

add codegen and clean code

b19b013

chenghao-intel reviewed Jul 23, 2015
View reviewed changes

hide some apis and clean code

67c253a

scala style

9546991

chenghao-intel reviewed Jul 24, 2015
View reviewed changes

add foldable and remove null checking

515519b

chenghao-intel reviewed Jul 31, 2015
View reviewed changes

davies reviewed Jul 31, 2015
View reviewed changes

davies mentioned this pull request Jul 31, 2015

[SPARK-8264][SQL]add substring_index function #7843

Closed

asfgit closed this in 6996bd2 Aug 1, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-8264][SQL]add substring_index function #7533

[SPARK-8264][SQL]add substring_index function #7533

zhichao-li commented Jul 20, 2015

SparkQA commented Jul 20, 2015

SparkQA commented Jul 21, 2015

SparkQA commented Jul 22, 2015

zhichao-li commented Jul 22, 2015

chenghao-intel Jul 22, 2015

zhichao-li Jul 22, 2015

chenghao-intel Jul 23, 2015

zhichao-li commented Jul 22, 2015

zhichao-li commented Jul 22, 2015

SparkQA commented Jul 22, 2015

zhichao-li commented Jul 22, 2015

SparkQA commented Jul 22, 2015

SparkQA commented Jul 22, 2015

chenghao-intel Jul 22, 2015

SparkQA commented Jul 23, 2015

chenghao-intel Jul 23, 2015

chenghao-intel Jul 23, 2015

SparkQA commented Jul 24, 2015

SparkQA commented Jul 24, 2015

chenghao-intel Jul 24, 2015

SparkQA commented Jul 24, 2015

zhichao-li commented Jul 27, 2015

chenghao-intel Jul 31, 2015

chenghao-intel commented Jul 31, 2015

davies Jul 31, 2015

davies commented Jul 31, 2015

[SPARK-8264][SQL]add substring_index function #7533

[SPARK-8264][SQL]add substring_index function #7533

Conversation

zhichao-li commented Jul 20, 2015

SparkQA commented Jul 20, 2015

SparkQA commented Jul 21, 2015

SparkQA commented Jul 22, 2015

zhichao-li commented Jul 22, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhichao-li commented Jul 22, 2015

zhichao-li commented Jul 22, 2015

SparkQA commented Jul 22, 2015

zhichao-li commented Jul 22, 2015

SparkQA commented Jul 22, 2015

SparkQA commented Jul 22, 2015

Choose a reason for hiding this comment

SparkQA commented Jul 23, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jul 24, 2015

SparkQA commented Jul 24, 2015

Choose a reason for hiding this comment

SparkQA commented Jul 24, 2015

zhichao-li commented Jul 27, 2015

Choose a reason for hiding this comment

chenghao-intel commented Jul 31, 2015

Choose a reason for hiding this comment

davies commented Jul 31, 2015