[FLINK-9853] [table] Add HEX support #6337

xueyumusic · 2018-07-15T13:33:16Z

What is the purpose of the change

This PR propose to add HEX in table api and sql, the syntax like mysql, which could take int or string arguments. For a integer argument N, it returns a hexadecimal string representation of the value of N. For a string argument str, it returns a hexadecimal string representation of str where each byte of each character in str is converted to two hexadecimal digits.

Syntax:

HEX(100) = 64

HEX('This is a test String.') = '546869732069732061207465737420537472696e672e'

Brief change log

The expressionDsl, scalarSqlFunctions and mathExpressions to add hex
The FunctionGenerator to support hex generator

Verifying this change

This change added tests and can be verified as follows:

(example:)

Added ScalaFunctionTests tests for table api and sql expressions

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (yes)
If yes, how is the feature documented? (docs)

yanghua

some little suggestion

yanghua · 2018-07-16T04:38:31Z

flink-libraries/flink-table/src/main/scala/org/apache/flink/table/api/scala/expressionDsl.scala

@@ -400,6 +400,12 @@ trait ImplicitExpressionOperations {
    * numeric is null. E.g. "4" leads to "100", "12" leads to "1100".
    */
  def bin() = Bin(expr)
+  /**


please insert a new blank line

yanghua · 2018-07-16T04:41:40Z

...es/flink-table/src/main/scala/org/apache/flink/table/runtime/functions/ScalarFunctions.scala

@@ -182,4 +184,6 @@ object ScalarFunctions {

    new String(data)
  }
+
+  def hex(x: String): String = Hex.encodeHexString(x.getBytes)


add the doc for the API looks better to me

thanks for review, @yanghua , fixed

yanghua · 2018-07-16T15:05:46Z

+1, from my side

hequn8128 · 2018-07-21T03:28:00Z

Thanks for your PR. +1 to merge.

walterddr

Thanks for the contribution @xueyumusic.

I am a bit confuse regarding the documentation. Seems like both "20" and 20L are valid input to the HEX function. It's worth to point out the difference IMO, especially they return different results: e.g.
10.hex() gives "a" and "10".hex() gives "3130"

+1 to merge once documentation is fixed.

walterddr · 2018-08-03T20:38:31Z

docs/dev/table/sql.md

+      {% endhighlight %}
+      </td>
+      <td>
+        <p>Returns a string representation of an integer numeric value or a string in hex format. Returns null if numeric is null. E.g. "20" leads to "14", "100" leads to "64", "hello,world" leads to "68656c6c6f2c776f726c64".</p>


maybe consider using backtick for the number literals? e.g. "20" --> `20`.

Otherwise its confusing whether we should pass in stringify integer or just integer values.

Thanks @walterddr , I clarify the doc adding explicit explain for numeric or for string

great change and explanations. actually my point was to have "20" --> `20` with specifically the backtick. since backtick quoted strings gets format differently in Flink doc page and it will be immediately clear that this is a numeral literal.

walterddr · 2018-08-03T20:41:12Z

docs/dev/table/tableApi.md

+{% endhighlight %}
+     </td>
+    <td>
+      <p>Returns a string representation of an integer numeric value or a string in hex format. Returns null if numeric is null. E.g. "20" leads to "14", "100" leads to "64", "hello,world" leads to "68656c6c6f2c776f726c64".</p>


walterddr · 2018-08-03T20:44:39Z

flink-libraries/flink-table/src/main/scala/org/apache/flink/table/api/scala/expressionDsl.scala

@@ -406,6 +406,13 @@ trait ImplicitExpressionOperations {
    */
  def bin() = Bin(expr)

+  /**
+    * Returns a string representation of an integer numeric value in hex format. Returns null if
+    * numeric is null. E.g. "20" leads to "14", "100" leads to "64", "hello,world" leads to


just remove the " if it is numeric

twalthr

Thank you @xueyumusic. I added one comment about the result string.

twalthr · 2018-08-13T11:58:57Z

...ibraries/flink-table/src/main/scala/org/apache/flink/table/expressions/mathExpressions.scala

+  override private[flink] def resultType: TypeInformation[_] = BasicTypeInfo.STRING_TYPE_INFO
+
+  override private[flink] def validateInput(): ValidationResult = child.resultType match {
+    case _: IntegerTypeInfo[_] =>


Rely on TypeCheckUtils.isIntegerFamily and TypeCheckUtils#isString instead.

twalthr · 2018-08-13T12:00:07Z

...ries/flink-table/src/test/scala/org/apache/flink/table/expressions/ScalarFunctionsTest.scala

+      'f2.hex(),
+      "f2.hex()",
+      "HEX(f2)",
+      "2a")


MySQL returns upper-case letters. Should we do this as well?

Yes, we should do it as well. Thanks @twalthr , I updated the code

twalthr · 2018-08-14T07:48:17Z

Thank you for the quick update @xueyumusic. I will merge this.

This closes apache#6337.

twalthr · 2018-08-14T09:45:59Z

@xueyumusic for future PRs use git rebase master YOURBRANCH instead of merge commits. Thank you.

This closes apache#6337. (cherry picked from commit ab1d1df)

xueyumusic added 2 commits July 15, 2018 20:01

hex support

8d3bef9

add doc

23dd1b1

yanghua reviewed Jul 16, 2018

View reviewed changes

add java doc

60b6ad3

Merge branch 'master' into hex

8647aa1

twalthr changed the title ~~[FLINK-9853][Tabel API & SQL] add HEX support~~ [FLINK-9853] [table] add HEX support Jul 31, 2018

twalthr changed the title ~~[FLINK-9853] [table] add HEX support~~ [FLINK-9853] [table] Add HEX support Jul 31, 2018

walterddr reviewed Aug 3, 2018

View reviewed changes

xueyumusic added 3 commits August 5, 2018 21:23

clarify hex docs

891363e

Merge branch 'hex' of https://github.com/xueyumusic/flink into hex

5654c41

Merge branch 'master' into hex

57ae3fa

twalthr reviewed Aug 13, 2018

View reviewed changes

xueyumusic added 5 commits August 13, 2018 23:44

feedback address

a7e544f

fix conflicts

54b3b3d

fix conflicts

6f160d2

remove unnecessary import

0ca766b

revert a blank line

cf7467e

twalthr pushed a commit to twalthr/flink that referenced this pull request Aug 14, 2018

[FLINK-9853] [table] Add HEX support for Table API & SQL

c520a17

This closes apache#6337.

twalthr pushed a commit to twalthr/flink that referenced this pull request Aug 14, 2018

[FLINK-9853] [table] Add HEX support for Table API & SQL

5bdbdf9

This closes apache#6337.

asfgit closed this in ab1d1df Aug 14, 2018

Clarkkkkk pushed a commit to Clarkkkkk/flink that referenced this pull request Mar 7, 2019

[FLINK-9853] [table] Add HEX support for Table API & SQL

30706b2

This closes apache#6337. (cherry picked from commit ab1d1df)

rmetzger added the component=API/TableSQL label Mar 18, 2019

flinkbot added component=TableSQL/API and removed component=API/TableSQL labels Mar 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-9853] [table] Add HEX support #6337

[FLINK-9853] [table] Add HEX support #6337

xueyumusic commented Jul 15, 2018

yanghua left a comment

yanghua Jul 16, 2018

yanghua Jul 16, 2018

xueyumusic Jul 16, 2018

yanghua commented Jul 16, 2018

hequn8128 commented Jul 21, 2018

walterddr left a comment

walterddr Aug 3, 2018

xueyumusic Aug 5, 2018

walterddr Aug 8, 2018

walterddr Aug 3, 2018

walterddr Aug 3, 2018

twalthr left a comment

twalthr Aug 13, 2018

twalthr Aug 13, 2018

xueyumusic Aug 13, 2018

twalthr commented Aug 14, 2018

twalthr commented Aug 14, 2018

[FLINK-9853] [table] Add HEX support #6337

[FLINK-9853] [table] Add HEX support #6337

Conversation

xueyumusic commented Jul 15, 2018

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

yanghua left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yanghua commented Jul 16, 2018

hequn8128 commented Jul 21, 2018

walterddr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

twalthr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

twalthr commented Aug 14, 2018

twalthr commented Aug 14, 2018