Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
42bbced
[FLINK-34111] Add support for json_quote, json_unquote, address feedb…
anupamaggarwal Jun 20, 2024
24a73dc
[FLINK-34111] Removed redundant checks, update documentation to remov…
anupamaggarwal Jun 20, 2024
9a60e44
[FLINK-34111] Update scala test
anupamaggarwal Jun 20, 2024
fd9e5a7
Merge branch 'apache-master' into 34111-jsonquote_unquote
anupamaggarwal Jun 20, 2024
0056727
[FLINK-34111] Refactor, update comments
anupamaggarwal Jun 20, 2024
3903d24
[FLINK-34111] Fix spotless warning
anupamaggarwal Jun 20, 2024
48b570f
[FLINK-34111] Add missing function definitions, fix test
anupamaggarwal Jun 20, 2024
4267018
[FLINK-34111] Fix java.util.IllegalFormatConversionException in json_…
anupamaggarwal Jul 5, 2024
141c63e
[FLINK-34111] CR: Update documentation, add separate function for neg…
anupamaggarwal Jul 10, 2024
990734b
[FLINK-34111] CR: Avoid throwing exception on invalid input
anupamaggarwal Jul 10, 2024
ad3ec47
[FLINK-34111] CR: Add, refactor negative tests
anupamaggarwal Jul 10, 2024
04db96e
[FLINK-34111] CR: Fix unquote logic, refactor, add tests
anupamaggarwal Jul 11, 2024
ba9ebfa
[FLINK-34111] CR: Rearranging order of methods public to private
anupamaggarwal Jul 11, 2024
80b693d
[FLINK-34111] CR: Catch only expected exceptions
anupamaggarwal Jul 11, 2024
7546d9c
[FLINK-34111] CR: Exception handling, add UT
anupamaggarwal Jul 17, 2024
cbcb044
[FLINK-34111] CR: Add documentation for json_(un)quote to chinese doc
anupamaggarwal Jul 17, 2024
37fa2e7
[FLINK-34111] CR: Update documentation
anupamaggarwal Aug 6, 2024
570d28f
Merge remote-tracking branch 'oss1/master' into 34111-jsonquote_unquote
anupamaggarwal Aug 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/data/sql_functions.yml
Original file line number Diff line number Diff line change
Expand Up @@ -394,6 +394,12 @@ string:
- sql: SUBSTR(string, integer1[, integer2])
table: STRING.substr(INTEGER1[, INTEGER2])
description: Returns a substring of string starting from position integer1 with length integer2 (to the end by default).
- sql: JSON_QUOTE(string)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing I forgot to mention: also need to update chinese doc.
It's ok to put there english version of description however it's better to keep them synced

Copy link
Contributor

@fhueske fhueske Jul 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point!
We can also create a Jira with a special tag to document gaps in the Chinese documentation: https://flink.apache.org/how-to-contribute/contribute-documentation/#chinese-documentation-translation

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for creation of jira
I would still vote for having filled chinese doc with english description which then could be translated within the jira issue you've created

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure @snuyanzin, fixed in cbcb044

table: STRING.JsonQuote()
description: Quotes a string as a JSON value by wrapping it with double quote characters, escaping interior quote and special characters ('"', '\', '/', 'b', 'f', 'n', 'r', 't'), and returning the result as a string. If the argument is NULL, the function returns NULL.
- sql: JSON_UNQUOTE(string)
table: STRING.JsonUnquote()
description: Unquotes JSON value, unescapes escaped special characters ('"', '\', '/', 'b', 'f', 'n', 'r', 't', 'u' hex hex hex hex), and returns the result as a string. If the argument is NULL, returns NULL. If the value does not start and end with double quotes or if it starts and ends with double quotes but is not a valid JSON string literal, the value is passed through unmodified.

temporal:
- sql: DATE string
Expand Down
6 changes: 6 additions & 0 deletions docs/data/sql_functions_zh.yml
Original file line number Diff line number Diff line change
Expand Up @@ -494,6 +494,12 @@ string:
- sql: SUBSTR(string, integer1[, integer2])
table: STRING.substr(INTEGER1[, INTEGER2])
description: 返回字符串的子字符串,从位置 integer1 开始,长度为 integer2(默认到末尾)。
- sql: JSON_QUOTE(string)
table: STRING.JsonQuote()
description: Quotes a string as a JSON value by wrapping it with double quote characters, escaping interior quote and special characters ('"', '\', '/', 'b', 'f', 'n', 'r', 't'), and returning the result as a string. If the argument is NULL, the function returns NULL.
- sql: JSON_UNQUOTE(string)
table: STRING.JsonUnquote()
description: Unquotes JSON value, unescapes escaped special characters ('"', '\', '/', 'b', 'f', 'n', 'r', 't', 'u' hex hex hex hex), and returns the result as a string. If the argument is NULL, returns NULL. If the value does not start and end with double quotes or if it starts and ends with double quotes but is not a valid JSON string literal, the value is passed through unmodified.

temporal:
- sql: DATE string
Expand Down
2 changes: 2 additions & 0 deletions flink-python/docs/reference/pyflink.table/expressions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -308,3 +308,5 @@ JSON functions
Expression.json_exists
Expression.json_value
Expression.json_query
Expression.json_quote
Expression.json_unquote
25 changes: 23 additions & 2 deletions flink-python/pyflink/table/expression.py
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,8 @@
:func:`~Expression.overlay`, :func:`~Expression.regexp_replace`,
:func:`~Expression.regexp_extract`, :func:`~Expression.substring`,
:py:attr:`~Expression.from_base64`, :py:attr:`~Expression.to_base64`,
:py:attr:`~Expression.ltrim`, :py:attr:`~Expression.rtrim`, :func:`~Expression.repeat`
:py:attr:`~Expression.ltrim`, :py:attr:`~Expression.rtrim`, :func:`~Expression.repeat`,
:func:`~Expression.json_quote`, :func:`~Expression.json_unquote`
"""

_temporal_doc_seealso = """
Expand Down Expand Up @@ -186,7 +187,8 @@ def _make_string_doc():
Expression.init_cap, Expression.like, Expression.similar, Expression.position,
Expression.lpad, Expression.rpad, Expression.overlay, Expression.regexp_replace,
Expression.regexp_extract, Expression.from_base64, Expression.to_base64,
Expression.ltrim, Expression.rtrim, Expression.repeat
Expression.ltrim, Expression.rtrim, Expression.repeat,
Expression.json_quote, Expression.json_unquote
]

for func in string_funcs:
Expand Down Expand Up @@ -1978,6 +1980,25 @@ def json_query(self, path: str, wrapping_behavior=JsonQueryWrapper.WITHOUT_ARRAY
on_empty._to_j_json_query_on_error_or_empty(),
on_error._to_j_json_query_on_error_or_empty())

def json_quote(self) -> 'Expression':
"""
Quotes a string as a JSON value by wrapping it with double quote characters,
escaping interior quote and special characters
('"', '\', '/', 'b', 'f', 'n', 'r', 't'), and returning
the result as a string. If the argument is NULL, the function returns NULL.
"""
return _unary_op("jsonQuote")(self)

def json_unquote(self) -> 'Expression':
"""
Unquotes JSON value, unescapes escaped special characters
('"', '\', '/', 'b', 'f', 'n', 'r', 't', 'u' hex hex hex hex) and
returns the result as a string.
If the argument is NULL, returns NULL. If the value starts and ends with
double quotes but is not a valid JSON string literal, an error occurs.
"""
return _unary_op("jsonUnquote")(self)


# add the docs
_make_math_log_doc()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,8 @@
import static org.apache.flink.table.functions.BuiltInFunctionDefinitions.IS_TRUE;
import static org.apache.flink.table.functions.BuiltInFunctionDefinitions.JSON_EXISTS;
import static org.apache.flink.table.functions.BuiltInFunctionDefinitions.JSON_QUERY;
import static org.apache.flink.table.functions.BuiltInFunctionDefinitions.JSON_QUOTE;
import static org.apache.flink.table.functions.BuiltInFunctionDefinitions.JSON_UNQUOTE;
import static org.apache.flink.table.functions.BuiltInFunctionDefinitions.JSON_VALUE;
import static org.apache.flink.table.functions.BuiltInFunctionDefinitions.LAST_VALUE;
import static org.apache.flink.table.functions.BuiltInFunctionDefinitions.LEFT;
Expand Down Expand Up @@ -1149,6 +1151,19 @@ public OutType regexpExtract(InType regex) {
unresolvedCall(REGEXP_EXTRACT, toExpr(), objectToExpression(regex)));
}

/**
* Returns a string by quotes a string as a JSON value and wrapping it with double quote
* characters.
*/
public OutType jsonQuote() {
return toApiSpecificExpression(unresolvedCall(JSON_QUOTE, objectToExpression(toExpr())));
}

/** Returns a string by unquoting JSON value. */
public OutType jsonUnquote() {
return toApiSpecificExpression(unresolvedCall(JSON_UNQUOTE, objectToExpression(toExpr())));
}

/** Returns the base string decoded with base64. */
public OutType fromBase64() {
return toApiSpecificExpression(unresolvedCall(FROM_BASE64, toExpr()));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1125,6 +1125,24 @@ ANY, and(logical(LogicalTypeRoot.BOOLEAN), LITERAL)
.outputTypeStrategy(explicit(DataTypes.STRING().nullable()))
.build();

public static final BuiltInFunctionDefinition JSON_QUOTE =
BuiltInFunctionDefinition.newBuilder()
.name("JSON_QUOTE")
.kind(SCALAR)
.inputTypeStrategy(sequence(logical(LogicalTypeFamily.CHARACTER_STRING)))
.outputTypeStrategy(nullableIfArgs(explicit(DataTypes.STRING())))
.runtimeClass(
"org.apache.flink.table.runtime.functions.scalar.JsonQuoteFunction")
.build();
public static final BuiltInFunctionDefinition JSON_UNQUOTE =
BuiltInFunctionDefinition.newBuilder()
.name("JSON_UNQUOTE")
.kind(SCALAR)
.inputTypeStrategy(sequence(logical(LogicalTypeFamily.CHARACTER_STRING)))
.outputTypeStrategy(nullableIfArgs(explicit(DataTypes.STRING())))
.runtimeClass(
"org.apache.flink.table.runtime.functions.scalar.JsonUnquoteFunction")
.build();
public static final BuiltInFunctionDefinition FROM_BASE64 =
BuiltInFunctionDefinition.newBuilder()
.name("fromBase64")
Expand Down
Loading