Skip to content

[FLINK-36941][docs] Update DATE_FORMAT Doc and Python Tests #25828

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yiyutian1 wants to merge 2 commits into from
Closed

[FLINK-36941][docs] Update DATE_FORMAT Doc and Python Tests #25828

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
238c093
[FLINK-36941][docs] Update DATE_FORMAT Doc and Python Tests
yiyutian1 Dec 20, 2024
7f3d4ed
[FLINK-36941] Address Comments
yiyutian1 Dec 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion docs/data/sql_functions.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -642,7 +642,10 @@ temporal:
description: Returns TRUE if two time intervals defined by (timepoint1, temporal1) and (timepoint2, temporal2) overlap. The temporal values could be either a time point or a time interval. E.g., (TIME '2:55:00', INTERVAL '1' HOUR) OVERLAPS (TIME '3:30:00', INTERVAL '2' HOUR) returns TRUE; (TIME '9:00:00', TIME '10:00:00') OVERLAPS (TIME '10:15:00', INTERVAL '3' HOUR) returns FALSE.
- sql: DATE_FORMAT(timestamp, string)
table: dateFormat(TIMESTAMP, STRING)
description: Converts timestamp to a value of string in the format specified by the date format string. The format string is compatible with Java's SimpleDateFormat.
description: Converts timestamp to a string in the format specified by the date format string. The format string is compatible with Java's DateTimeFormatter.
- sql: DATE_FORMAT(string, string)
table: dateFormat(STRING, STRING)
description: Re-format a timestamp string to another string, using a custom format. The format string is compatible with Java's SimpleDateFormat.
Copy link

@nictownsend nictownsend Dec 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we do the same for UNIX_TIMESTAMP(string1[, string2]) (https://github.com/apache/flink/pull/25828/files#diff-539fb22ee6aeee4cf07230bb4155500c6680c4cc889260e2c58bfa9d63fb7de5R662) please?

flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/utils/DateTimeUtils.java

Line 1468 in 7fa4f78

public static long unixTimestamp(String dateStr, String format, TimeZone tz) {
calls into

flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/utils/DateTimeUtils.java

Line 960 in 7fa4f78

private static long internalParseTimestampMillis(String dateStr, String format, TimeZone tz) {
too - so the format is also using SimpleDateFormat

Copy link

@nictownsend nictownsend Dec 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming I've read

flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/BuiltInFunctionDefinitions.java

Line 2304 in 7fa4f78

public static final BuiltInFunctionDefinition UNIX_TIMESTAMP =
to mean it invokes unixTimestamp(dateStr, format, null) as there is no tz third parameter in the SQL function.

Copy link
Contributor

@davidradl davidradl Dec 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nictownsend sounds like a reasonable idea, @snuyanzin @yiyutian1 if you agree we could raise this separately.

Copy link

@nictownsend nictownsend Dec 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies, I hadn't appreciated the scope of the underlying Flink issue, happy to raise a new one if it's not appropriate to fix here

Copy link
Contributor

@snuyanzin snuyanzin Dec 20, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a slightly different thought
Since it is Flink 2.0 and anyway it brings some breaking changes. May be we can completely switch to DateTimeFormatter in 2.0 while for 1.19 and 1.20 we just fix documentation.
However this option requires a discussion in ML
i can start one if no objections

Copy link
Contributor Author

@yiyutian1 yiyutian1 Dec 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@snuyanzin I like the idea.
Before that happens, should we fix the current doc?
If to only commit for 1.19 and 1.20, should I create PR against release-1.19 and release-1.20?

- sql: TIMESTAMPADD(timeintervalunit, interval, timepoint)
- sql: TIMESTAMPDIFF(timepointunit, timepoint1, timepoint2)
table: timestampDiff(TIMEPOINTUNIT, TIMEPOINT1, TIMEPOINT2)
Expand Down
6 changes: 4 additions & 2 deletions docs/data/sql_functions_zh.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -757,8 +757,10 @@ temporal:
`(TIME '9:00:00', TIME '10:00:00') OVERLAPS (TIME '10:15:00', INTERVAL '3' HOUR)` 返回 FALSE。
- sql: DATE_FORMAT(timestamp, string)
table: dateFormat(TIMESTAMP, STRING)
description: |
将时间戳 timestamp 转换为日期格式字符串 string 指定格式的字符串值。格式字符串与 Java 的 SimpleDateFormat 兼容。
description: Converts timestamp to a string in the format specified by the date format string. The format string is compatible with Java's DateTimeFormatter.
- sql: DATE_FORMAT(string, string)
table: dateFormat(STRING, STRING)
description: Re-format a timestamp string to another string, using a custom format. The format string is compatible with Java's SimpleDateFormat.
- sql: TIMESTAMPADD(timeintervalunit, interval, timepoint)
- sql: TIMESTAMPDIFF(timepointunit, timepoint1, timepoint2)
table: timestampDiff(TIMEPOINTUNIT, TIMEPOINT1, TIMEPOINT2)
Expand Down
19 changes: 13 additions & 6 deletions flink-python/pyflink/table/expressions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -350,14 +350,21 @@ def temporal_overlaps(left_time_point,
def date_format(timestamp, format) -> Expression:
"""
Formats a timestamp as a string using a specified format.
The format must be compatible with MySQL's date formatting syntax as used by the
date_parse function.
Copy link
Contributor

@davidradl davidradl Dec 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if this has been removed - do we not need to deprecate first before replacing?

Copy link
Contributor Author

@yiyutian1 yiyutian1 Dec 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davidradl , the comment I deleted is not accurate.
MySQL's date formatting syntax follows its own date formatting syntax, not Java's SimpleDateFormat or DateTimeFormatter. MySQL is written in C++, not Java, and doesn't have much significance as a reference here.


For example `date_format(col("time"), "%Y, %d %M")` results in strings formatted as
"2017, 05 May".
Supported functions:
1. date_format(TIMESTAMP, STRING) -> STRING
Copy link
Contributor

@davidradl davidradl Dec 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should mention the difference between the supported format strings.

Converts timestamp to a string, using a format string.
The format string is compatible with Java's DateTimeFormatter.
2. date_format(STRING, STRING) -> STRING
Converts timestamp string, using a format string.
The format string is compatible with Java's SimpleDateFormat.

Example:
::

>>> table.select(date_format(to_timestamp('2020-04-15'), "yyyy/MM/dd"))
>>> table.select(date_format("2020-04-15", "MM/dd/yyyy"))

:param timestamp: The timestamp to format as string.
:param format: The format of the string.
:return: The formatted timestamp as string.
"""
return _binary_op("dateFormat", timestamp, format)
Expand Down
9 changes: 9 additions & 0 deletions flink-python/pyflink/table/tests/test_expression.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -299,6 +299,15 @@ def test_expressions(self):
lit(2).hours)))
self.assertEqual("dateFormat(time, '%Y, %d %M')",
str(date_format(col("time"), "%Y, %d %M")))
self.assertEqual("dateFormat(toTimestamp('1970-01-01 08:01:40'), 'yyyy-MM-dd HH:mm:ss')",
str(date_format(to_timestamp('1970-01-01 08:01:40'),
"yyyy-MM-dd HH:mm:ss")))
self.assertEqual("DATE_FORMAT(TO_TIMESTAMP('1970-01-01 08:01:40.123456+02:00'), "
"'yyyy-MM-dd HH:mm:ss.SSSSSS z')",
str(date_format(to_timestamp('1970-01-01 08:01:40.123456+02:00'),
'yyyy-MM-dd HH:mm:ss.SSSSSS z')))
self.assertEqual("dateFormat('1970-01-01 08:01:40', 'yyyy-MM-dd HH:mm:ss')",
Copy link
Contributor

@davidradl davidradl Dec 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you add a test with the format that the DateTimeFormatter could cope with - with precision and timezone

Copy link
Contributor Author

@yiyutian1 yiyutian1 Dec 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @davidradl , sure I can add that.
Also want to point out that these test_expressions don't test any implementations, they just test the "signature" of defined functions.

str(date_format("1970-01-01 08:01:40", "yyyy-MM-dd HH:mm:ss")))
self.assertEqual("timestampDiff(DAY, cast('2016-06-15', DATE), cast('2016-06-18', DATE))",
str(timestamp_diff(
TimePointUnit.DAY,
Expand Down