[FLINK-16650][python] Support LocalZonedTimestampType for Python UDF in blink planner #11439

dianfu · 2020-03-18T11:52:12Z

What is the purpose of the change

This pull request adds support of LocalZonedTimestampType for Python UDF in blink planner.

Brief change log

Update the Python operators to pass the timezone information to the Python process
Add support of LocalZonedTimestampType in blink planner

Verifying this change

This change added tests and can be verified as follows:

Added tests test_data_types_only_supported_in_blink_planner in test_udf.py

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

…in blink planner

flinkbot · 2020-03-18T11:54:46Z

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit b8aa27e (Wed Mar 18 11:54:45 UTC 2020)

Warnings:

No documentation files were touched! Remember to keep the Flink docs up to date!

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

❓ 1. The [description] looks good.
❓ 2. There is [consensus] that the contribution should go into to Flink.
❓ 3. Needs [attention] from.
❓ 4. The change fits into the overall [architecture].
❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

Details

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
@flinkbot approve all to approve all aspects
@flinkbot approve-until architecture to approve everything until architecture
@flinkbot attention @username1 [@username2 ..] to require somebody's attention
@flinkbot disapprove architecture to remove an approval you gave earlier

flinkbot · 2020-03-18T12:08:20Z

CI report:

49d8110 UNKNOWN
e09703f Travis: SUCCESS Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run travis re-run the last Travis build
@flinkbot run azure re-run the last Azure build

hequn8128

@dianfu Thanks a lot for the PR. Some minor comments.

hequn8128 · 2020-03-19T03:26:39Z

...ain/scala/org/apache/flink/table/planner/plan/nodes/physical/batch/BatchExecPythonCalc.scala

      calcProgram,
      "BatchExecPythonCalc",
-      planner.getTableConfig.getConfiguration)
+      planner.getTableConfig)


Change to getConfig(planner.getTableConfig)? so that we don't need to change the signature of createPythonOneInputTransformation()

hequn8128 · 2020-03-19T03:33:51Z

flink-python/pyflink/table/types.py

+            if dt.tzinfo is not None:
+                offset = dt.utcoffset()
+                offset = offset if offset else datetime.timedelta()
+                offset_microseconds =\
+                    (offset.days * 86400 + offset.seconds) * 10 ** 6 + offset.microseconds
+            else:
+                offset_microseconds = self.EPOCH_ORDINAL
            seconds = (calendar.timegm(dt.utctimetuple()) if dt.tzinfo
                       else time.mktime(dt.timetuple()))
-            return int(seconds) * 10 ** 6 + dt.microsecond
+            return int(seconds) * 10 ** 6 + dt.microsecond + offset_microseconds


Take local timezone into consideration? For example, time 0h in Japen should return -1h in China.

hequn8128 · 2020-03-19T03:40:01Z

flink-python/pyflink/fn_execution/coders.py

        return datetime.datetime


+class LocalZonedTimestampCoder(DeterministicCoder):


Add python tests for the coder. e.g. tests in test_coders_common.py.

dianfu · 2020-03-19T05:39:29Z

@hequn8128 Thanks a lot for the review. Updated the PR.

hequn8128 · 2020-03-19T06:01:35Z

LGTM. Merge this once test passed.

hequn8128 · 2020-03-19T15:40:16Z

@flinkbot run azure

dianfu · 2020-03-20T01:21:05Z

The failed azure test is a known issue and has already been tracked in https://issues.apache.org/jira/browse/FLINK-16676

…in blink planner (apache#11439)

[FLINK-16650][python] Support LocalZonedTimestampType for Python UDF …

b8aa27e

…in blink planner

hequn8128 self-assigned this Mar 18, 2020

rmetzger added the review=description? label Mar 18, 2020

rmetzger added the component=API/Python label Mar 18, 2020

fix tests

d65c0ec

hequn8128 reviewed Mar 19, 2020

View reviewed changes

Address review

a6a131a

dianfu force-pushed the FLINK-16650 branch from 49d8110 to a6a131a Compare March 19, 2020 05:43

dianfu added 2 commits March 19, 2020 15:14

fix tests

b7fb730

fix tests

e09703f

hequn8128 merged commit 5ccb167 into apache:master Mar 20, 2020

liuzhixing1006 pushed a commit to liuzhixing1006/flink that referenced this pull request Mar 20, 2020

[FLINK-16650][python] Support LocalZonedTimestampType for Python UDF …

4b6d32f

…in blink planner (apache#11439)

dianfu deleted the FLINK-16650 branch June 10, 2020 03:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-16650][python] Support LocalZonedTimestampType for Python UDF in blink planner #11439

[FLINK-16650][python] Support LocalZonedTimestampType for Python UDF in blink planner #11439

Uh oh!

dianfu commented Mar 18, 2020

Uh oh!

flinkbot commented Mar 18, 2020

Uh oh!

flinkbot commented Mar 18, 2020 •

edited

Loading

Uh oh!

hequn8128 left a comment

Uh oh!

hequn8128 Mar 19, 2020

Uh oh!

hequn8128 Mar 19, 2020

Uh oh!

hequn8128 Mar 19, 2020

Uh oh!

dianfu commented Mar 19, 2020

Uh oh!

hequn8128 commented Mar 19, 2020

Uh oh!

hequn8128 commented Mar 19, 2020

Uh oh!

dianfu commented Mar 20, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		return datetime.datetime


		class LocalZonedTimestampCoder(DeterministicCoder):

[FLINK-16650][python] Support LocalZonedTimestampType for Python UDF in blink planner #11439

[FLINK-16650][python] Support LocalZonedTimestampType for Python UDF in blink planner #11439

Uh oh!

Conversation

dianfu commented Mar 18, 2020

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

flinkbot commented Mar 18, 2020

Automated Checks

Review Progress

Uh oh!

flinkbot commented Mar 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

hequn8128 left a comment

Choose a reason for hiding this comment

Uh oh!

hequn8128 Mar 19, 2020

Choose a reason for hiding this comment

Uh oh!

hequn8128 Mar 19, 2020

Choose a reason for hiding this comment

Uh oh!

hequn8128 Mar 19, 2020

Choose a reason for hiding this comment

Uh oh!

dianfu commented Mar 19, 2020

Uh oh!

hequn8128 commented Mar 19, 2020

Uh oh!

hequn8128 commented Mar 19, 2020

Uh oh!

dianfu commented Mar 20, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

flinkbot commented Mar 18, 2020 •

edited

Loading