[BEAM-2914] Add portable merging window support to Python. #12995

robertwb · 2020-10-01T23:34:35Z

This adds support to the Python SDK and the Python ULR.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	Dataflow	Samza	Twister2
Go	---	---	---
Java
Python		---	---
XLang	---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---		---	---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

robertwb · 2020-10-01T23:34:46Z

R: @angoenka

codecov · 2020-10-03T00:45:39Z

Codecov Report

Merging #12995 (8c10756) into master (b3fd2bd) will increase coverage by 0.04%.
The diff coverage is 96.72%.

@@            Coverage Diff             @@
##           master   #12995      +/-   ##
==========================================
+ Coverage   82.86%   82.90%   +0.04%     
==========================================
  Files         466      466              
  Lines       57599    57723     +124     
==========================================
+ Hits        47730    47858     +128     
+ Misses       9869     9865       -4

Impacted Files	Coverage Δ
...on/apache_beam/runners/dataflow/dataflow_runner.py	`82.28% <ø> (ø)`
...hon/apache_beam/runners/worker/bundle_processor.py	`93.99% <95.23%> (+0.03%)`	⬆️
...eam/runners/portability/fn_api_runner/execution.py	`93.68% <96.87%> (+1.64%)`	⬆️
sdks/python/apache_beam/dataframe/frames.py	`91.07% <100.00%> (ø)`
sdks/python/apache_beam/io/localfilesystem.py	`91.66% <0.00%> (-0.76%)`	⬇️
sdks/python/apache_beam/dataframe/expressions.py	`91.02% <0.00%> (-0.14%)`	⬇️
.../runners/portability/fn_api_runner/translations.py	`92.10% <0.00%> (-0.08%)`	⬇️
...apache_beam/runners/portability/portable_runner.py	`76.30% <0.00%> (ø)`
...eam/runners/portability/fn_api_runner/fn_runner.py	`89.87% <0.00%> (ø)`
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 091ecdf...8c10756. Read the comment docs.

robertwb · 2020-10-12T16:52:51Z

Ping @angoenka

robertwb · 2020-12-03T01:34:24Z

Run Python PreCommit

y1chi · 2021-01-06T22:08:19Z

sdks/python/apache_beam/runners/portability/fn_api_runner/execution.py

+    if self._worker_handle is None:
+      worker_handler_manager = self._execution_context_ref(
+      ).worker_handler_manager
+      self._worker_handler = worker_handler_manager.get_worker_handlers(


self._worker_handler -> self._worker_handle?

Good catch. Fixed.

y1chi

LGTM, can we merge this?

y1chi · 2021-02-08T20:26:19Z

sdks/python/apache_beam/runners/portability/flink_runner_test.py

@@ -400,6 +400,9 @@ def test_callbacks_with_exception(self):
  def test_register_finalizations(self):
    raise unittest.SkipTest("BEAM-6868")

+  def test_custom_merging_window(self):


we may need to add this to spark_runner_test.py as well.

robertwb · 2021-02-08T21:20:52Z

Thanks for looking at this. I'll fix the merge conflicts, address your comments, and we can get this in.

robertwb · 2021-02-10T20:01:08Z

Run Python PreCommit

robertwb · 2021-02-10T20:01:15Z

Run Portable_Python PreCommit

robertwb · 2021-02-11T01:40:31Z

Flake in apache_beam.runners.interactive.utils_test.ProgressIndicatorTest.test_progress_in_HTML_JS_when_in_notebook

robertwb · 2021-02-11T01:43:40Z

Run Python PreCommit

TheNeuralBit · 2021-02-11T19:45:27Z

I think this broke XVR_Direct, see BEAM-11803

TheNeuralBit · 2021-02-11T21:42:21Z

I think it also broke Java VR on ULR: https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/534/

…pache#12995)" This reverts commit 625ee1f.

[BEAM-11822] Revert "[BEAM-2914] Add portable merging window support to Python. (#12995)"

…thon. (apache#12995)"" This reverts commit 9c60fd5.

…gger * github/master: (123 commits) [BEAM-11899] Bump commons-pool to 2.8.1 and bump commons-dbcp to 2.8.0, Because there is a library dependency Update pillars.yaml (apache#14142) [BEAM-10632] Checkerframework nullness cleanups (apache#14107) [BEAM-11213] Instantiate SparkListenerApplicationStart in a Spark 3 compatible way Fix typos for excluding testMergingCustomWindowsWithoutCustomWindowTypes Specify the time resolution for TestStreamPayload. [BEAM-10961] Enable strict depdency checking for sdks/java/extensions/euphoria (second attempt) [BEAM-11848] Store Docker images in a variable for consistency. Splitting old Go Precommit and new ULR integration test precommit. Moving runner imports out of ptest. Add the TO_STRING capability to Java and Python [BEAM-11848] Fix Docker images list. jdbc python supported Dataflow runner (apache#13960) Adding a warning to use multi-workers on FnApiRunner Fix legend for Python Directrunner microbenchmarks [BEAM-11740] Estimate PCollection byte size [BEAM-10961] enable strict dependency checking for sdks/java/extensions/zetasketch (apache#14093) Map Dataflow JOB_STATE_CANCELLING to Beam RUNNING state [BEAM-11833] Fix reported watermark after restriction split in UnboundedSourceAsSDFRestrictionTracker [BEAM-10761] add reference to BEAM-11761 [BEAM-10961] enable strict dependency checking for flink/job-server Exclude MapState example integration tests from Dataflow runner v2 suite Remove InvalidWindows from Java SDK, instead track "already merged" bit Fix checkstyle in watermark latency benchmark Fix compile breakage in WindmillStateInternals Improve test, error on ALREADY_MERGED. [BEAM-10961] Strict dependency checking for sdks/java/io/gcp (apache#13791) Initial watermark latency benchmark Attempting improvements on DirectRunner Python dash [BEAM-10961] enable strict dependency checking for sdks/java/extensions/google-cloud-platform-core (apache#14084) Merge pull request apache#13802: [BEAM-1474]. Adding MapState and SetState support for the Dataflow runner Remove some false positives Remove nullness warning suppression [BEAM-11861] Add methods to explicitly provide coder for ParquetIO's Parse and ParseFiles (apache#14078) [BEAM-11531] Use pandas 1.2 for python>=3.7 (apache#14099) [BEAM-10961] add reference to BEAM-11761 [BEAM-10961] add explicit compile for auto_value_annotations in sdks/extensions/ml/build.gradle Attempting improvements on DirectRunner Python dash Recognize JOB_STATE_PENDING from Dataflow and map to RUNNING never run checkerframework on tests Puts more expensive BQ empty table check to the right of the 'and' condition (apache#14094) Use the windowing strategy of the input, not output, PCollection of GBK. Do not stage dataflow worker jar when use runner_v2. [BEAM-11870] Re-raise underlying exception for InvocationTargetException (apache#14098) [BEAM-11778] Create a wrapper for ZetaSQL catalog and refactor accordingly. (apache#13934) [BEAM-9378] Add ignored tests which fail in various ways when querying nested structures (apache#14077) Merge Fn API and runner v2 configurations for DataflowRunner Fix up! formatting Add validate runner test for testing custom merging windows fn without custom window types Revert "Revert "[BEAM-2914] Add portable merging window support to Python. (apache#12995)"" [BEAM-10961] fix stray reordering of lines [BEAM-10961] enable strict dependency checking for sdks/java/extensions/sorter [BEAM-10961] enable strict dependency checking for sdks/java/extensions/sketching [BEAM-10961] enable strict dependency checking for sdks/java/extensions/schemaio-expansion-service [BEAM-10961] enable strict dependency checking for sdks/java/extensions/protobuf [BEAM-10961] enable strict dependency checking for sdks/java/extensions/ml [BEAM-10961] enable strict dependency checking for sdks/java/extensions/kyro [BEAM-10961] enable strict dependency checking for sdks/java/extensions/join-library [BEAM-10961] undo line moves (originally intended for alphabeticization) [BEAM-10961] enable strict dependency checking for sdks/java/extensions/jackson [BEAM-10961] Enable strict dependency checking on sdks/java/extensions/sql (apache#13830) [BEAM-10961] enable strict dependency checking for sdks/java/extensions/euphoria [BEAM-10961] enable strict dependency checking for sdks/java/io/parquet (apache#14062) [BEAM-10961] enable strict dependency checking for sdks/java/io/thrift (apache#14066) Refactor ZetaSqlDialectSpecTest and add some passing tests. (apache#14080) [BEAM-11864] Use objects.equals instead of raw comparison [BEAM-11707] Change WindmillStateCache cache invalidation to be based upon reference invalidation instead of expensive set management. Reduce operations of shared cache by caching per-key object sets locally and flushing as groups to shared cache. Remove byte tracking which could be racy based upon background evictions in favor of just iterating for rendering the status page. This also lets us capture more stats. [BEAM-11730] Reduce context switching overhead for appliance reads by issuing reads directly from calling threads in the case that there is no reads being queued. Fix preview Show string from Dataflow service when job terminates in unrecognized state Log a warning when Dataflow returns an unrecognized state Merge pull request apache#14033 from [BEAM-11408] Integrate Python BigQuery sink with GroupIntoBatches Remove SYNCHRONIZED_PROCESSING_TIME from model proto Remove use of model SYNCHRONIZED_PROCESSING_TIME Merge redundant model feature columns in capability matrix Remove MapReduce runner from capability matrix, because it is on a branch and unreleased Remove JStorm runner from capability matrix, because it is on a branch and unreleased Remove retractions from capability matrix, because they do not exist yet Remove metadata-driven triggers from capability matrix, because they do not exist [BEAM-10937] Add Tour of Beam page (apache#13747) [BEAM-11344] Apply "Become a Committer" changes from Website Revamp (apache#14036) Merge pull request apache#14046 from [BEAM-11791] Adding a microbenchmark for TestStream Returning successful writes in FhirIO.Write.Result (apache#14034) Fixup [BEAM-10961] enable strict dependency checking for sdks/java/io/file-based-io-tests (apache#14052) [BEAM-10961] enable strict dependency checking for sdks/java/io/contextualtextio (apache#14049) [BEAM-10961] enable strict dependency checking for sdks/java/io/kinesis (apache#14058) [BEAM-10961] enable strict dependency checking for sdks/java/io/bigquery-io-perf-tests (apache#14048) [BEAM-10961] enable strict dependency checking for sdks/java/io/elasticsearch (apache#14050) [BEAM-10961] enable strict dependency checking for sdks/java/io/expansion-service (apache#14051) [BEAM-10961] enable strict dependency checking for sdks/java/io/jdbc (apache#14055) [BEAM-10961] enable strict dependency checking for sdks/java/io/jms (apache#14056) [BEAM-10961] enable strict dependency checking for sdks/java/io/kafka (apache#14057) [BEAM-10961] enable strict dependency checking for sdks/java/io/hcatalog (apache#14053) [BEAM-11859] Fixed bug in python S3 IO [BEAM-10114] Fix PerSubscriptionPartitionSdf to not rely on the presence of BundleFinalizer [BEAM-10114] Fix PerSubscriptionPartitionSdf to not rely on the presence of BundleFinalizer [BEAM-10961] fix spacing [BEAM-10961] enable strict dependency checking for sdks/java/io/xml [BEAM-10961] enable strict dependency checking for sdks/java/io/tika ...

…thon. (apache#12995)"" This reverts commit 9c60fd5.

[BEAM-2914] Add portable merging window support to Python.

8bca5d7

probot-autolabeler bot added the python label Oct 1, 2020

robertwb added 2 commits October 2, 2020 16:39

yapf

23dc93d

Skip flink, grpc fixes.

6afead4

robertwb added 5 commits October 2, 2020 18:05

yapf

8c95a28

Merge branch 'master' into custom-window-merging

745b86b

Merge branch 'master' into custom-window-merging

d8a2445

fix merge

e647cfe

merge fix

3eb748e

robertwb added 2 commits October 12, 2020 09:53

lint

8af2665

lint

37986fe

y1chi reviewed Jan 7, 2021

View reviewed changes

y1chi approved these changes Feb 8, 2021

View reviewed changes

y1chi reviewed Feb 8, 2021

View reviewed changes

robertwb added 4 commits February 8, 2021 13:34

Address reviewer comments.

be5c297

lint

2d29890

mypy

9978c43

yapf

8c10756

robertwb merged commit 625ee1f into apache:master Feb 11, 2021

ajamato pushed a commit to ajamato/beam that referenced this pull request Feb 17, 2021

Revert "[BEAM-2914] Add portable merging window support to Python. (a…

9c60fd5

…pache#12995)" This reverts commit 625ee1f.

robertwb added a commit that referenced this pull request Feb 17, 2021

Merge pull request #14004 from ajamato/rollback

871e023

[BEAM-11822] Revert "[BEAM-2914] Add portable merging window support to Python. (#12995)"

robertwb added a commit to robertwb/incubator-beam that referenced this pull request Feb 25, 2021

Revert "Revert "[BEAM-2914] Add portable merging window support to Py…

7a27c79

…thon. (apache#12995)"" This reverts commit 9c60fd5.

kileys pushed a commit to kileys/beam that referenced this pull request Mar 11, 2021

Revert "Revert "[BEAM-2914] Add portable merging window support to Py…

00812db

…thon. (apache#12995)"" This reverts commit 9c60fd5.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-2914] Add portable merging window support to Python. #12995

[BEAM-2914] Add portable merging window support to Python. #12995

robertwb commented Oct 1, 2020

robertwb commented Oct 1, 2020

codecov bot commented Oct 3, 2020 •

edited

robertwb commented Oct 12, 2020

robertwb commented Dec 3, 2020

y1chi Jan 6, 2021

robertwb Feb 8, 2021

y1chi left a comment

y1chi Feb 8, 2021

robertwb Feb 8, 2021

robertwb commented Feb 8, 2021

robertwb commented Feb 10, 2021

robertwb commented Feb 10, 2021

robertwb commented Feb 11, 2021

robertwb commented Feb 11, 2021

TheNeuralBit commented Feb 11, 2021

TheNeuralBit commented Feb 11, 2021

[BEAM-2914] Add portable merging window support to Python. #12995

[BEAM-2914] Add portable merging window support to Python. #12995

Conversation

robertwb commented Oct 1, 2020

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

robertwb commented Oct 1, 2020

codecov bot commented Oct 3, 2020 • edited

Codecov Report

robertwb commented Oct 12, 2020

robertwb commented Dec 3, 2020

y1chi Jan 6, 2021

Choose a reason for hiding this comment

robertwb Feb 8, 2021

Choose a reason for hiding this comment

y1chi left a comment

Choose a reason for hiding this comment

y1chi Feb 8, 2021

Choose a reason for hiding this comment

robertwb Feb 8, 2021

Choose a reason for hiding this comment

robertwb commented Feb 8, 2021

robertwb commented Feb 10, 2021

robertwb commented Feb 10, 2021

robertwb commented Feb 11, 2021

robertwb commented Feb 11, 2021

TheNeuralBit commented Feb 11, 2021

TheNeuralBit commented Feb 11, 2021

codecov bot commented Oct 3, 2020 •

edited