Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-10603] Add describe and cancel to RecordingManager #12703

Merged
merged 3 commits into from Sep 8, 2020

Conversation

rohdesamuel
Copy link
Contributor

Change-Id: I414463dc40a58c1452b7f4a6e819b9e054e4cac8

This adds the ability to introspect into the RecordingManager to get the start time and size on cache for a recording or pipeline (RecordingManager). Also adds the "cancel" method to a recording. This allows to gracefully shutdown a recording if a kernel interrupt is caught.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang SDK Dataflow Flink Samza Spark Twister2
Go Build Status --- Build Status --- Build Status ---
Java Build Status Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status Build Status
Build Status
Build Status
Build Status
Python Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
--- Build Status ---
XLang Build Status --- Build Status --- Build Status ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website Whitespace
Non-portable Build Status Build Status
Build Status
Build Status
Build Status
Build Status Build Status Build Status
Portable --- Build Status --- --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests

See CI.md for more information about GitHub Actions CI.

@rohdesamuel
Copy link
Contributor Author

Run Portable_Python PreCommit

@codecov
Copy link

codecov bot commented Aug 31, 2020

Codecov Report

Merging #12703 into master will increase coverage by 0.04%.
The diff coverage is 20.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #12703      +/-   ##
==========================================
+ Coverage   40.22%   40.26%   +0.04%     
==========================================
  Files         454      455       +1     
  Lines       53669    53847     +178     
==========================================
+ Hits        21587    21681      +94     
- Misses      32082    32166      +84     
Impacted Files Coverage Δ
...beam/runners/interactive/background_caching_job.py 25.00% <0.00%> (-0.25%) ⬇️
...am/runners/interactive/options/capture_limiters.py 40.32% <0.00%> (ø)
...ache_beam/runners/interactive/recording_manager.py 29.05% <20.58%> (-0.55%) ⬇️
...he_beam/runners/interactive/pipeline_instrument.py 14.86% <50.00%> (+0.16%) ⬆️
...hon/apache_beam/runners/direct/test_stream_impl.py 41.17% <0.00%> (-1.58%) ⬇️
.../runners/portability/fn_api_runner/translations.py 13.62% <0.00%> (-0.16%) ⬇️
sdks/python/apache_beam/portability/python_urns.py 100.00% <0.00%> (ø)
...s/python/apache_beam/io/gcp/bigquery_file_loads.py 23.36% <0.00%> (ø)
sdks/python/apache_beam/io/kinesis.py 66.66% <0.00%> (ø)
...hon/apache_beam/runners/worker/bundle_processor.py 28.92% <0.00%> (+0.24%) ⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 31af8b1...0d7add9. Read the comment docs.

Change-Id: I414463dc40a58c1452b7f4a6e819b9e054e4cac8
@rohdesamuel
Copy link
Contributor Author

Run Python PreCommit

1 similar comment
@rohdesamuel
Copy link
Contributor Author

Run Python PreCommit

@rohdesamuel rohdesamuel marked this pull request as ready for review September 1, 2020 21:01
@rohdesamuel
Copy link
Contributor Author

R: @davidyan74

@rohdesamuel
Copy link
Contributor Author

R: @KevinGG

@rohdesamuel
Copy link
Contributor Author

Run Python PreCommit

Copy link
Contributor

@KevinGG KevinGG left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@davidyan74 davidyan74 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Change-Id: Ifd0bd62c95958a96728c293628b6ed5be2b38790
@rohdesamuel
Copy link
Contributor Author

Run Portable_Python PreCommit

@rohdesamuel
Copy link
Contributor Author

Run PythonDocker PreCommit

1 similar comment
@rohdesamuel
Copy link
Contributor Author

Run PythonDocker PreCommit

@rohdesamuel
Copy link
Contributor Author

R: @pabloem

@rohdesamuel
Copy link
Contributor Author

Run Portable_Python PreCommit

@rohdesamuel
Copy link
Contributor Author

Run PythonDocker PreCommit

@pabloem pabloem self-requested a review September 8, 2020 13:56
Change-Id: Ie74c50e7624df1b0611b35fe98d12eff905520aa
@pabloem pabloem merged commit 71d5395 into apache:master Sep 8, 2020
ibzib pushed a commit to ibzib/beam that referenced this pull request Sep 30, 2020
…ncel to RecordingManager

* Add describe and cancel to RecordingManager

Change-Id: I414463dc40a58c1452b7f4a6e819b9e054e4cac8

* use the existing method to cancel the BCJ

Change-Id: Ifd0bd62c95958a96728c293628b6ed5be2b38790

* Turn cache_key and var into properties

Change-Id: Ie74c50e7624df1b0611b35fe98d12eff905520aa

Co-authored-by: Sam Rohde <srohde@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants