Skip to content

Use daemon thread to report lull operation and move code to FnApiWork…#15676

Merged
y1chi merged 1 commit intoapache:masterfrom
y1chi:lull_logging
Oct 14, 2021
Merged

Use daemon thread to report lull operation and move code to FnApiWork…#15676
y1chi merged 1 commit intoapache:masterfrom
y1chi:lull_logging

Conversation

@y1chi
Copy link
Copy Markdown

@y1chi y1chi commented Oct 7, 2021

…erStatusHandler

Use a deamon thread guarantees that the lull operations are checked with fixed time interval instead of relying on trigger of runner process_bundle_progress request (which may not be happening frequently as needed if no work split is intended).
Code has been moved from sdk worker to the worker status handler since all logic of printing thread dump is also there. Removed full thread dump logging since it is often unreadable in logs plus there are other alternatives to get pretty print full thread dumps such as requesting sdk_status page from runner.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

ValidatesRunner compliance status (on master branch)

Lang ULR Dataflow Flink Samza Spark Twister2
Go --- Build Status Build Status Build Status Build Status ---
Java Build Status Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Python --- Build Status
Build Status
Build Status
Build Status
Build Status
Build Status Build Status ---
XLang Build Status Build Status Build Status Build Status Build Status ---

Examples testing status on various runners

Lang ULR Dataflow Flink Samza Spark Twister2
Go --- --- --- --- --- --- ---
Java --- Build Status
Build Status
Build Status
--- --- --- --- ---
Python --- --- --- --- --- --- ---
XLang --- --- --- --- --- --- ---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go Java Python
Build Status Build Status Build Status
Build Status
Build Status

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website Whitespace Typescript
Non-portable Build Status
Build Status
Build Status
Build Status
Build Status
Build Status Build Status Build Status Build Status
Portable --- Build Status Build Status --- --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests

See CI.md for more information about GitHub Actions CI.

@codecov
Copy link
Copy Markdown

codecov bot commented Oct 7, 2021

Codecov Report

Merging #15676 (d296688) into master (d6b9156) will decrease coverage by 0.30%.
The diff coverage is 80.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #15676      +/-   ##
==========================================
- Coverage   83.80%   83.50%   -0.31%     
==========================================
  Files         444      445       +1     
  Lines       60465    61290     +825     
==========================================
+ Hits        50675    51182     +507     
- Misses       9790    10108     +318     
Impacted Files Coverage Δ
...ks/python/apache_beam/runners/worker/sdk_worker.py 89.25% <ø> (+0.09%) ⬆️
...python/apache_beam/runners/worker/worker_status.py 79.71% <80.00%> (+2.15%) ⬆️
sdks/python/apache_beam/io/gcp/bigquery.py 62.72% <0.00%> (-12.84%) ⬇️
...ython/apache_beam/runners/interactive/sql/utils.py 76.09% <0.00%> (-7.91%) ⬇️
...he_beam/runners/interactive/sql/beam_sql_magics.py 49.75% <0.00%> (-4.79%) ⬇️
.../python/apache_beam/testing/test_stream_service.py 88.37% <0.00%> (-4.66%) ⬇️
...ython/apache_beam/io/gcp/bigquery_read_internal.py 53.92% <0.00%> (-4.24%) ⬇️
sdks/python/apache_beam/io/gcp/bigquery_tools.py 82.91% <0.00%> (-3.82%) ⬇️
...hon/apache_beam/runners/direct/test_stream_impl.py 94.02% <0.00%> (-2.24%) ⬇️
...che_beam/runners/interactive/interactive_runner.py 90.74% <0.00%> (-1.86%) ⬇️
... and 49 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d6b9156...d296688. Read the comment docs.

@y1chi y1chi force-pushed the lull_logging branch 2 times, most recently from 6fb8418 to e4371bb Compare October 7, 2021 18:26
@y1chi y1chi requested a review from pabloem October 7, 2021 19:36
@y1chi
Copy link
Copy Markdown
Author

y1chi commented Oct 7, 2021

R: @pabloem

@y1chi
Copy link
Copy Markdown
Author

y1chi commented Oct 11, 2021

Run Python PreCommit

@y1chi
Copy link
Copy Markdown
Author

y1chi commented Oct 12, 2021

@pabloem gentle ping for review

@pabloem
Copy link
Copy Markdown
Member

pabloem commented Oct 14, 2021

LGTM! Thanks @y1chi great improvement

@pabloem
Copy link
Copy Markdown
Member

pabloem commented Oct 14, 2021

feel free to merge

@y1chi y1chi merged commit bd0ac38 into apache:master Oct 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants