Add logging in data plane code when processing is stuck waiting on data for an instruction.

[tvalentyn]

Log when elements for an instruction are missing in data plane more than 5 minutes. This can help surface processing inefficiency between an SDK and a runner.

[github-actions]

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @jrmccluskey added as fallback since no labels match configuration

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[github-actions]

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @jrmccluskey added as fallback since no labels match configuration

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[tvalentyn]

Run Python_PVR_Flink PreCommit

[tvalentyn]

Run Python_Coverage PreCommit

[tvalentyn]

Run PythonDocker PreCommit

[tvalentyn]

cc: @robertwb

[codecov]

Codecov Report

Attention: 17 lines in your changes are missing coverage. Please review.

Comparison is base (c713425) 38.34% compared to head (545f44e) 38.33%.
Report is 23 commits behind head on master.

Files	Patch %	Lines
...ks/python/apache_beam/runners/worker/data_plane.py	55.55%	4 Missing ⚠️
sdks/python/apache_beam/transforms/external.py	0.00%	4 Missing ⚠️
sdks/python/apache_beam/dataframe/expressions.py	0.00%	3 Missing ⚠️
sdks/python/apache_beam/runners/worker/logger.py	0.00%	3 Missing ⚠️
...beam/transforms/fully_qualified_named_transform.py	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #29399      +/-   ##
==========================================
- Coverage   38.34%   38.33%   -0.01%     
==========================================
  Files         693      694       +1     
  Lines      102199   102265      +66     
==========================================
+ Hits        39185    39204      +19     
- Misses      61422    61469      +47     
  Partials     1592     1592              

Flag	Coverage Δ
python	29.88% <22.72%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[liferoad]

can we add log_interval_sec in the log and mention this frequency?
Was this tested with Dataflow jobs?

[tvalentyn]

Thanks, yes I tested it on dataflow to make sure there were no typos; it was a modified version of the PR to trigger the branches without waiting for 5 min.

[tvalentyn]

can we add log_interval_sec in the log and mention this frequency?

done

[jrmccluskey]

LGTM, but +1 to @liferoad's question asking if this had been tested yet