[BEAM-3177][BEAM-5468] Add pipeline options to set default logging level

[Abacn]

• Python counterpart of existing --defaultSdkHarnessLogLevel pipeline options
  in Java sdk

• Still need updates from worker side to make it working in DataflowRunner

Please add a meaningful description for your change here

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Add a link to the appropriate issue in your description, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[asf-ci]

Can one of the admins verify this patch?

[asf-ci]

Can one of the admins verify this patch?

[codecov]

Codecov Report

Merging #21724 (8847486) into master (b133282) will increase coverage by 0.11%.
The diff coverage is 93.33%.

@@            Coverage Diff             @@
##           master   #21724      +/-   ##
==========================================
+ Coverage   74.01%   74.12%   +0.11%     
==========================================
  Files         699      702       +3     
  Lines       92675    93678    +1003     
==========================================
+ Hits        68592    69440     +848     
- Misses      22828    22983     +155     
  Partials     1255     1255              

Flag	Coverage Δ
python	83.67% <93.33%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdks/python/apache_beam/runners/worker/logger.py	93.65% <50.00%> (ø)
...dks/python/apache_beam/options/pipeline_options.py	94.95% <100.00%> (+0.01%)	⬆️
...thon/apache_beam/runners/worker/sdk_worker_main.py	75.81% <100.00%> (+1.69%)	⬆️
sdks/python/apache_beam/io/localfilesystem.py	90.97% <0.00%> (-0.76%)	⬇️
...eam/runners/portability/fn_api_runner/execution.py	92.44% <0.00%> (-0.65%)	⬇️
...hon/apache_beam/runners/worker/bundle_processor.py	93.30% <0.00%> (-0.25%)	⬇️
sdks/python/apache_beam/transforms/util.py	96.06% <0.00%> (-0.16%)	⬇️
sdks/python/apache_beam/io/fileio.py	96.23% <0.00%> (-0.10%)	⬇️
sdks/python/apache_beam/ml/inference/__init__.py	100.00% <0.00%> (ø)
...thon/apache_beam/ml/inference/pytorch_inference.py	0.00% <0.00%> (ø)
... and 20 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b133282...8847486. Read the comment docs.

[Abacn]

R: @tvalentyn
R: @robertwb

[Abacn]

Was doing BEAM-14147 but apparently this is a blocker. Along with this change still need an update from dataflow worker side to call logger.initialize with the added log_level parameter.

[tvalentyn]

cc: @ryanthompson591

[robertwb]

The failure at apache_beam.runners.portability.fn_api_runner.fn_runner_test.FnApiRunnerTestWithGrpcAndMultiWorkers.test_pardo_large_inputlooks irrelevant.

[robertwb]

Run Python PreCommit

[johnjcasey]

LGTM

[Abacn]

rebased due to conflicts of TODO label changes on master

[Abacn]

Run Python PreCommit