[BEAM-9561] Improve WontImplement reporting.

[robertwb]

Only track as WontImplement those tests that are actually raising WontImplement errors.

Also report summary on failure.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	SDK	Dataflow	Flink	Samza	Spark	Twister2
Go		---		---		---
Java
Python				---		---
XLang		---		---		---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---		---	---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[codecov]

Codecov Report

Merging #12815 into master will increase coverage by 41.92%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           master   #12815       +/-   ##
===========================================
+ Coverage   40.43%   82.36%   +41.92%     
===========================================
  Files         449      451        +2     
  Lines       53502    53748      +246     
===========================================
+ Hits        21635    44271    +22636     
+ Misses      31867     9477    -22390     

Impacted Files	Coverage Δ
sdks/python/apache_beam/dataframe/doctests.py	97.26% <100.00%> (+72.37%)	⬆️
...he_beam/testing/benchmarks/nexmark/nexmark_util.py	0.00% <0.00%> (ø)
...eam/testing/benchmarks/nexmark/nexmark_launcher.py	0.00% <0.00%> (ø)
...he_beam/testing/benchmarks/nexmark/nexmark_perf.py	0.00% <0.00%> (ø)
sdks/python/apache_beam/utils/histogram.py	94.28% <0.00%> (ø)
sdks/python/apache_beam/coders/slow_stream.py	92.43% <0.00%> (+1.68%)	⬆️
sdks/python/apache_beam/utils/profiler.py	32.11% <0.00%> (+1.83%)	⬆️
...on/apache_beam/runners/direct/sdf_direct_runner.py	36.21% <0.00%> (+2.46%)	⬆️
sdks/python/apache_beam/coders/coder_impl.py	95.25% <0.00%> (+2.64%)	⬆️
sdks/python/apache_beam/io/azure/blobstorageio.py	26.95% <0.00%> (+3.19%)	⬆️
... and 272 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d2fc192...40cdaf6. Read the comment docs.

[robertwb]

Run Portable_Python PreCommit

[robertwb]

R: @y1chi

[robertwb]

E.g. Before

test_dataframe_tests (apache_beam.dataframe.pandas_doctests_test.DoctestTest) ... 
572 total test cases:
  223 skipped (39.0%)
  112 won't implement (19.6%)
    33  (29.5%)
    22 WontImplementError: order-sensitive (19.6%)
    20 DeferredBase[*] (17.9%)
    11 WontImplementError: non-deferred value (9.8%)
    7 WontImplementError: non-deferred column values (6.2%)
    7 NameError (6.2%)
    6 WontImplementError: non-lazy (5.4%)
    2 col1    int64\ncol2    int64\ndtype: object\n (1.8%)
    2 name         object\nscore       float64\nemployed       bool\nkids          int64\ndtype: object\n (1.8%)
    2 WontImplementError: scalar value (1.8%)
  237 passed (41.4%)

after

test_dataframe_tests (apache_beam.dataframe.pandas_doctests_test.DoctestTest) ... 
572 total test cases:
  223 skipped (39.0%)
  55 won't implement (9.6%)
    22 order-sensitive (40.0%)
    11 non-deferred value (20.0%)
    7 non-deferred column values (12.7%)
    7 NameError following WontImplementError (12.7%)
    6 non-lazy (10.9%)
    2 scalar value (3.6%)
  294 passed (51.4%)

[y1chi]

LGTM

[y1chi]

I assume all the errors here are under the precondition that WontImplementError is seen thus even the regex is not matched we know it is following WontImplementError?

[robertwb]

Correct.

[robertwb]

Run Python2_PVR_Flink PreCommit