[BEAM-10880] Log error counts to debug BigQuery streaming insert requ…

[ihji]

…ests for Python SDK

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	SDK	Dataflow	Flink	Samza	Spark	Twister2
Go		---		---		---
Java
Python				---		---
XLang		---		---		---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---		---	---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[codecov]

Codecov Report

Merging #12822 into master will increase coverage by 41.92%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           master   #12822       +/-   ##
===========================================
+ Coverage   40.38%   82.31%   +41.92%     
===========================================
  Files         451      451               
  Lines       53721    53849      +128     
===========================================
+ Hits        21696    44324    +22628     
+ Misses      32025     9525    -22500     

Impacted Files	Coverage Δ
...dks/python/apache_beam/options/pipeline_options.py	94.98% <ø> (+38.99%)	⬆️
sdks/python/apache_beam/io/gcp/bigquery.py	80.55% <100.00%> (+53.45%)	⬆️
sdks/python/apache_beam/io/gcp/bigquery_tools.py	88.05% <100.00%> (+59.48%)	⬆️
sdks/python/apache_beam/utils/histogram.py	94.28% <100.00%> (+54.28%)	⬆️
...apache_beam/portability/api/beam_runner_api_pb2.py	100.00% <0.00%> (ø)
...e_beam/portability/api/beam_runner_api_pb2_urns.py	100.00% <0.00%> (ø)
...ache_beam/runners/interactive/pipeline_analyzer.py
...beam/testing/benchmarks/nexmark/queries/query10.py	0.00% <0.00%> (ø)
sdks/python/apache_beam/coders/slow_stream.py	92.43% <0.00%> (+1.68%)	⬆️
sdks/python/apache_beam/utils/profiler.py	32.11% <0.00%> (+1.83%)	⬆️
... and 275 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5705526...dd6daaf. Read the comment docs.

[ihji]

Logging example:

[Streaming Insert API Statistics since 2020-09-12 01:30:39.483000] Total number of streaming insert requests: 2708, P99: 4078ms, P90: 623ms, P50: 232ms HttpError counts: { "invalid(400)": 6, "notFound(404)": 9 }

[ihji]

R: @chamikaramj @ajamato

[ajamato]

Please convert this to the human readable format instead of the numeric http code. See the doc I shared with you.

At least before logging
https://cloud.google.com/apis/design/errors

[ihji]

It records string error codes and numeric http codes from BigQuery API JSON response https://cloud.google.com/bigquery/docs/error-messages. For example:

{quotaExceeded(403): 461, rateLimitExceeded(403): 2724}

I think string error codes here are human readable and clearly shows the cause of errors. Do you want to append any other string messages to them?

[ihji]

@ajamato PTAL.

[chamikaramj]

What's the status of this ?

[ihji]

@chamikaramj Alex and I talked about current implementation and wanted to exploit existing Metric framework as working on #13017. We may close this and #12852 since adding error counts is impossible on the Metric framework (if we don't use the Metric framework it will add much complexity but the benefit is rather small).