[observability] Add gRPC client call and additional functional metrics

[csweichel]

Description

Adds client call metrics to gRPC calls form TS and Go, and makes the server -> image-builder connection use the default gRPC unary options. The lack of the latter probably was the cause of the recent image-builder related outage.

Also adds cluster registration metrics to ws-manager-bridge, and image-build counts to image-builder-mk3

Thanks @JanKoehnlein for figuring out the nitty gritty of the grpc-js interceptor business.

Related Issue(s)

Fixes #5706, #5707

How to test

1. start a workspace to produce some data all the way 'round
2. look at the metric endpoints of server, image-builder-mk3, ws-manager-bridge, ws-manager, ws-proxy, ws-scheduler. You should see grpc_client_calls_total metrics on all those services.

Release Notes

[server] Add gRPC client call metrics
[ws-manager-bridge] Add gRPC client call and cluster registration metrics
[image-builder-mk3] Add image-build metrics

[ArthurSens]

My first walk through the code...

I really liked the decision of using go-grpc-middleware to expose client-side gRPC metrics! When you announced this in the all-hands I thought you were going to implement this in common-go alone 😅.

[csweichel]

I've addressed the comments. @ArthurSens, @aledbf and @geropl please have another look.

[codecov]

Codecov Report

Merging #5726 (34c0801) into main (f5eea47) will decrease coverage by 7.36%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #5726      +/-   ##
==========================================
- Coverage   19.04%   11.67%   -7.37%     
==========================================
  Files           2        6       +4     
  Lines         168     1310    +1142     
==========================================
+ Hits           32      153     +121     
- Misses        134     1137    +1003     
- Partials        2       20      +18     

Flag	Coverage Δ
components-local-app-app-linux	?
components-local-app-app-windows	?
components-registry-facade-app	11.67% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
components/local-app/pkg/auth/pkce.go
components/local-app/pkg/auth/auth.go
...omponents/registry-facade/pkg/registry/registry.go	0.00% <0.00%> (ø)
...omponents/registry-facade/pkg/registry/manifest.go	9.82% <0.00%> (ø)
components/registry-facade/pkg/registry/blob.go	0.00% <0.00%> (ø)
...omponents/registry-facade/pkg/registry/imagecfg.go	0.00% <0.00%> (ø)
components/registry-facade/pkg/registry/metrics.go	0.00% <0.00%> (ø)
...onents/registry-facade/pkg/registry/layersource.go	28.80% <0.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f5eea47...34c0801. Read the comment docs.

[JanKoehnlein]

/lgtm

[roboquat]

LGTM label has been added.

Git tree hash: 6ff787bc06800936c3677f0c554ed15998326d7b

[JanKoehnlein]

/approve cancel

Looking at our logs, there seems to be a severe problem with ws-manager in core-dev workspaces with this change
https://console.cloud.google.com/logs/query;query=%0Aseverity%3DERROR%0Aresource.labels.namespace_name%3D%22staging-cw-fix-5706-5707%22%0Aresource.labels.pod_name%3D%22ws-daemon-xcmp6%22;timeRange=PT1H;summaryFields=labels%252F%2522compute.googleapis.com%252Fresource_name%2522:false:32:beginning;cursorTimestamp=2021-09-17T09:08:54Z?project=gitpod-core-dev

[aledbf]

Looking at our logs, there seems to be a severe problem with ws-manager in core-dev workspaces with this change

The error you see in the logs was fixed here #5696, i.e., is not related to this PR

[JanKoehnlein]

Thanks for the clarification @aledbf. I just wanted to make sure we don't commit anything that sets our logging on fire

/approve

[aledbf]

/approve

[roboquat]

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by: aledbf, JanKoehnlein

Associated issue: #5706

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [JanKoehnlein,aledbf]
• components/common-go/OWNERS [aledbf]
• components/content-service-api/OWNERS
• components/ee/ws-scheduler/OWNERS [aledbf]
• components/gitpod-protocol/OWNERS [JanKoehnlein]
• components/image-builder-api/OWNERS
• components/registry-facade/OWNERS [aledbf]
• components/server/OWNERS [JanKoehnlein]
• components/ws-manager/OWNERS [aledbf]
• components/ws-manager-api/OWNERS [aledbf]
• components/ws-manager-bridge/OWNERS [JanKoehnlein]
• components/ws-proxy/OWNERS [aledbf]
• dev/gpctl/OWNERS [aledbf]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment