Publish more prometheus stats throughout system

[allada]

Now that Prometheus is added and the API is established, we need to spread the usage around the system.

• We can't consider our Prometheus publishing stable until Custom Collector with sub_registry_with_prefix outputs name multiple times prometheus/client_rust#155 is fixed.
• GrpcServices
  • Implement for AcServer
  • Implement for BytestreamServer
  • Implement for CapabilitiesServer
  • Implement for CasServer
  • Implement for ExecutionServer
  • Implement for WorkerApiServer
• Schedulers
  • Implement for PlatformPropertiesManager
  • Implement for CacheLookupScheduler
  • Implement for GrpcScheduler
  • Add metrics to SimpleScheduler and Worker #216 Implement for SimpleScheduler
  • Add metrics to SimpleScheduler and Worker #216 Implement for scheduler::Worker
• Stores
  • Implement for CompressionStore
  • Implement for DedupStore
  • Implement fast slow metrics. #263 Implement for FastSlowStore
  • Add prometheus publishing and hook up FilesystemStore #202 Implement for FilesystemStore
  • Implement for GrpcStore
  • Add prometheus stats to MemoryStore #207 Implement for MemoryStore
  • Implement for S3Store
  • Implement for SizePartitioningStore
  • Add prometheus stats for VerifyStore #208 Implement for VerifyStore
• Workers
  • Add metrics to LocalWorker and RunningActionsManager #213 Implement for LocalWorker
  • Add metrics to LocalWorker and RunningActionsManager #213 Implement for RunningActionsManager
  • Implement for WorkerApiClientWrapper
• Global State
  • Prometheus now publishes connected clients #230 How many active GRPC connections there are
  • Total GRPC connections since server started
  • Time server started (timestamp)
  • Global config setting used
  • Maybe internal tokio metrics. This might require unstable tokio stuff 😦

[chrisstaite]

That's quite the project. I tried to grab some stats today, after enabling the endpoint I just got #EOF but it's probably because it was on the scheduler which doesn't have anything logging any metrics yet...

I did want to try and get some metrics off the scheduler based on the user, but Goma doesn't appear to pass the authenticated user through to the RBE. I need to look into that.

[allada]

I'm about to send the behemoth PR to enable metrics for workers. I suggest waiting a little to see how that PR looks before starting to implement it.

As for enabling it for a specific user... this is going to be tricky, we'd have to pass down a lot of context to enable it for a specific strand. Right now we always gather metrics regardless of what user it is.

If this feature is really requested, here's a thought...

• Disable metrics globally (would be trivial, since everything is currently wrapped in prometheus_utils.rs), set a global thread_local flag that prevents metric collection with env flag or disable it at runtime with a service endpoint.
• Add a special endpoint that you can add specific IP addresses or endpoints to trigger it into a "debug" mode.
• When users connect check to see if they are in the list of "debug" users, if they are create a new thread, setup a new tokio runtime for that thread manually and set the thread_local flag to capture metrics.

In theory we'd only capture metrics for debug endpoints in that case with near zero runtime cost.

[allada]

I did want to try and get some metrics off the scheduler based on the user

I started some of the work here. To get it fully to what you want, we need a way to reset metrics (also should be trivial, since we already have a visitor), an endpoint to trigger the reset and to create a new thread and runtime for specific attached clients.
#215

[blakehatch]

Taking a crack at "Total GRPC connections since server started":

• Tracking how many clients have connected with a lazily initialized counter for thread safety.
• Publishing just total number of client connections, not sure if other metrics are wanted like with Prometheus now publishes connected clients #230, don't want it to be bloated since it's never removing any clients unlike when only active connections are recorded