Monitoring

[snarfed]

We have a monitoring dashboard, https://console.cloud.google.com/monitoring/dashboards/builder/4f0ac7cc-258d-4058-8208-fafa32088378?project=bridgy-federated , but it's degraded. Time to take another pass at it!

[snarfed]

Also: alerts on task queue length.

[snarfed]

Here's a draft of what I'd like:

Monitoring

app

• requests
• errors
• latency, p50 90 99
• /r, /convert, webfinger requests, by domain
• outbound HTTP requests by domain
• instances

router

• tasks by queue
• task responses, errors
• task latency
• already seen activity ids
• unsupported activity types
  • ...also in protocols like AP that short circuit them
• receive tasks by protocol, source domain
• send tasks by protocol, destination domain
• new users
• user deactivations
• activities for blocked, opted out, limited, blocklisted users
• CPU, memory

atproto-hub

• firehose client:
  • total events
  • our events: from our users, to our users
• firehose server:
  • emitted events
  • connected clients
• CPU, memory

scoreboard

• users
• activities processed
• activities emitted (do we collapse AP inbox deliveries? or separate them?)

Alerts

• app error rate
• app latency
• task queue length, errors, rate drops
• atproto firehose:
  • last connection from Bluesky relay over ~75m ago
  • emitted events rate dropoff
• app instances sustained spike (can we limited to billed instances?)
• router, atproto-hub CPU
• router, atproto-hub serving instances sustained != 1

[qazmlp]

We have a monitoring dashboard, https://console.cloud.google.com/monitoring/dashboards/builder/4f0ac7cc-258d-4058-8208-fafa32088378?project=bridgy-federated , but it's degraded. Time to take another pass at it!

That dashboard is private, by the way.

Not sure if you're planning to have a public status page at all/if the privacy setting there is intentional, but I thought I'd mention it in case it's accidental.

[snarfed]

Hey, yes, not accidental. Not sure if I can make it public, but I'd like to!

[snarfed]

Looks like Google Cloud Monitoring dashboards can't be public 😕

https://cloud.google.com/monitoring/access-control#predefined_roles
https://cloud.google.com/iam/docs/principal-identifiers

Update: filed https://issuetracker.google.com/issues/354101899

[snarfed]

Got a first pass at a new dashboard that I'm reasonably happy with.

[snarfed]

...and a first pass at alerts:

[snarfed]

There are a few still outstanding, but this is pretty much done!