Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: collect key metrics from the build infrastructure #47325

25 tasks
cagedmantis opened this issue Jul 21, 2021 · 0 comments
25 tasks

x/build: collect key metrics from the build infrastructure #47325

cagedmantis opened this issue Jul 21, 2021 · 0 comments


Copy link

@cagedmantis cagedmantis commented Jul 21, 2021

This is a tracking issue for the collection of key operational metrics from the build infrastructure. These metrics are being collected for the following reasons:

  • Increasing speed of root cause analysis when an issue arrises.
  • Understanding how changes correlate to performance changes.
  • Facilitating key areas for possible optimizations.
  • Facilitating monitoring and alerting on metrics.

The task list below will be appended to once a detailed list of key metrics are identified.

  • Collect Metrics

  • Create Dashboards

  • GCP Aggregate Service API metrics

  • AWS Aggregate Service API

  • GitHub Aggregate Service API

  • Gerrit Aggregate Service API

  • TLS certificate lifetime

  • General OS/Application/Container specific metrics


  • Amount of time waiting for VM quota
  • Buildlet creation latency by stage and type
  • Total buildlet creation latency
  • VM instance creation failures
  • Instance creation queue depth
  • Instance creation queue latency
  • Active Trybot count, latency, failures by type
  • Buildlet count by pool
  • Pending build count by type
  • Pending build latency by type
  • Uptime
  • Build rate
  • General API instrumentation (like ochttp)


  • Sessions created
  • Sessions destroyed
  • Session duration
  • Command usage (SSH, put, etc.)


@cagedmantis cagedmantis added this to the Unreleased milestone Jul 21, 2021
@cagedmantis cagedmantis self-assigned this Jul 21, 2021
@cagedmantis cagedmantis added this to Planned in Go Release Team Jul 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant