# Improve Metric Accuracy for Issues, PRs, and Review Distribution#26131
# Improve Metric Accuracy for Issues, PRs, and Review Distribution#26131gemini-cli-robot wants to merge 3 commits into
Conversation
## 1. What the change is This PR refactors the `open_issues.ts` and `open_prs.ts` metric scripts to use the GitHub GraphQL API's `totalCount` field instead of relying on the CLI's `gh issue list` command with a hardcoded limit. It also updates `review_distribution.ts` to include `COLLABORATOR` in the maintainer association check. ## 2. Why it is recommended The current implementation of `open_issues.ts` and `open_prs.ts` used `--limit 1000`, which caused metrics to be capped at 1000 even when the actual backlog was much larger (~2400 issues). This provided a misleading view of repository health and the true scale of the backlog. Using GraphQL `totalCount` ensures accurate counts regardless of list size. Additionally, `review_distribution.ts` was inconsistently excluding `COLLABORATOR` associations, which could lead to an inaccurate representation of review work distribution if many maintainers are designated as Collaborators. This led to a `review_distribution_variance` of 0 in recent runs. ## 3. Which metric or aspect of productivity is expected to be improved - **open_issues**: Will now reflect the true total count (expected to jump from 1000 to ~2400). - **open_prs**: Will reflect the true total count of open pull requests. - **review_distribution_variance**: Will more accurately reflect how review work is shared among all maintainers (including collaborators). ## 4. By how much the metric is expected to improve The `open_issues` metric is expected to increase by approximately **140%** (from 1000 to ~2400) once accurate data is collected. The `review_distribution_variance` is expected to become non-zero, providing a real baseline for monitoring reviewer workload balance.
|
Size Change: -4 B (0%) Total Size: 33.9 MB
ℹ️ View Unchanged
|
- BerriAI/litellm#26766 Azure AD token auth + api-key header fix (merge-after-nits) - BerriAI/litellm#26763 Excepton -> Exception log typo fix (merge-as-is) - google-gemini/gemini-cli#26131 metric accuracy via GraphQL totalCount (merge-after-nits)
PR Review: Recommendations & CalloutsOverall, this is a very solid improvement. Here are a few recommendations and callouts to consider: 1. Behavioral Change in
|
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request improves the accuracy and reliability of repository health metrics. By transitioning from CLI-limited list commands to direct GraphQL totalCount queries, the system now correctly accounts for all open issues and pull requests regardless of volume. Additionally, the inclusion of collaborators in review distribution metrics provides a more complete picture of team contribution and workload balance. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request refactors several metrics scripts to utilize the GitHub GraphQL API more securely by passing parameters through environment variables and handling API errors explicitly. It also expands the reviewer definition to include collaborators. Feedback focuses on standardizing the output format across scripts using the MetricOutput interface for consistency and timestamps, and ensuring normalized login names are used in review_distribution.ts to prevent duplicate entries.
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| import { GITHUB_OWNER, GITHUB_REPO } from '../types.js'; |
There was a problem hiding this comment.
To maintain consistency with other metric scripts in this repository, this script should use the MetricOutput type for its output. This ensures that all metrics follow the same schema, including the required timestamp.
| import { GITHUB_OWNER, GITHUB_REPO } from '../types.js'; | |
| import { GITHUB_OWNER, GITHUB_REPO, type MetricOutput } from '../types.js'; |
| throw new Error(response.errors.map((e: { message: string }) => e.message).join(', ')); | ||
| } | ||
| const count = response.data.repository.issues.totalCount; | ||
| console.log(`open_issues,${count}`); |
There was a problem hiding this comment.
The output format should be updated to JSON lines using the MetricOutput interface to match the standard used by other scripts in the metrics suite. This improves system consistency and ensures that a timestamp is included with the metric value.
const timestamp = new Date().toISOString();
process.stdout.write(
JSON.stringify(<MetricOutput>{
metric: 'open_issues',
value: count,
timestamp,
}) + '\n',
);| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| import { GITHUB_OWNER, GITHUB_REPO } from '../types.js'; |
There was a problem hiding this comment.
To maintain consistency with other metric scripts in this repository, this script should use the MetricOutput type for its output. This ensures that all metrics follow the same schema, including the required timestamp.
| import { GITHUB_OWNER, GITHUB_REPO } from '../types.js'; | |
| import { GITHUB_OWNER, GITHUB_REPO, type MetricOutput } from '../types.js'; |
| throw new Error(response.errors.map((e: { message: string }) => e.message).join(', ')); | ||
| } | ||
| const count = response.data.repository.pullRequests.totalCount; | ||
| console.log(`open_prs,${count}`); |
There was a problem hiding this comment.
The output format should be updated to JSON lines using the MetricOutput interface to match the standard used by other scripts in the metrics suite. This improves system consistency and ensures that a timestamp is included with the metric value.
const timestamp = new Date().toISOString();
process.stdout.write(
JSON.stringify(<MetricOutput>{
metric: 'open_prs',
value: count,
timestamp,
}) + '\n',
);| for (const review of pr.reviews.nodes) { | ||
| if ( | ||
| ['MEMBER', 'OWNER'].includes(review.authorAssociation) && | ||
| ['MEMBER', 'OWNER', 'COLLABORATOR'].includes(review.authorAssociation) && |
There was a problem hiding this comment.
Please use the lowercased login variable when adding to the reviewersOnPR set to ensure consistent casing and prevent duplicate entries in reviewCounts, which would skew the variance calculation.
gemini-cli
Bot
added
the
status/need-issue
|
@gemini-cli-robot please fix the failing checks in this PR. |
4 participants
Improve Metric Accuracy for Issues, PRs, and Review Distribution
1. What the change is
This PR refactors the
open_issues.tsandopen_prs.tsmetric scripts to use the GitHub GraphQL API'stotalCountfield instead of relying on the CLI'sgh issue listcommand with a hardcoded limit. It also updatesreview_distribution.tsto includeCOLLABORATORin the maintainer association check.2. Why it is recommended
The current implementation of
open_issues.tsandopen_prs.tsused--limit 1000, which caused metrics to be capped at 1000 even when the actual backlog was much larger (~2400 issues). This provided a misleading view of repository health and the true scale of the backlog. Using GraphQLtotalCountensures accurate counts regardless of list size.Additionally,
review_distribution.tswas inconsistently excludingCOLLABORATORassociations, which could lead to an inaccurate representation of review work distribution if many maintainers are designated as Collaborators. This led to areview_distribution_varianceof 0 in recent runs.3. Which metric or aspect of productivity is expected to be improved
4. By how much the metric is expected to improve
The
open_issuesmetric is expected to increase by approximately 140% (from 1000 to ~2400) once accurate data is collected. Thereview_distribution_varianceis expected to become non-zero, providing a real baseline for monitoring reviewer workload balance.