Ability to search multiple repositories for PR query

[maximbaz]

Hi @kminehart,

You mentioned in #50 (comment) that it might be possible to query multiple repositories per PR query, do you know if this is something that is impossible to do, or there is still some chance?

If not, could this perhaps be done on the datasource side?

The reason I'm asking is that we are a huge org with many teams and tons of repositories, and it actually is more interesting to draw some charts on per-team level, i.e. how many PRs are currently active, how many have been closed over time, etc., and even within a team we just want to have an overview of which PRs still require a review or merge, we aren't as interested in per-repo granularity.

Using dashboard variables and repeating a table kinda works, but the dashboard becomes quite large, if we have 20+ repos and only 2 repos have active PRs, and so the entire dashboard consists of empty tables... Instead I would really like to just see one table with all the PRs (where repo is just a column), and also have a single stat with number of open PRs per entire team.

What do you think? 🙂

[jneilliii]

Couldn't a merge transform work in this case with multiple queries possibly? I'm still trying to get any graphing to work with running issue totals, but haven't been able to figure out how.

[kminehart]

@jneilliii Unfortunately that's because the GitHub data that's being parsed isn't a timeseries. It has a time, like closed_at or created_at, but no value.

To get metrics like "total number of commits over time", we would need to know what the total number of issues was at each interval in your time range. While calculating this value (at each interval) is definitely possible, it's looking like for medium-large repositories it's going to very quickly max out your access token's allowance. There's definitely some ways to work around this, like getting the total, then performing the query normally, and then using the difference between the number of results and the total as the starting point.

All that to say that features like this will come in a later version, when we have aggregations and groupings implemented. At the moment the only thing we're displaying is tabular data, which is useful in variables and annotations, but obviously not useful in graphs. It's something I'd love to implement but I'm looking at the best way to do it so I can universally add these kinds of functions.

If you wanted to create a feature request just for tracking purposes you're certainly welcome to!

---

@maximbaz I feel like this should be doable.

The value you put into the repository input gets placed in this query:

(You can try this out at https://developer.github.com/v4/explorer/)

{
  search(query: "is:pr repo:grafana/loki", type: ISSUE) {
    issueCount
  }
}

In this example, we can see that loki has 1504 PRs.

However, change this to

{
  search(query: "is:pr repo:grafana/lok*", type: ISSUE) {
    issueCount
  }
}

and the count reduces to 0. So it obviously doesn't use wildcards.

buuuut change it to

{
  search(query: "is:pr org:grafana", type: ISSUE) {
    issueCount
  }
}

and our response is

{
  "data": {
    "search": {
      "issueCount": 27036
    }
  }
}

The offending LoC is here:

https://github.com/grafana/github-datasource/blob/master/pkg/github/pull_requests.go#L130

With that, it doesn't seem possible, however this could easily be changed to set "org:" if "repo" is not set, or if you can find a way to use a search query (documentation here: https://docs.github.com/en/github/searching-for-information-on-github/searching-issues-and-pull-requests) that includes multiple repositories.

[kminehart]

I just tried a quick workaround, you're going to love this.

Put a space in "organization" and "repository", and in the query field, put org:{your-org-name}.

If it's slow, that's likely because of the number of sequential requests that have to be made for pagination.

---

edit: I take that back. Didn't seem to work as I thought it would. :( This should be an easy fix though, I'll try to get this in as well tonight.

With this change I'll also need to remember to add the repository that the pull request belongs to.

[maximbaz]

Interesting! I'm only not sure how to then filter that down to a given team within the org, if using either team name or prefix on repo name seems to be not supported 🤔

[maximbaz]

The merge transform approach is interesting too and seems it could work for a table or PRs for a given team within an org, but I can't find a way to repeat the query per variable, unless I add the same query manually for every repo that a given team owns - do you know if repeating query is somehow possible?

[kminehart]

Yeah it's very unfortunate. It seems that searching multiple repositories is only supported by using the org: query term. Sometimes GitHub lets you search for things that contain your term, but not this time.

Another thing, including the org term apparently causes the query to completely ignore the repo: term, so my workaround should work. You could put anything in there, and include org: in the query box.

[kminehart]

I don't believe that there is a way to repeat queries in a single panel like there is a way to repeat panels. I think the rationale is that most timeseries offer a way to return multiple metrics per query.

grafana/grafana#5508

I think this is something we could achieve with a multi-value variable.

[u001920]

is there any easy way to display the metrics of all repositories?