expand stats #39

andyfeller · 2023-04-28T17:07:15Z

Reformatting GraphQL queries for maintainability
Refactored curl for gh api calls, enhancements
Removing unnecessary first:1 for totalCount use
Expand data captured, predictable order

This commit is solely for expanding the compacted GraphQL queries on a single line as this form is unmaintainable for most people. This is ONLY affecting whitespace, which will be confirmed following the commit.

- completely replaces `curl` usage for `gh api` and `gh api graphql` calls in order to avoid further tricks to allow passing multi-line GraphQL queries - removes most of the debugging logic built in as emitting the tokens used is insecure, instead relying up `gh api` debugging via `GH_DEBUG=api` - lowers the defaults of various page size flags given that they are always too large in most cases - API remaining notifications has been enhanced to capture both GraphQL points and REST API calls, the previous messaging was inaccurate and only covered the first case - error handling has been simplified to look at exit codes rather than status codes

unsure how this impacts point usage or performance, but it is absolutely unnecessary when retrieving totalCount field, so let's remove what is unneeded.

These changes add several new data points to the output including: - Number of branches - Number of tags - Number of discussions - Whether wiki is enabled Additionally: - explicit alphabetical ordering of repositories has been enforced - removed redundant documentation on output header - removed unused graphql query fields

gh-repo-stats

andyfeller · 2023-04-28T17:11:05Z

Example of running changes:

$ gh repo-stats -o tinyfists              

######################################################
######################################################
############# GitHub repo list and sizer #############
######################################################
######################################################


------------------------------------------------------
Falling back to GitHub token via 'gh auth token'
Creating file header...
------------------------------------------------------
Getting repositories for org: tinyfists
Rate limits remaining: 5,000 GraphQL points  14,986 REST calls
Analyzing Repo: actions-experiments
Analyzing Repo: archived-template
Analyzing Repo: archived-template-demo
Analyzing Repo: asdfasdf
Analyzing Repo: git-xargs
Analyzing Repo: git-xargs-1
Analyzing Repo: git-xargs-2
Analyzing Repo: git-xargs-3
Analyzing Repo: githubcustomer
Analyzing Repo: intro-to-github-niall
Rate limits remaining: 5,000 GraphQL points  14,986 REST calls
Analyzing Repo: issue-driven-github-admin
Analyzing Repo: issue-form-preview
Analyzing Repo: multi-runner-poc
Analyzing Repo: old-repo-name
Analyzing Repo: old-repo-name-2
Analyzing Repo: pages-demo
Analyzing Repo: publish-packages-to-repo-demo
Analyzing Repo: required-workflows
Analyzing Repo: required-workflows-demo
Analyzing Repo: secret-scanning-tests
Rate limits remaining: 5,000 GraphQL points  14,986 REST calls
Analyzing Repo: services
Analyzing Repo: spring-framework
Analyzing Repo: stacks-experiment
Analyzing Repo: stacks-experiment-01
Analyzing Repo: template-bar
Gathered all repositories for org: tinyfists

######################################################
The script has completed

Results file:[tinyfists-all_repos-202304281302.csv]
######################################################

Resulting output: tinyfists-all_repos-202304281302.csv

Example of debug output:

$ gh repo-stats -o tinyfists -d 

######################################################
######################################################
############# GitHub repo list and sizer #############
######################################################
######################################################


------------------------------------------------------
Falling back to GitHub token via 'gh auth token'
* Request at 2023-04-28 13:09:28.235137 -0400 EDT m=+0.034106872
* Request to https://api.github.com/user
> GET /user HTTP/1.1
> Host: api.github.com
> Accept: 
> Authorization: token ████████████████████
> Content-Type: application/json; charset=utf-8
> Time-Zone: America/New_York
> User-Agent: GitHub CLI 2.25.1

< HTTP/2.0 200 OK
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
< Cache-Control: private, max-age=60, s-maxage=60
< Content-Security-Policy: default-src 'none'
< Content-Type: application/json; charset=utf-8
< Date: Fri, 28 Apr 2023 17:09:28 GMT
< Etag: W/"f38c9d9ee03086d45fdb02a98cf38d306d1cb1e0d444f39f4996f8853effc6dd"
< Last-Modified: Wed, 26 Apr 2023 15:20:07 GMT
< Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
< Server: GitHub.com
< Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
< Vary: Accept, Authorization, Cookie, X-GitHub-OTP
< Vary: Accept-Encoding, Accept, X-Requested-With
< X-Accepted-Oauth-Scopes: 
< X-Content-Type-Options: nosniff
< X-Frame-Options: deny
< X-Github-Api-Version-Selected: 2022-11-28
< X-Github-Media-Type: github.v3; format=json
< X-Github-Request-Id: FAB0:7023:9FFEA:148FA5:644BFDC8
< X-Oauth-Client-Id: 178c6fc778ccc68e1d6a
< X-Oauth-Scopes: admin:enterprise, admin:org, admin:org_hook, gist, project, repo, workflow
< X-Ratelimit-Limit: 15000
< X-Ratelimit-Remaining: 14985
< X-Ratelimit-Reset: 1682702653
< X-Ratelimit-Resource: core
< X-Ratelimit-Used: 15
< X-Xss-Protection: 0

{
  "login": "andyfeller",
  "id": 2089743,
  "node_id": "MDQ6VXNlcjIwODk3NDM=",
  "avatar_url": "https://avatars.githubusercontent.com/u/2089743?v=4",
  "gravatar_id": "",
  "url": "https://api.github.com/users/andyfeller",
  "html_url": "https://github.com/andyfeller",
  "followers_url": "https://api.github.com/users/andyfeller/followers",
  "following_url": "https://api.github.com/users/andyfeller/following{/other_user}",
  "gists_url": "https://api.github.com/users/andyfeller/gists{/gist_id}",
  "starred_url": "https://api.github.com/users/andyfeller/starred{/owner}{/repo}",
  "subscriptions_url": "https://api.github.com/users/andyfeller/subscriptions",
  "organizations_url": "https://api.github.com/users/andyfeller/orgs",
  "repos_url": "https://api.github.com/users/andyfeller/repos",
  "events_url": "https://api.github.com/users/andyfeller/events{/privacy}",
  "received_events_url": "https://api.github.com/users/andyfeller/received_events",
  "type": "User",
  "site_admin": true,
  "name": "Andy Feller",
  "company": "GitHub",
  "blog": "",
  "location": null,
  "email": "andyfeller@github.com",
  "hireable": null,
  "bio": "DevOps Engineer with GitHub's Expert Services, I'm a self proclaimed disciple of J. Wellington Wimpy and Shawn Spencer, looking for delicious flavor.",
  "twitter_username": null,
  "public_repos": 43,
  "public_gists": 10,
  "followers": 32,
  "following": 9,
  "created_at": "2012-08-03T13:23:51Z",
  "updated_at": "2023-04-26T15:20:07Z"
}

* Request took 183.287732ms
Successfully validated access to GHE Instance...
Version: cloud
Creating file header...
* Request at 2023-04-28 13:09:28.53851 -0400 EDT m=+0.037259876
* Request to https://api.github.com/orgs/tinyfists/memberships/andyfeller
> GET /orgs/tinyfists/memberships/andyfeller HTTP/1.1
> Host: api.github.com
> Accept: 
> Authorization: token ████████████████████
> Content-Type: application/json; charset=utf-8
> Time-Zone: America/New_York
> User-Agent: GitHub CLI 2.25.1

< HTTP/2.0 200 OK
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
< Cache-Control: private, max-age=60, s-maxage=60
< Content-Security-Policy: default-src 'none'
< Content-Type: application/json; charset=utf-8
< Date: Fri, 28 Apr 2023 17:09:28 GMT
< Etag: W/"89b6287d7aca23ad0f59a8d77f86898966d9c117d45a6af6b0011975b6d54e2b"
< Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
< Server: GitHub.com
< Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
< Vary: Accept, Authorization, Cookie, X-GitHub-OTP
< Vary: Accept-Encoding, Accept, X-Requested-With
< X-Accepted-Oauth-Scopes: admin:org, read:org, repo, user, write:org
< X-Content-Type-Options: nosniff
< X-Frame-Options: deny
< X-Github-Api-Version-Selected: 2022-11-28
< X-Github-Media-Type: github.v3; format=json
< X-Github-Request-Id: FAB1:1B38:AB9A2:1602E2:644BFDC8
< X-Oauth-Client-Id: 178c6fc778ccc68e1d6a
< X-Oauth-Scopes: admin:enterprise, admin:org, admin:org_hook, gist, project, repo, workflow
< X-Ratelimit-Limit: 15000
< X-Ratelimit-Remaining: 14984
< X-Ratelimit-Reset: 1682702653
< X-Ratelimit-Resource: core
< X-Ratelimit-Used: 16
< X-Xss-Protection: 0

{
  "url": "https://api.github.com/orgs/tinyfists/memberships/andyfeller",
  "state": "active",
  "role": "admin",
  "organization_url": "https://api.github.com/orgs/tinyfists",
  "user": {
    "login": "andyfeller",
    "id": 2089743,
    "node_id": "MDQ6VXNlcjIwODk3NDM=",
    "avatar_url": "https://avatars.githubusercontent.com/u/2089743?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/andyfeller",
    "html_url": "https://github.com/andyfeller",
    "followers_url": "https://api.github.com/users/andyfeller/followers",
    "following_url": "https://api.github.com/users/andyfeller/following{/other_user}",
    "gists_url": "https://api.github.com/users/andyfeller/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/andyfeller/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/andyfeller/subscriptions",
    "organizations_url": "https://api.github.com/users/andyfeller/orgs",
    "repos_url": "https://api.github.com/users/andyfeller/repos",
    "events_url": "https://api.github.com/users/andyfeller/events{/privacy}",
    "received_events_url": "https://api.github.com/users/andyfeller/received_events",
    "type": "User",
    "site_admin": true
  },
  "organization": {
    "login": "tinyfists",
    "id": 90800543,
    "node_id": "MDEyOk9yZ2FuaXphdGlvbjkwODAwNTQz",
    "url": "https://api.github.com/orgs/tinyfists",
    "repos_url": "https://api.github.com/orgs/tinyfists/repos",
    "events_url": "https://api.github.com/orgs/tinyfists/events",
    "hooks_url": "https://api.github.com/orgs/tinyfists/hooks",
    "issues_url": "https://api.github.com/orgs/tinyfists/issues",
    "members_url": "https://api.github.com/orgs/tinyfists/members{/member}",
    "public_members_url": "https://api.github.com/orgs/tinyfists/public_members{/member}",
    "avatar_url": "https://avatars.githubusercontent.com/u/90800543?v=4",
    "description": ""
  }
}

* Request took 166.043497ms
You are an owner. Getting Repo Stats
------------------------------------------------------
Getting repositories for org: tinyfists
* Request at 2023-04-28 13:09:28.789282 -0400 EDT m=+0.037580291
* Request to https://api.github.com/rate_limit
> GET /rate_limit HTTP/1.1
> Host: api.github.com
> Accept: 
> Authorization: token ████████████████████
> Content-Type: application/json; charset=utf-8
> Time-Zone: America/New_York
> User-Agent: GitHub CLI 2.25.1

< HTTP/2.0 200 OK
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
< Cache-Control: no-cache
< Content-Security-Policy: default-src 'none'
< Content-Type: application/json; charset=utf-8
< Date: Fri, 28 Apr 2023 17:09:29 GMT
< Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
< Server: GitHub.com
< Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
< Vary: Accept-Encoding, Accept, X-Requested-With
< X-Accepted-Oauth-Scopes: 
< X-Content-Type-Options: nosniff
< X-Frame-Options: deny
< X-Github-Api-Version-Selected: 2022-11-28
< X-Github-Media-Type: github.v3; format=json
< X-Github-Request-Id: FAB2:0A48:BED20:1863A8:644BFDC8
< X-Oauth-Client-Id: 178c6fc778ccc68e1d6a
< X-Oauth-Scopes: admin:enterprise, admin:org, admin:org_hook, gist, project, repo, workflow
< X-Ratelimit-Limit: 15000
< X-Ratelimit-Remaining: 14984
< X-Ratelimit-Reset: 1682702653
< X-Ratelimit-Resource: core
< X-Ratelimit-Used: 16
< X-Xss-Protection: 0

{
  "resources": {
    "core": {
      "limit": 15000,
      "used": 16,
      "remaining": 14984,
      "reset": 1682702653
    },
    "search": {
      "limit": 30,
      "used": 0,
      "remaining": 30,
      "reset": 1682701829
    },
    "graphql": {
      "limit": 5000,
      "used": 37,
      "remaining": 5000,
      "reset": 1682702452
    },
    "integration_manifest": {
      "limit": 5000,
      "used": 0,
      "remaining": 5000,
      "reset": 1682705369
    },
    "source_import": {
      "limit": 100,
      "used": 0,
      "remaining": 100,
      "reset": 1682701829
    },
    "code_scanning_upload": {
      "limit": 1000,
      "used": 0,
      "remaining": 1000,
      "reset": 1682705369
    },
    "actions_runner_registration": {
      "limit": 10000,
      "used": 0,
      "remaining": 10000,
      "reset": 1682705369
    },
    "scim": {
      "limit": 15000,
      "used": 0,
      "remaining": 15000,
      "reset": 1682705369
    },
    "dependency_snapshots": {
      "limit": 100,
      "used": 0,
      "remaining": 100,
      "reset": 1682701829
    }
  },
  "rate": {
    "limit": 15000,
    "used": 16,
    "remaining": 14984,
    "reset": 1682702653
  }
}

* Request took 146.329116ms
Rate limits remaining: 5,000 GraphQL points  14,984 REST calls
Getting repos
* Request at 2023-04-28 13:09:29.077208 -0400 EDT m=+0.034324082
* Request to https://api.github.com/graphql
> POST /graphql HTTP/1.1
> Host: api.github.com
> Accept: 
> Authorization: token ████████████████████
> Content-Length: 2325
> Content-Type: application/json; charset=utf-8
> Time-Zone: America/New_York
> User-Agent: GitHub CLI 2.25.1

GraphQL query:
query($login: String!, $pageSize: Int!, $endCursor: String) {
    organization(login: $login) {
      repositories(first: $pageSize, after: $endCursor, orderBy: {field: NAME, direction: ASC}) {
        totalDiskUsage
        pageInfo {
          endCursor
          hasNextPage
        }
        nodes {
          branches: refs(refPrefix:"refs/heads/") {
            totalCount
          }
          branchProtectionRules {
            totalCount
          }
          commitComments {
            totalCount
          }
          collaborators {
            totalCount
          }
          diskUsage
          discussions {
            totalCount
          }
          hasWikiEnabled
          isEmpty
          isFork
          issues(first: $pageSize) {
            totalCount
            pageInfo {
              endCursor
              hasNextPage
            }
            nodes {
              timeline {
                totalCount
              }
              comments {
                totalCount
              }
            }
          }
          milestones {
            totalCount
          }
          name
          owner {
            login
          }
          projects {
            totalCount
          }
          pullRequests(first: $pageSize) {
            totalCount
            pageInfo {
              endCursor
              hasNextPage
            }
            nodes {
              comments {
                totalCount
              }
              commits {
                totalCount
              }
              number
              reviews(first: $pageSize) {
                totalCount
                pageInfo {
                  endCursor
                  hasNextPage
                }
                nodes {
                  comments {
                    totalCount
                  }
                }
              }
              timeline {
                totalCount
              }
            }
          }
          pushedAt
          releases {
            totalCount
          }
          tags: refs(refPrefix: "refs/tags/") {
            totalCount
          }
          updatedAt
        }
      }
    }
  }
GraphQL variables: {"login":"tinyfists","pageSize":10}

ssulei7

Let's address the checks and we can proceed. @andyfeller

gh-repo-stats

andyfeller · 2023-04-28T18:02:38Z

Let's address the checks and we can proceed. @andyfeller

@ssulei7 : we're on the same page 👍 I'm also working on updating the documentation and potentially changing some of the flags because this is going to now rely upon gh api, so the flags for tokens and url should behave differently

ssulei7

LGTM. Thank you for the contribution!

With previous changes involving refactoring the underlying calls to use gh api, we had to rework the -u,--url flags because they didn't make sense. These changes ensure the user knows that standard GH CLI environment works and does some different approaches to get URLs or detect GHES vs GHEC. Additionally, this change refreshes the README to be a little more concise and includes more explanation of what is being outputted.

robandpdx · 2023-05-01T15:47:54Z

gh-repo-stats

-
-    DebugJQ "${USER_DATA}"
+    USER_DATA=$(gh api /user)
+    ERROR_CODE=$?


Perhaps a better name for this variable is EXIT_CODE since it's only an error code if non-zero.

I'm not going to change these style uses given that they are preferential and pre-existing.

andyfeller added 4 commits April 27, 2023 16:06

Reformatting GraphQL queries for maintainability

1fdf4fa

This commit is solely for expanding the compacted GraphQL queries on a single line as this form is unmaintainable for most people. This is ONLY affecting whitespace, which will be confirmed following the commit.

Removing unnecessary first:1 for totalCount use

3d8581f

unsure how this impacts point usage or performance, but it is absolutely unnecessary when retrieving totalCount field, so let's remove what is unneeded.

andyfeller requested a review from a team as a code owner April 28, 2023 17:07

github-advanced-security bot found potential problems Apr 28, 2023

View reviewed changes

andyfeller self-assigned this Apr 28, 2023

ssulei7 reviewed Apr 28, 2023

View reviewed changes

Address shellcheck return code checks

bd2c1c4

github-advanced-security bot found potential problems Apr 28, 2023

View reviewed changes

gh-repo-stats Fixed Show fixed Hide fixed

gh-repo-stats Fixed Show fixed Hide fixed

Address globbing / word splitting check

4b8a389

ssulei7 approved these changes Apr 28, 2023

View reviewed changes

JohnnyLongie approved these changes Apr 29, 2023

View reviewed changes

andyfeller added 2 commits May 1, 2023 11:40

Removing markdown lint "errors"

3b1beba

Disabling shellcheck "errors" on printf exit codes

299ae09

robandpdx reviewed May 1, 2023

View reviewed changes

andyfeller added 2 commits May 1, 2023 11:50

Telling shellcheck to live with style "errors"

9456059

I'm not going to change these style uses given that they are preferential and pre-existing.

Last time having existing style "errors" accepted

5db2431

andyfeller merged commit 3095c9b into main May 1, 2023

andyfeller deleted the expand-stats branch May 1, 2023 15:59

This was referenced May 2, 2023

Fix breaking GraphQL query due to invalid field #43

Closed

Add support for secure token storage #36

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

expand stats #39

expand stats #39

andyfeller commented Apr 28, 2023

andyfeller commented Apr 28, 2023 •

edited

Loading

ssulei7 left a comment

andyfeller commented Apr 28, 2023

ssulei7 left a comment

robandpdx May 1, 2023

expand stats #39

expand stats #39

Conversation

andyfeller commented Apr 28, 2023

andyfeller commented Apr 28, 2023 • edited Loading

ssulei7 left a comment

Choose a reason for hiding this comment

andyfeller commented Apr 28, 2023

ssulei7 left a comment

Choose a reason for hiding this comment

robandpdx May 1, 2023

Choose a reason for hiding this comment

andyfeller commented Apr 28, 2023 •

edited

Loading