Support getting both JSON and template output

[rwalkerands]

Describe the feature or problem you’d like to solve

I'd like commands such as gh pr list to be able to give me both JSON output and to be able to apply a template.

The comment at #3012 (comment) notes as an aside:

Since filter and template are mutually exclusive ...

I'm using gh pr list in a (Bash) script that both generates pretty output for the user, and also does some other processing of the PRs.

I'd like to avoid having to run gh pr list twice.

What doesn't really work for me: add {{ . }} in the template. This gives me output in Go syntax, which I can't otherwise use.

Proposed solution

I have no particular preference for a solution. Obvious thought: add an option to output the JSON to a file.

Or, a completely different approach: add a new command that takes JSON input (from stdin and/or a file) and applies a template.

[andyfeller]

@rwalkerands : thank you for opening up this issue! 🤗

How would you expect to separate out multiple output streams if --template and --json both go to standard output?

I don't really see a clean way to achieve what you're hoping for unfortunately. That said, I can talk about how I typically work with JSON for scripting without duplicate gh calls.

Like mona-actions/gh-repo-stats extension, you can dump the JSON to a file on disk and work with it as base-64 strings allowing you to loop over entries with normal shell scripting:

  REPOS=$(echo "${PARSE_DATA}" | jq -r '.data.organization.repositories.nodes')
  for REPO_DATA in $(echo -n "${REPOS}" | jq -r '.[] | @base64'); do
    _jq() {
    echo -n "${REPO_DATA}" | base64 --decode | jq -r "${1}"
    }

    OWNER=$(_jq '.owner.login' | tr '[:upper:]' '[:lower:]')
    REPO_NAME=$(_jq '.name' | tr '[:upper:]' '[:lower:]')

This way, you can pull and even paginate data while limiting the number of calls you're trying to make.

[lmmx]

I followed this issue because I had the same feature request, and am surprised this is considered too difficult to implement.

The template feature is just wrapping templating functionality in Go to my understanding, and the feature request is that the input to this template can be filtered using jq.

At least, in my case that was all I wanted, and was surprised to find it incompatible (so I had to pipe to jq manually, despite this being bundled in gh, making my script less portable).

I don't need to handle "multiple output streams" and I don't want to rely on writing to disk, this seems pretty simple: just pipe the output of the jq filter into the template format.

[rwalkerands]

How would you expect to separate out multiple output streams if --template and --json both go to standard output?

I wouldn't, which is why I offered two proposed solutions which are different from that.

To repeat them:

1. Add an option to output the JSON to a file (i.e., with the filename specified as the parameter of the option).
2. Or, a completely different approach: add a new command that takes JSON input (from stdin and/or a file) and applies a template.

To expand on (2): add a new top-level command, say gh template --template <template>, that takes JSON input either from standard input or from a file specified as another command-line option (or, I guess the JSON could also be provided as a command-line argument), and applies the template provided as the command-line argument. (By "JSON input" I mean, of course, the output of a previous gh ... -jq ... command.)

I happen to like the implementation of templates provided by gh. Here's part of my script that produces a report on open PRs for a repo:

fields=assignees,author,baseRefName,body,createdAt,headRefName,headRepositoryOwner,id,labels,number,reviewDecision,reviews,statusCheckRollup,title,updatedAt,url
#PR=$(gh pr list -R "${REPO}" --json "${fields}"  --jq '.[]')

gh pr list -R "${REPO}" --json "${fields}" \
--template \
'====================================================================================================
{{range .}}
{{printf "#%v" .number | color "green"}} {{.title}}
{{.url}}
Author: {{.author.name}} (@{{.author.login}} https://github.com/{{.author.login}})
Created at: {{timeago .createdAt}}
Updated at: {{timeago .updatedAt}}
{{if .reviewDecision}}Review decision: {{.reviewDecision}}{{end -}}

This is a request to merge: {{.headRepositoryOwner.login}}/{{.headRefName}} -> {{.baseRefName}}

Description:

{{.body}}

{{if .labels}}{{tablerow "LABEL" "DESCRIPTION"}}{{range .labels}}{{tablerow .name .description}}{{end}}{{tablerender}}
{{end -}}
{{if .assignees}}{{tablerow "ASSIGNEE" "NAME"}}{{range .assignees}}{{tablerow .login .name}}{{end}}{{tablerender}}
{{end -}}
{{if .reviews}}{{tablerow "REVIEWER" "STATE" "COMMENT"}}{{range .reviews}}{{tablerow .author.login .state .body}}{{end}}{{tablerender}}
{{end -}}
{{if .statusCheckRollup}}{{tablerow "CHECK" "START_TIME" "STATE" "URL"}}{{range .statusCheckRollup}}{{tablerow .context .startedAt .state .targetUrl}}{{end}}{{tablerender}}
{{end -}}
====================================================================================================
{{end}}

'

Yes, I probably could implement this template using Bash/jq. But I'd rather not have to.

[andyfeller]

@rwalkerands : Thanks for clarifying your thoughts further as initially it seemed you might be asking for something that was impossible: having JSON and template output both streaming through standard out. Seeing that you and @lmmx were considering multi-phase processing via JSON + JQ then template processing, I believe this issue has come up previously as I recall asking a similar question.

At the time, the feedback I received was:

--jq and --template were both designed to convert JSON into something more convenient for terminals. I guess you could in theory use --jq to massage data passed to --template (if we allowed the combination of flags) but I think allowing that would be fiddly and not lead to great user experiences. it would be better to add helpers to --template to make massaging data there more easier

Personally, I'm open to the idea still though the --jq support isn't 1:1 to jq; see https://github.com/itchyny/gojq#difference-to-jq for more information. Understanding how this will play out with @heaths changes in #1268 for being able to provide a single merged JSON response from the GitHub API will be interesting.

Or, a completely different approach: add a new command that takes JSON input (from stdin and/or a file) and applies a template.

This is certainly outside of the scope of what the GitHub CLI team and maintainers can support. It is something I myself have looked into for similar reasons however I didn't truly find anything that allowed users to arbitrary define Go templates and pass in JSON via file or standard input.

Add an option to output the JSON to a file (i.e., with the filename specified as the parameter of the option).

Currently, this wouldn't work or work well given commands that require a --paginate flag or have to iterate on multiple pages of data from the API.

[heaths]

Why not just have gh emit JSON, tee that, and send stdout through jq or some Go templating app. Off hand I'm not aware of one, but you could use a similar design to my extension https://github.com/heaths/gh-merge-json (gh ext install heaths/gh-merge-json) that is an alternative to #1268 until I was allowed to complete that work (soon).

E.g.,

gh api ... | tee output.json | jq ...

Having a CLI do "everything" seems not only a maintenance issue, but also limiting: it's not always going to work the way people want, and as a consequence devs keep having to add knobs which increases maintenance costs for complexity. Keeping it flexible might be better.

If you want something easy to use in scripts, consider at least keeping it in the ecosystem as an extension. You could easily have an extension both tee and format JSON using Go templates and install that if needed with the --force switch I added back in #7173 for that very purpose that I use it for in many of our scripts:

#!/usr/bin/env bash

# this will install or uprade the extension, or no-op if already installed
gh ext install --force heaths/gh-merge-json

# using `gh ext exec` is good for scripts and helps avoid future possible ambiguity, which happened to me a while back
gh api ... | gh ext exec merge-json

Just my $0.02.

[heaths]

I added support to merge-json in heaths/gh-merge-json#6 and pushed a new release. Seems appropriate, given that the merge-json extension requires JSON input anyway...for which I also fixed an input validation bug prior to this PR.

Example usage:

gh ext install --force heaths/gh-merge-json > /dev/null # or just $null in pwsh
gh api graphql --paginate -f query='
    query($endCursor: String) {
      viewer {
        repositories(first: 100, after: $endCursor) {
          nodes { nameWithOwner }
          pageInfo {
            hasNextPage
            endCursor
          }
        }
      }
    }
  ' | gh merge-json --template '{{range .data.viewer.repositories.nodes}}{{tablerow .nameWithOwner}}{{end}}' --tee repos.json

[rwalkerands]

Why not just have gh emit JSON, tee that, and send stdout through jq ...

Because using jq to implement the sort of template(s) I want to have (see #8415 (comment)) appears to be quite cumbersome ... if it is even possible. (E.g., how to do {{timeago .createdAt}}? An inability to do that wouldn't be a showstopper, but it's nice to be able to mimic what's shown in the GitHub web interface.)

or some Go templating app. Off hand I'm not aware of one,

You answered your own question. I already looked, but didn't find. Very happy to be pointed at a small standalone-app that does JSON through a Go-like template to text.

Indeed, thank you for the pointer to https://github.com/heaths/gh-merge-json ; I have now tried it and confirmed that it works for me.

[andyfeller]

Thanks for sharing your insights, @heaths ❤️ There is definitely a fine line of how GitHub CLI aligns with other tools in the Unix design principle we're trying to exercise being an extremely small team.

All of that to say, @rwalkerands , I hope you have some options for short/medium term workarounds as discussed above while I bring this up for discussion with the other maintainers on ramifications of refactoring the code involved:

cli/pkg/cmdutil/json_flags.go

Lines 185 to 220 in 57f6787

	// Write serializes data into JSON output written to w. If the object passed as data implements exportable,
	// or if data is a map or slice of exportable object, ExportData() will be called on each object to obtain
	// raw data for serialization.
	func (e *jsonExporter) Write(ios *iostreams.IOStreams, data interface{}) error {
	buf := bytes.Buffer{}
	encoder := json.NewEncoder(&buf)
	encoder.SetEscapeHTML(false)
	if err := encoder.Encode(e.exportData(reflect.ValueOf(data))); err != nil {
	return err
	}
	w := ios.Out
	if e.filter != "" {
	indent := ""
	if ios.IsStdoutTTY() {
	indent = " "
	}
	if err := jq.EvaluateFormatted(&buf, w, e.filter, indent, ios.ColorEnabled()); err != nil {
	return err
	}
	} else if e.template != "" {
	t := template.New(w, ios.TerminalWidth(), ios.ColorEnabled())
	if err := t.Parse(e.template); err != nil {
	return err
	}
	if err := t.Execute(&buf); err != nil {
	return err
	}
	return t.Flush()
	} else if ios.ColorEnabled() {
	return jsoncolor.Write(w, &buf, " ")
	}
	_, err := io.Copy(w, &buf)
	return err
	}

[rwalkerands]

All of that to say, @rwalkerands, I hope you have some options for short/medium term workarounds as discussed above while I bring this up for discussion with the other maintainers ...

Indeed, I am now using gh merge-json --template and it does what I need.

[heaths]

[shameless plug] @rwalkerands if you like it, consider starring heaths/gh-merge-json to to help discovery on https://github.com/topics/gh-extension 😉 If not, that's cool too. Won't change support or feature ideas unlike some repos in the news recently. Just helps others find it easier in case it might help them.

On that note: @andyfeller it'd be cool if GitHub defaulted a search from a topic to include that topic when you hit / or click in the search box e.g. topic:gh-extension merge json. I use search a lot and didn't even realize topic: was a valid prefix. Just guessed it might work and tried it with pleasant surprise!

[rwalkerands]

consider starring heaths/gh-merge-json to

Done (and was happy to do so).

[andyfeller]

On that note: @andyfeller it'd be cool if GitHub defaulted a search from a topic to include that topic when you hit / or click in the search box e.g. topic:gh-extension merge json. I use search a lot and didn't even realize topic: was a valid prefix. Just guessed it might work and tried it with pleasant surprise!

Kind of like jumping to search within a repo will add repo:..., owner:..., or repo:... path:....