Improving readability of PR dry-run comments for complex changes

[StephHope]

Background

We're in the process of rolling out safe-settings across our organisation and have been really impressed with the tool — the policy-as-code approach is exactly what we needed. Thank you to everyone who has contributed to it.

As we've started working with more complex configurations (multiple suborgs, repos with many teams and permission levels), we've noticed that the PR dry-run comment becomes difficult to review in practice. The wide table format means column headings scroll off screen, and nested JSON objects inside cells make it hard to parse at a glance — especially for reviewers who aren't deeply familiar with the raw API shapes.

Comment length truncation makes this worse

A more significant problem compounds the readability issue: GitHub imposes a maximum length on PR comments, and for larger configurations the safe-settings comment is truncated before all affected repos are shown. This means reviewers may be looking at an incomplete picture of what will change, and the indication that the comment has been cut off is easy to miss. The data that does appear varies between runs (depending on processing order), so there's no reliable way to know which repos are missing from any given comment.

What we've done locally

We've written a script our team runs locally to reformat the comment into a more human-readable structure. Rather than a wide table with JSON blobs, it outputs a hierarchical view grouped by setting type, with additions, modifications, and deletions broken out clearly and nested fields rendered as indented key-value pairs.

The difference in reviewability for a change touching several teams across multiple repos is significant — our reviewers can actually confirm what's changing without needing to scroll horizontally or mentally parse JSON.

What we're wondering

Before investing time in a PR, we wanted to open a discussion on a couple of things:

1. Is the comment format something the maintainers are open to changing or making configurable? We understand there may be deliberate reasons for the current format (e.g. machine-parseability, consistency with the checkrun output, or just the difficulty of formatting Markdown tables that work across all GitHub contexts).

2. If there's appetite for a change, what approach would you prefer? We're considering a few options, roughly in order from most to least impactful:

   • Split the output across multiple comments when it would exceed GitHub's comment length limit, so all affected repos are always shown
   • Replace the table with a structured Markdown format (collapsible <details> sections per repo/setting type) that is more compact and readable within GitHub's comment constraints
   • Make the comment format configurable (e.g. a PR_COMMENT_FORMAT env var with table and readable options)
   • Keep the current format but filter out unchanged/null fields to reduce noise and stay within the comment length limit more often
   • At minimum, make the truncation warning more obvious when the comment has been cut off, so reviewers know the output is incomplete

3. If this isn't something you'd want in the main project, that's completely fine — we'll continue maintaining it in our fork. We just didn't want to duplicate effort if others have the same pain point.

Happy to share screenshots of the current vs. reformatted output if that would help illustrate the problem.

Thanks for building and maintaining this — it's been a great foundation for us.