Implement report.md formatting

[matejchalk]

As user whose trying out Code PushUp for the first time, I want an easy to read (i.e. not raw JSON) report after I've succesfully ran the CLI for the first time (based on getting started docs).

Based on discussion in #93

General notes

• scores (0-1) formatted as Math.round(score * 100) (0-100, only integers)
• links to headings have to slugify the heading text
• everything that's in <details> (including <summary>) must use HTML formatting, Markdown won't be rendered
• score colours:
  • category and groups scores are annotated with 🟢, 🟡, or 🔴
  • audit value and score are annotated with 🟩, 🟨, or 🟥
  • determined by score value:
    • score < 0.5 => red
    • 0.5 <= score < 0.9 => yellow
    • score >= 0.9 => green

Structure

• heading 1: "Code PushUp report"
• intro section
  • Markdown table with category scores
  • Category cells contain title with link to category detail
  • Audits cells should number of referenced audits (from category refs, ref to group expanded to number of audits)
• heading 2: "Categories"
• categories section - lists all categories
  • category detail:
    • heading 3: category title
    • score
    • unordered list based on category refs (regardless of weight)
      • referenced audits:
        • colour based on audit's score
        • audit title with link to audit detail
        • plugin title in brackets
        • display value in bold - if missing, show raw value instead
      • referenced groups:
        • colour based on group's score
        • group title with plugin title in brackets
        • sub-list with all audits referenced by group
          • audit's formatting is the same as for those referenced by category directly (see above) - only difference is plugin title is omitted
• heading 2: "Audits"
• audits section - lists all audits from all plugins
  • audit detail:
    • heading 3: audit title with plugin in brackets
    • result summary: colour from score, display/raw value in bold, score in brackets
    • if audit has details with issues, result used as summary for expandable details section:
      • heading 4: "Issues"
      • table which list all issues (we only have source code issues at present)
        • severity - icons: error => 🚨, warning => ⚠️, info => ℹ️ (or nothing?)
        • message (nice-to-have: transform MD to HTML)
        • source file path - file links part of Git URLs in Markdown reports and diffs #149
        • source file line(s) - 7 if only start line, 7-10 if both (different) start and end lines, - if no lines
    • audit description (if present)
    • link to audit docs (if present)
• heading 2: "About"
• Report was created by Code PushUp on DATE.
  • as URL, use CLI package README link for now (later: our website)
  • date formatted as new Date().toString()
• table with metadata about the run:
  • commit message and truncated hash (current commit in Git) - commit links part of Git URLs in Markdown reports and diffs #149
  • package version
  • duration (formatted in seconds)
  • number of plugins
  • number of audits (in total)
  • number of categories
• The following plugins were run:
• table with metadata for each plugin:
  • plugin title, with link to plugin docs (if present)
  • number of audits (from this plugin)
  • plugin's package version (if present, otherwise n/a)
  • plugin's duration

Example

Follows below line...

---

Code PushUp report

🏷️ Category	⭐ Score	🛡️ Audits
Performance	🟢 92	5
Code style	🟡 54	12

🏷️ Categories

Performance

🟢 Score: 92

• 🟢 Performance (Lighthouse):
  • 🟨 Largest Contentful Paint - 1.5 s
• 🟥 Disallow missing key props in iterators/collection literals (ESLint) - 1 error

Code style

🟡 Score: 54

• 🟥 Require or disallow method and property shorthand syntax for object literals (ESLint) - 4 warnings
• 🟩 Require let or const instead of var (ESLint) - 0

🛡️ Audits

Largest Contentful Paint (Lighthouse)

🟨 1.5 s (score: 75)

📖 Docs

Disallow missing key props in iterators/collection literals (ESLint)

🟥 1 error (score: 0)

Issues

Severity	Message	Source file	Line(s)
🚨 error	Missing "key" prop for element in iterator	src/components/TodoList.jsx	7

ESLint rule jsx-key, from react plugin. 📖 Docs

Require or disallow method and property shorthand syntax for object literals (ESLint)

🟥 4 warnings (score: 0)

Issues

Severity	Message	Source file	Line(s)
⚠️ warning	Expected property shorthand	src/hooks/useTodos.js	19
⚠️ warning	Expected property shorthand	src/hooks/useTodos.js	32
⚠️ warning	Expected property shorthand	src/hooks/useTodos.js	33

ESLint rule object-shorthand. 📖 Docs

Require let or const instead of var (ESLint)

🟩 0 (score: 100)

ESLint rule no-var. 📖 Docs

About

Report was created by Code PushUp on Tue Oct 24 2023 09:38:51 GMT+0200 (Central European Summer Time).

Commit	Version	Duration	Plugins	Categories	Audits
Implement todos list (3ac01d1)	0.1.0	1.6 s	2	3	25

The following plugins were run:

Plugin	Audits	Version	Duration
ESLint	20	0.1.0	0.3 s
Lighthouse	5	0.1.0	1.2 s

[BioPhoton]

I recommend listing also the audits weighted with 0 as they might be critical things but not possible to score...

Thoughts?

[BioPhoton]

If we would inline the score of a category we could save some lines

BEFORE:

Performance

🟢 Score: 92

AFTER:

🟢 Performance: 92

[BioPhoton]

Remark on the "Categories Section".

Let's also show how multiple resulting issues in the examples e.g. 1 error, 4 warnings
The order should be: error, warning, info.

Same for the issue table.

BEFORE:

Code style

🟡 Score: 54

• 🟥 Require or disallow method and property shorthand syntax for object literals (ESLint) - 4 warnings

AFTER:

Code style

🟡 Score: 54

• 🟥 Require or disallow method and property shorthand syntax for object literals (ESLint) - 1 error, 4 warnings

[BioPhoton]

It is a bit "scrolly" to focus on a goal e.g. performance and have to scroll from the list of categories down to the audits and look up where it. Then start again in categories.

In stead of grouping by data attribute we could take the view of our personas (we have to write them up) and try to make it easy for them to fulfil their needs. Getting better numbers in goal XYZ.

I would create a persona, and put an issue in the back lock. This can be considered when we have the PR comment version

[BioPhoton]

I believe that the colors ad quite some complexity. From a product owner perspective I would try to evaluate the need for that feature with the customer.

If we keep it we have to consider that not all goals are "green" at a score of 90 but also can be e.g. 60.

I guess we can understand that problem better when we have implemented the assertion logic and the user defined budgets

[matejchalk]

It is a bit "scrolly" to focus on a goal e.g. performance and have to scroll from the list of categories down to the audits and look up where it. Then start again in categories.

Hmm, there's quite a simple way to solve that, I think 🤔 In each audit's detail, we could include something like "Referenced by categories: Performance, Code style.", that way you can easily jump back to the category.

I don't want to change the scope of this issue, but I can create a follow-up issue for that. What do you think?

In stead of grouping by data attribute we could take the view of our personas (we have to write them up) and try to make it easy for them to fulfil their needs. Getting better numbers in goal XYZ.

I would create a persona, and put an issue in the back lock. This can be considered when we have the PR comment version

Sure, personas need to be considered. Again, let's separate that discussion from this issue. But I can mention quickly how I think about it conceptually in relation to how the report is structured:

• "non-technical role" - only cares about and understands categories (that's why we always list them first)
• "technical role" - cares about categories and audits, sometimes may dig into issues (that's why those are collapsed)

I believe that the colors ad quite some complexity. From a product owner perspective I would try to evaluate the need for that feature with the customer.
If we keep it we have to consider that not all goals are "green" at a score of 90 but also can be e.g. 60.

The color is always gonna be a bit of a simplification, because there's a limited set of values ("good", "bad" and "something in between"). But I don't think that's a problem if we also show the actual score next to it, the goal is just to add some extra visual cues. It's possible the thresholds won't be universal and we can solve that by enabling the color grading to be configurable, but I wouldn't complicate it too early until we get some more definitive customer feedback.

[matejchalk]

Sorry, didn't scroll up all the way and I missed some of the earlier comments 🤦

I recommend listing also the audits weighted with 0 as they might be critical things but not possible to score...
Thoughts?

Yes, I didn't mention this explicitly (added it now), but I would simply list every audit/group the category references, regardless of weight.

If we would inline the score of a category we could save some lines

That also looks good for categories, in my opinion. But I think we should be consistent with audits as well (a result should ideally always look the same, e.g. in my proposal it's always <color> <text with main value in bold>), and there the title would become too cluttered. And I wouldn't be worried about saving lines here, the main thing is readability.

Let's also show how multiple resulting issues in the examples e.g. 1 error, 4 warnings
The order should be: error, warning, info.

Yes, that's how it will be formatted 🙂 It's actually already handled by the ESLint plugin (implementation here - you can see it uses helper functions from @code-pushup/utils) and provided as the displayValue.

This actually makes the formatter's logic quite simple, it basically just has to do chalk.bold(audit.displayValue || audit.value.toString()).

[BioPhoton]

I don't want to change the scope of this issue, but I can create a follow-up issue for that. What do you think?

Yes let's create something and use it a bit to feel it in real life.