[RFC] UI for Scorecard component -> Tech Insights stage 2

[Irma12]

Status: Open for comments

Proposal

This RFC contains proposed solution for continuation of PRFC Tech Insights, specifically going into stage 2.
The purpose of this PR is to propose overall UI ‘scorecard’ components generated as a result of stage 1 in a linked Tech Insights PRFC.

Initially we want to focus on entity level, but in the future there might be a possibility to approach this in a more general level, so we would create an aggregation of multiple scorecards and group them per owning group or multiple entities.

Idea we have is to display ‘scorecards’ widget in entity Overview page where it would be easy to see number of checks passed for each scorecard.

Just for a general idea purpose, we showed case where there would be multiple scorecards (Production ready and Development ready). Each of the scorecards would contain more details which would be seen when scorecards page is opened

.

In this overview it is possible to expand checks in order to get information about them and also proposed actions that could be taken in order to pass them. If specific check is calculated based on gauge or graph fact calculations, those would also be displayed in this section. Please keep in mind the checks stated in the wireframe are just example of what could be seen, but it can be anything, such as:

• Service has been deployed at least once in the past 30 days.
• Service has an owner defined.
• Test coverage is greater than 50%
• React package version is greater than 17.0.0
• Service has no critical security vulnerabilities in Snyk

Actions could include various things such as a link to an entity-info file, link to the test coverage calculation, link to vulnerabilities list, link to a package.json file etc.
For initial version we would hold all checks equally important so we wouldn’t make a difference in a ‘weight’ of a specific check. However, changes in their importance or value may be introduced afterwards.

[andrewthauer]

First off thank you! It is great to see some serious thought going into what could be an extremely powerful tool within Backstage. Our team is very interested in the idea of scorecards within Backstage.

As discussed in #7193, I still think having a single binary check state (pass/fail) is too limiting. Backstage has the potential to bring concerns or issues to service owners attention under a single pane of glass. Even a simple grading system like from A to E would be much more broadly useful then a pass/fail imo. For example, some "pass" states could be quite difficult and labour intensive to achieve, so being able to improve incrementally would be helpful from a motivation and accomplishment perspective.

That said, having binary pass/fail checks to start is still useful and in some cases might be all that is needed. I do worry that using "checks" as the initial use case might make the grading/scoring systems more difficult to add in afterwards versus up front.

[dtuite]

@andrewthauer In my mind that's an enhancement which could be built on top of the checks. For example, give the grade E if 2 out of 10 checks are satisfied, and give the grade A if all 10 checks are satisfied. The checks themselves are still boolean values though. Does that cover your need?

[andrewthauer]

@dtuite - I know @Xantier has designed the backend to support this in the future (which is great). It just seems like such a common use case which could likely be handled up front easier then trying to tack on afterwards. For instance, if stage 2 proceeds with the idea around just binary checks it is very likely much of the UI will be tailored to pass/fails and trying to retrofit handling a multi grade scoring system could be significant lift and/or require a completely new UI.

For example, give the grade E if 2 out of 10 checks are satisfied, and give the grade A if all 10 checks are satisfied.

This implies that binary checks would be required for a grading system. There are lots of cases where single fact number could derive a grade. For example, 0 outstanding dependency updates is a grade A, 1-5 is a B, 6-9 is a C, >= 10 is D.

The binary checks being proposed as the initial check presentation could be easily expressed as a grading/scoring system of 2 thresholds. So my overall point here, is I feel that multi state scoring/grading is a such a common use case that it makes sense to handle it up front. Again, my worry is it would be large effort adding after vs up front.

[Xantier]

As discussed in #7193, I still think having a single binary check state (pass/fail) is too limiting.

That's a valid point and I agree with you on this. Having scorecards that handle different kinds of checks is valuable and should be considered as well.

That being said, I'm kinda failing to understand the limitations of these specific mockups when it comes to designing scorecards. Maybe it's because I have too much implicit information about the actual implementation of this first iteration of these components.

If we think about scorecards in general they are variations of the same pattern. They have an identifier, the actual value and how it relates to a 'grading system' (pass/fail, A-F, 4-10, 1-5, w/e) and some extra information. In this case the mocks show 'check name' as the identifier, a type of boolean check - a checkmark - value, as value/grading system pair and additional information which may or may not be visible depending on the position of the clickable caret. The debated area is the 'checkmark' component here.

I do worry that using "checks" as the initial use case might make the grading/scoring systems more difficult to add in afterwards versus up front

I'm not sure this will end up being a problem. Without going too much into implementation details, we do get the type of the check as part of the result and can swap the rendering path based on that to display different grading systems. The overall design would still work with different grading types, even if they are not explicitly drawn into visuals in these mocks.

The only problem I'm seeing are cases where for some reason different systems would be used in the same scorecard. Aggregating the scorecard results from those would be impossible without complex comparison table logic. I do believe that is something that we should restrict from being to be done. It is better to create multiple scorecards for those cases.

[Irma12]

In addition to what @Xantier elaborated in his comment, and in order to visualize it better, I modified mockups a bit. In a term of 'status' component, I understand checkmark/x sign might be looking a bit too strict, but that would be used only for boolean type of checks. We can display something else (grade, percentage..) all depending on the type.

[andrewthauer]

Thank you @Xantier and @Irma12. My concern was due to the fact that all the examples being used seemed to binary based which is why I brought this up. It's great to hear this is being built in up front. So these clarifications are super helpful!

Very excited to see this coming together and contribute at some point.

Am I correct in assuming the that a non binary scoring system would require a different backend module? From my understanding there would need to be an equivalent to the initial jsonfc module in #7679. Just curious about what would be required for additional check types.

[Xantier]

Yup, that is the correct assumption. A similar structure to jsonfc would be needed to fetch facts and mangle them into a preferred format. For multiple different engines running at the same time the kinda hand-wavy plan is to share the 'CheckRegistry' and have a single CheckEngine acting as an adapter for more specific engines.

[seshank-db]

@Irma12 any update on stage 2?

[Irma12]

hey @seshank-db, this is released in tech insights plugin :)

[freben]

@Irma12 Am I understanding it right that this issue was actually addressed and can be closed? 🤔

[seshank-db]

@Irma12 tech insights README plugin still shows a simple boolean scorecard. Is this view already present?

[Irma12]

@freben Yup, it has been worked on in #8086. The main idea for this RFC was to get initial thoughts and direction in which we would like to proceed FE wise :) I think we managed to gather the thoughts in tech-insights plugin so I think for other suggestions and continuation we could probably open new RFC.

@seshank-db Yes, currently there is only Boolean scorecard view present, which will show values for the @backstage/tech-insights-backend plugin. Since currently tech-insights-backend-module-jsonfc (which is an addition to tech-insights-backend) provides functionality to run checks against a json-rules-engine and return boolean values, we went ahead and created a Boolean Scorecard in order to present this.

However, if you wish to customize or create new type of scorecard and maybe use other values than boolean, you will firstly need to create a similar structure to tech-insights-backend-module-jsonfc in order to fetch facts and mangle them into a preferred format, and then create customized scorecard you wish to display instead of Boolean. In this file you can see that defaultCheckResultRenderers will display BooleanCheck (boolean scorecard) for 'json-rules-engine' type. In similar way you could proceed with some new/customized scorecards. I hope this makes sense :)