Clarify the average score for checks

[jpmckinney]

On the compiled release page, for each type of check, there's:

7 checks in total with avg score 63%

We should add an info bubble to explain how the average is calculated.

Also, at the OCDS retreat, people wanted an explanation of the pass/fail numbers in the tables. We can perhaps describe these in the same way we describe the numbers for field-level checks.

[hrubyjan]

We added info icon

I would also maybe change the methodology of how the number is calculated. Right now it's a simply average of green numbers but this gives real low result when there there is almost no failure but high number of compiled releases where the calculation is Not available

[jpmckinney]

Hmm, yes, I think we should change the methodology. What do you suggest?

[hrubyjan]

@jpmckinney please review the text but otherwise the calculation was modified as we agreed. It's a percentage of passed check from total number of passed and failed checks.

Maybe this sentence should be changed too as it's not an average anymore

[jpmckinney]

@yolile @pindec Are these new descriptions easy to interpret?

2 checks in total with average score:

2 checks with aggregate score:

The score is computed as the percentage of total successful check applications from total applications; excluding not available applications.

The pass rate of compiled releases against applicable checks.

[pindec]

2 checks with aggregate score

The pass rate of compiled releases against applicable checks

I think "aggregate" and "applicable" are a bit ambiguous. It took me a bit of time to figure out how the new score is calculated.

I agree that where there are, say, 7 checks, but only 2 are used to give the 'average' score, the current text is misleading.

Would it be clearer to:

• explicitly separate the numbers for available checks and "Not available" / n/a checks, and
• use 'adjusted' instead of 'aggregate'?

Suggested onscreen text:

2 checks available (of 7), adjusted pass rate: 87%

Suggested explanation:

The average of the pass rates for available checks, each adjusted to exclude any not available fields.

?

Or would it be clearer to also add the 'adjusted pass rates', the basis of the adjusted/aggregated score, also to the summary table?

[yolile]

I think "aggregate" and "applicable" are a bit ambiguous. It took me a bit of time to figure out how the new score is calculated.

+1 to this, but maybe

2 checks available (of 7) and passed, adjusted pass rate: 87%

[jpmckinney]

N/A here stands for not applicable. The check is available, but it's not applicable.

Why is "adjusted" clearer than "aggregate"? "Aggregate" at least suggests a combination of each check's pass rate. "Adjusted" suggests an adjustment, which seems like less information.

How about: "2 out of 7 checks report a pass rate of %X"

And then: "N/A results are excluded from the pass rate, which is: all passed / (all passed + all failed)."

Or would it be clearer to also add the 'adjusted pass rates', the basis of the adjusted/aggregated score, also to the summary table?

I think four percentages per row would be too confusing.

[pindec]

N/A here stands for not applicable.

That's what I assumed at first for N/A, but the details page shows "not available" given for that field.

So to clarify: the check is available (checks are always available), but some data is not available, so the check is not applicable for all compiled releases (in this case 7%), is that correct?

If so, the fields and tool tip here should be amended:

• "Not available" field -> "Not applicable (N/A)" or just "N/A"
• Explanation/tooltip currently "Each compiled release is checked, and it either passes, fails or the check is inapplicable" -> "Each compiled release is checked, and it either passes, fails or the check is not applicable (N/A)" - to explicitly link it to the n/a column on the summary page.

Why is "adjusted" clearer than "aggregate"?

Good question! Neither exactly conveys the removal of N/A results but to me aggregate implies combining without removing. I also agree adjusting is more ambiguous since there is no information conveyed about the method of adjustment, but I was trying to avoid users assuming from 'aggregate' that it was a straight combination of scores. The 'adjust' was a suggestion to encourage users to read the explanation/tooltip.

How about: "2 out of 7 checks report a pass rate of %X"

I think this omits two important pieces of information:

• That those 2 checks are not arbitrarily chosen from the seven, but chosen on the basis of any pass/fail data being calculable
• That there is some adjustment to the pass rate so it's not a straight average of the data immediately below this message, but with N/As removed.

How about: "Of 7 checks, 2 are calculable, with an applicable pass rate of %X".

And then a version of your suggested tooltip: "N/A results are excluded from the applicable pass rate, which is: all passed / (all passed + all failed)."

Not totally convinced by using applicable above, because of its ambiguity, but I'm struggling to find a good alternative description for the pass rate that immediately conveys "removal of some n/a data".

[jpmckinney]

? So to clarify: the check is available (checks are always available), but some data is not available, so the check is not applicable for all compiled releases (in this case 7%), is that correct?

Yes. The label on the detail page is indeed confusing. Created a new issue: https://github.com/open-contracting/pelican/issues/62

Of 7 checks, 2 are calculable, with an applicable pass rate of %X
N/A results are excluded from the applicable pass rate, which is: all passed / (all passed + all failed).

Okay, getting there :) My edit:

averageScore.description: 2 of 7 checks are applicable, with a pass rate of X%, excluding n/a
averageScore.tooltip: The pass rate is calculated as: all passed / (all passed + all failed)

[pindec]

2 of 7 checks are applicable, with a pass rate of X%, excluding n/a
The pass rate is calculated as: all passed / (all passed + all failed)

+1 :)

[jpmckinney]

Forgot to reassign @hrubyjan, since the new text is now agreed as above.

[jpmckinney]

Blocked by #104