🌱 convert CII Best Practices check to probes

[AdamKorcz]

What kind of change does this PR introduce?

feature

• PR title follows the guidelines defined in our pull request documentation

What is the new behavior (if this is a feature change)?**

This PR converts the CII Best Practices to 6 different probes.

• Tests for the changes have been added (for bug fixes/features)

Which issue(s) this PR fixes

NONE

Does this PR introduce a user-facing change?

No.

[codecov]

Codecov Report

Merging #3520 (8a85ba1) into main (6857320) will decrease coverage by 12.65%.
The diff coverage is 73.25%.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #3520       +/-   ##
===========================================
- Coverage   76.39%   63.74%   -12.65%     
===========================================
  Files         209      198       -11     
  Lines       14310    13547      -763     
===========================================
- Hits        10932     8636     -2296     
- Misses       2737     4355     +1618     
+ Partials      641      556       -85     

[AdamKorcz]

@laurentsimon @spencerschrock PTAL again.

[evverx]

I don't think #3520 (comment) was addressed. That paragraph is still misleading and "that" hasn't been removed.

The OpenSSF Best Practices badge indicates whether or not that the project uses a set of security-focused best development practices for open source software.

[AdamKorcz]

I don't think #3520 (comment) was addressed. That paragraph is still misleading and "that" hasn't been removed.

The OpenSSF Best Practices badge indicates whether or not that the project uses a set of security-focused best development practices for open source software.

Missed that the other probes had the same issue. Fixed now.

[evverx]

To be honest I'm much more interested in getting the wording right :-) It's just that I don't think that badges should be conflated with the best practices. All this check shows is that projects have badges and it doesn't necessarily mean that they actually follow those practices. Projects without badges can follow the practices too.

[laurentsimon]

To be honest I'm much more interested in getting the wording right :-) It's just that I don't think that badges should be conflated with the best practices. All this check shows is that projects have badges and it doesn't necessarily mean that they actually follow those practices. Projects without badges can follow the practices too.

I think it's fine to reword. Any suggestion? I'm not sure which best practices are automatically checked, and which are not (in which case the best practice badge is essentially about education; and that's something we could highlight, eg maintainers have followed an introductory course on ...)

@david-a-wheeler please feel free to suggest

[evverx]

Any suggestion?

To judge from https://github.com/coreinfrastructure/best-practices-badge/blob/main/docs/vetting.md "The best practices badge approach is based on self-certification" with some basic checks so I'd say it's supposed to encourage good practices and let projects self-assess themselves. Personally I think it's too vague to be useful in automated tools like scorecard. I'm not sure why projects with no badges are downgrade either.

that's something we could highlight, eg maintainers have followed an introductory course on

I think it's being discussed in #3534

[david-a-wheeler]

Personally I think it's too vague to be useful in automated tools like scorecard.

They aren't vague at all. They are carefully worded requirements. Most of them can't be directly measured by Scorecard, so the best practices badge provides a way to measure many criteria that otherwise Scorecard cannot measure.

I'm not sure why projects with no badges are downgrade either.

Because a project not pursuing a badge is unlikely to meet its requirements. Once it starts pursuing a badge it's more likely to reach them.

[david-a-wheeler]

All this check shows is that projects have badges and it doesn't necessarily mean that they actually follow those practices

It does in general mean they follow the practices. If it's automated then it's checked. If it's a human making a claim, then it requires a listed human who is asserting that it's met, typically with a justification. The claim is also posted for all to see. We don't get many lies; if someone does lie, the badge is pulled.

[evverx]

It does in general mean they follow the practices

It does not. I often look at projects with badges and they are usually out-of-date because they were received once when companies felt the need to show their commitment to security by participating in that program and have never been updated since then. Basically it's necessary to manually verify all the claims but if I have the resources and the expertise to verify all those claims I'm not exactly sure how the badge is supposed to help me to evaluate anything.

[david-a-wheeler]

It does not. I often look at projects with badges and they are usually out-of-date

If you have specific cases, file an issue & let's fix them. We're planning to improve the automation & rechecking of results, that should also help in some cases.

But ignoring the best practices badge information isn't an improvement.

[spencerschrock]

/scdiff generate CII-Best-Practices

[github-actions]

Here's a link to the scdiff run