[feedback] How can projects express they meet a given OSPS level?

[TheFoxAtWork]

tl;dr

Projects need a mechanism, today, that allows them to express how they meet a given OSPS level or set of OSPS criteria. We've been point at security-insights.yml but it isnt ready yet.

Details

We've rolled out the baselines, defined the criteria, and how projects can implement the criteria, but not yet provided them a mechanism by which they can capture the how. We currently are currently surfing some momentum given OSPS awareness, but this can quickly die if we fail to provide the means for projects to do this. Many individuals I've presented the OSPS baselines to cite this makes sense, they're already doing a lot of them, now how do they show it?

UX

For now, i've recommended to project to create a project page or file in their repo that grabs the criteria and just write out how they do it. This means they'll have double effort first in writing it out and then in converting to the security-insights.yml spec when it is ready. We need to narrow the window for projects jumping on this in the time between our announcement and the tooling/spec to allow expression to happen so we don't inadvertently introduce more effort or stale information that projects then need to go back and convert.

Desired Outcome

• Propose starter template for projects to address Add SECURITY_INSIGHTS.yml starter file to ossf/project-template #20
  • https://github.com/ossf/project-template/pull/6/files
• Propose changes to the security-insights.yml to define OSPS baseline expression by projects. alternatively reconfirm this is the path forward we want to take.
• Campaign/advocate on the concurred changes to security-insights.yml (or other agreed mechanism) to inform projects this is the expression location of OSPS baseline criteria implementation.

[eddie-knight]

We've had reasonably positive feedback from projects adopting Security Insights v2, and I plan to keep pointing folks there where applicable. The only open gap I'm aware of is wrt EoX, which has an enhancement proposal open... but no other gaps are coming to mind at the moment. Do you see anything else missing?

[TheFoxAtWork]

Pointing people at Security Insights v2 makes sense, however thinking about the specifics of what we're asking projects to do and how that information is intended to be conveyed in a digestible format:

• The current iteration of security insights provides a layout for a conglomerate of information that aligns with the OSPS criteria but is not granular as to which areas of the spec match to which criteria.
  • this makes is difficult for any observer to ascertain whether a project does meet level 1 without interpretation on their part.
  • Perhaps this can be solved with an "interpreter" that checks the security insights file contents against the criteria mapping for OSPS to issue an osps lvl repo "badge" to make it easy for the casual observer. This is how most foundation oversight committees will be verifying this as a requirement for moving levels.
• Projects looking to express conformance with the OSPS baseline may expect to leverage the control listing as a list to ensure they didn't miss any criteria. The current structure of the security-insights file doesn't lend itself to this UX and may result in confusion by the maintainer to express in that manner without improvements.
  • Adjusting or modifying the specification could allow for better tracking/visibility of how criteria are implemented to better detect or monitor changes by the project (or tooling).
• The absence of information may leave an observer of the repo to question full implementation at a level. Several of the baseline criteria are configurable but not immediately observable without special permissions (i.e. MFA). We can infer they're in place (depending on the source) however inference isn't a measurable statement (at least in this context).

Is this helpful?

[funnelfiasco]

My concern with anything more than a "we attest that we meet Baseline level X (version Y) as of date Z" is that it will lead to the kinds of problems Scorecard has had: people opening issues demanding various changes to the project without providing any help or support for it.

I agree that maintainers need better ways to evaluate what level they meet (and where the gaps are for reaching the next level). However, that's a separate question from what's presented to observers. It's very important that we don't lump the two together.

[TheFoxAtWork]

Thats a great point @funnelfiasco . I would like to avoid the problems scorecard had. So this really should be broken into two areas like you suggested:

• improving the way maintainers can evaluate and track their implementation of a given OSPS baseline
• clearly conveying to observers what level a project implements (i.e. simple badging) and how (link to insights/mapping to OSPS)

[SecurityCRob]

We also can leverage the POWER of the Best Practices Badges to deploy a badge to a repo if they meet a level of criteria

[TheFoxAtWork]

I'm glad you mentioned that CRob -
As I've shared the OSPS Baseline with community members, i keep hearing the same feedback: we need reconciliation with the other tools and programs (Scorecard and Best Practices Badge). We're asking maintainers to do a whole lot across a lot of things and its not clear which they should be doing with little time to do them. We need to keep it simple. And that we need tooling to make this easy.

[SecurityCRob]

In a future ORBIT meeting I'd like us to talk about our desired workflows and get those documented. I see one path where projects enter the "Baseline Ecosystem" through the BP Badge. Another path would be would be through using a tool like Darnit. The attestations we're talking about will also be another signal for us (Insights, in-toto, etc.). there will be a manual path where an "expert" gives a project a white-glove experience walking them through the framework. Then tools like Scorecard, LFX that will be picking up Baseline criteria will inspire people to question what to do as "next steps", so we should have instructions on what they could do.

[david-a-wheeler]

The Best Practices Badge already includes a mechanism to include a description, for each criteria, on why that criteria is met (or in some cases not met). We recommend a short description with a link to any details. At higher levels we require that data for some criteria, because automated tools have false+ and false-.