Update Graduation Requirements + CII Levels

[caniszczyk]

On the last TOC call there was a call to look at updating the graduation requirements and also to look at the various CII levels: https://github.com/cncf/toc/blob/master/process/graduation_criteria.adoc

One idea was to potentially have the incubating level require the "silver" level for CII and graduated "gold" level:

https://github.com/coreinfrastructure/best-practices-badge/blob/master/doc/other.md#silver-passing1-criteria
https://github.com/coreinfrastructure/best-practices-badge/blob/master/doc/other.md#gold-passing2-criteria

On the next TOC call, we will have one of the main CII authors to go over the levels and ask any questions.

[JustinCappos]

As much as I am a supporter of the CII badging program, I think it would be a mistake to have the silver and gold criteria required for CNCF projects. Note, there are only 8 project with a silver badge and 2 with a gold badge. The TUF project is the only CNCF project to have a Silver CII badge or higher. Of the 2 projects with a gold badge, one is the "Badge App" project. Obtaining a higher badge is a large amount of work and there are often good reasons why a project with a serious security posture may not go through all the steps. For example, the TUF project cannot get a gold badge due to issues with 2FA on github via hardware tokens for devs that don't have a cell phone. There are quite a few other steps that may seem sensible at the outset but which may not be a reasonable requirement for some organizations. As such, I think having a posture where the question "What steps didn't you do to get a silver or gold badge and why?" is a much saner way to go forward. I'm saying this as someone with experience using CII best practice badges across different projects. I think the effort is great and often actively promote it to others. I just don't think it makes sense to mandate CII badges above the basic level for CNCF projects. Justin

…

On Tue, Feb 12, 2019 at 10:41 AM Chris Aniszczyk ***@***.***> wrote: On the last TOC call there was a call to look at updating the graduation requirements and also to look at the various CII levels: https://github.com/cncf/toc/blob/master/process/graduation_criteria.adoc One idea was to potentially have the incubating level require the "silver" level for CII and graduated "gold" level: https://github.com/coreinfrastructure/best-practices-badge/blob/master/doc/other.md#silver-passing1-criteria https://github.com/coreinfrastructure/best-practices-badge/blob/master/doc/other.md#gold-passing2-criteria On the next TOC call, we will have one of the main CII authors to go over the levels and ask any questions. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#196>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA0XD2h9xO0Z8JtOBq_UKWvGm6KoE29Vks5vMuCygaJpZM4a2teY> .

[mattklein123]

As much as I am a supporter of the CII badging program, I think it would be a mistake to have the silver and gold criteria required for CNCF projects.

I agree. My feeling is that we need a blended badge, or we should work with the CII folks to make modifications to the existing levels. From my perspective, some of the requirements in basic should be in higher levels, and some of the requirements from higher levels should be in the basic level (or whatever level we choose CNCF graduated projects to adhere to).

My specific requests for graduated projects above basic CII would be:

• Define/document project governance (from Silver, already called out in CNCF requirements)
• "Bus factor" of 2 or more (from Silver, already essentially called out in CNCF requirements for multiple orgs and maintainers)
• Pick & follow coding standards (from Silver, at least some sort of documented style guide and optimally automated style checking/fixing where available in CI, e.g., clang-format, gofmt, etc.)
• Statement coverage 90%+ (from Gold, as CII says, high statement coverage is not indicative of a good test suite, but low test coverage is indicative of a bad test suite so still useful IMO)
• At least 100% of all modifications are reviewed by another (Gold only calls for 50%. I think this is not sufficient and any graduated project must require code reviews from at least one other person, optimally a person from a different org to avoid forcing through org specific changes).
• Have a security review (internal or external) (from Gold. I think security reviews are dubiously useful, but like code coverage, it can't hurt and should flush out obvious issues)
• Have a reproducible build / CI (from Gold, table stakes IMO)

[ultrasaurus]

Generally agree with @mattklein123 -- we want to encourage projects to create a resilient, diverse community of contributors (which seems to be reserved for "Gold" in CII)

On security, we've been talking about creating some guidance about projects being aware of their own security concerns via self-assessment and peer review. We came up with a presentation template for projects to communicate what prospective users or integrators might need to know about the security implications of adopting the project, and we were planning to invite some projects to present to our collective of security experts to see if it works well as a part of a security audit process.

There are a bunch of things we can do in CNCF (with limited # of projects and established community) that are beyond what CII can offer, so we can consider taking a more nuanced approach to some of the "higher level" badging criteria.

[jbeda]

One other idea for graduation criteria -- a documented and reasonable support policy. Not sure what we want there but we want to make sure that at least N previous versions (or N months after a version released) will receive critical (including security) bug fixes.

[timothysc]

@quinton-hoole ^ This is exactly what we were talking about during our LTS conversation.

[mattklein123]

One other idea for graduation criteria -- a documented and reasonable support policy. Not sure what we want there but we want to make sure that at least N previous versions (or N months after a version released) will receive critical (including security) bug fixes.

Joe, I like this idea, but I think we should talk through what this means from a resourcing perspective. I.e., this requires some type of release management, branch management, cherry picking, etc. which can end up being a non-trivial amount of effort. We have shied away from doing this on Envoy because we have no one that has shown interest in this yet. So, like the idea, but concerned about funding/time.

[mattklein123]

(Also, to clarify, lots of people ask for this on Envoy, so we would love it to happen if we could fund it.)

[bgrant0607]

As I mentioned during the meeting, one reason to work with CII is to broaden the impact beyond our direct projects. One motivation is that our projects depend on many other projects. Dependencies will be our weakest link if we don't address them.

[bgrant0607]

The small numbers of projects with silver and gold badges suggests to me that those criteria should be easy to change

[amye]

Closing as we've updated guidelines more recently