[Graduation] cert-manager Graduation Application

[SgtCoDFish]

cert-manager Graduation Application

@kgamanji suggested we raise this issue to replace the PR we raised initially (#1212), as the process for graduation has changed since we raised that PR.

Project Repo(s): https://github.com/cert-manager/cert-manager (and others under https://github.com/cert-manager)
Project Site: https://cert-manager.io/
Sub-Projects: trust-manager, approver-policy, csi-driver, csi-driver-spiffe, istio-csr
Communication: Kubernetes slack (#cert-manager-dev), regular meetings, mailing list (cert-manager-maintainers@googlegroups.com)

Project points of contact:

• Catch-all maintainer mailing list: cert-manager-maintainers@googlegroups.com
• #cert-manager-dev channel on Kubernetes Slack

Graduation Criteria Summary for cert-manager

Adoption Assertion

The project has been adopted by the following organizations in a testing and integration or production capacity:

Criteria

Application Process Principles

Suggested

N/A

Required

• Give a presentation and engage with the domain specific TAG(s) to increase awareness

  • TAG Security ([Presentation] cert-manager Graduation Overview tag-security#1254) - completed on 2024-05-22
  • TAG Contributor Strategy - @maelvls and others joined on 2024-05-02. Engagement ongoing! See this issue.

• TAG provides insight/recommendation of the project in the context of the landscape

TAG-security suggested that a joint assessment would be helpful following our self assessment, to broaden the docs we have around security. We'll engage in that process as the self-assessment completes.

All questions we received during the TAG security meeting were answered and nobody on the call had any red flags about cert-manager.

The self assessment can be viewed in the tag-security repo.

• All project metadata and resources are vendor-neutral.

• Review and acknowledgement of expectations for graduated projects and requirements for moving forward through the CNCF Maturity levels.

  • Met during Project's application on 28-Nov-2023 (in our initial PR cert-manager graduation proposal #1212 )

Completion of this due diligence document, resolution of concerns raised, and presented for public comment satisifies the Due Diligence Review criteria.

• Additional documentation as appropriate for project type, e.g.: installation documentation, end user documentation, reference implementation and/or code samples.

Governance and Maintainers

Note: this section may be augmented by the completion of a Governance Review from TAG Contributor Strategy.

Suggested

• Governance has continuously been iterated upon by the project as a result of their experience applying it, with the governance history demonstrating evolution of maturity alongside the project's maturity evolution.

We've looked to expand our governance and we're looking to continue expanding it

Required

• Clear and discoverable project governance documentation.

https://github.com/cert-manager/community/blob/main/GOVERNANCE.md

• Governance is up to date with actual project activities, including any meetings, elections, leadership, or approval processes.

This was a WIP item for us, but our current governance docs are up to date. We might iterate more on spefically the steering committee stuff, but we've now had a steering committee meeting and iterated on our roadmap from that.

• Governance clearly documents vendor-neutrality of project direction.

Most of our decisions today are made by maintainers who're actively involved in the project, and we've set out a clear path for people to become maintainers. We have maintainers spread across several companies and we'd gladly accept more.

Our roadmap is based on (and links to) this document which has vendor neutrality at its core.

• Document how the project makes decisions on leadership roles, contribution acceptance, requests to the CNCF, and changes to governance or project goals.

This is included in our governance docs and in our docs for contributors. See for example our feature policy.

• Document how role, function-based members, or sub-teams are assigned, onboarded, and removed for specific teams (example: Security Response Committee).

• Document complete list of current maintainers, including names, contact information, domain of responsibility, and affiliation.

All maintainers share all domains of responsbility currently. List here.

• A number of active maintainers which is appropriate to the size and scope of the project.

Our maintainer list currently lists 8 maintainers, with the expectation that at least one more will be onboarded soon.

• Document a complete maintainer lifecycle process (including roles, onboarding, offboarding, and emeritus status).

Documented in GOVERNANCE.md.

• Demonstrate usage of the maintainer lifecycle with outcomes, either through the addition or replacement of maintainers as project events have required.

April 2024: We've moved a couple of maintainers to emeritus status as they've drifted away from the project. We're hoping to onboard at least one a new maintainer soon to test our documented processes, and we have at least three people on the path to becoming maintainers.

Update July 2024:

We onboarded a new maintainer - Adam (cert-manager/community#23) - which was a good test of the process.

We've also offboarded a maintainer - Irbe (cert-manager/community#31) - at her request since she didn't have the time to contribute.

As of the time of writing, a new maintainer - Erik (cert-manager/community#32) - from a new organisation has passed the vote to become a maintainer. As he's currently on holiday, we'll complete the onboarding when he returns.

We have a few other community members at differing stages along the maintainership path, including Houssem who's an approver for istio-csr and Peter who helps a tonne with testing, validation and admin tasks such as announcements or taking notes in meetings.

• Project maintainers from at least 2 organizations that demonstrates survivability.

As of April 2024: Venafi, Diagrid, Tailscale and G-Research.

As of July 2024: Venafi, Diagrid, G-Research

• Code and Doc ownership in Github and elsewhere matches documented governance roles.

This is documented in the governance process too. Maintainers get roles in GCP for managing test / release infra, and maintainers get elevated privileges in the GitHub org.

• Document agreement that project will adopt CNCF Code of Conduct.

We've operated under the CNCF CoC for years.

• CNCF Code of Conduct is cross-linked from other governance documents.

It's one of the first requirements we have to become a contributor: https://github.com/cert-manager/community/blob/4d35a69437d21b76322157e6284be4cd64e6d2b7/GOVERNANCE.md#contributor

• All subprojects, if any, are listed.

We list them in the website, usually mentioning them when they're appropriate too.

• If the project has subprojects: subproject leadership, contribution, maturity status documented, including add/remove process.

Mentioned at the beginning of our governance docs:

This governance charter applies to every project under the cert-manager GitHub organisation. The term "cert-manager project" refers to any work done under the cert-manager GitHub organisation and includes the cert-manager/cert-manager repository itself as well as cert-manager/trust-manager, cert-manager/approver-policy and all the other repositories under the cert-manager GitHub organisation.

Contributors and Community

Note: this section may be augmented by the completion of a Governance Review from TAG Contributor Strategy.

Suggested

• Contributor ladder with multiple roles for contributors.

Clearly defined in governance docs: https://github.com/cert-manager/community/blob/4d35a69437d21b76322157e6284be4cd64e6d2b7/GOVERNANCE.md#cert-manager-governance

Required

• Clearly defined and discoverable process to submit issues or changes.

GitHub issues / PRs. We have a policy so people know what to expect when they want to propose a bigger change.

• Project must have, and document, at least one public communications channel for users and/or contributors.

We have several, listed at the top of this issue.

• List and document all project communication channels, including subprojects (mail list/slack/etc.). List any non-public communications channels and what their special purpose is.

All listed here.

• Up-to-date public meeting schedulers and/or integration with CNCF calendar.

We have an up-to-date meeting schedule on our website. As a result of raising this issue, I've reached out to the CNCF to look to include our events on the public CNCF calendar.

UPDATE 2024-05-02: We have now integrated with the CNCF calendar for our biweekly meeting, thanks to @maelvls!

• Documentation of how to contribute, with increasing detail as the project matures.

Details here and in other places on the website.

• Demonstrate contributor activity and recruitment.

We regularly onboard new contributors, and each minor cert-manager release contains a thank-you list of who was involved. For example, the alpha release of the next minor release already lists 10 contributors and that will continue to grow: https://github.com/cert-manager/cert-manager/releases/tag/v1.15.0-alpha.0

Engineering Principles

• Document project goals and objectives that illustrate the project’s differentiation in the Cloud Native landscape as well as outlines how this project fulfills an outstanding need and/or solves a problem differently.

The first line of every cert-manager release re-states the ambition we have:

cert-manager is the easiest way to automatically manage certificates in Kubernetes and OpenShift clusters.

We've become essentially the standard for managing certificates in a cloud-native (kubernetes-native) way.

• Document what the project does, and why it does it - including viable cloud native use cases.

Extensive documentation on https://cert-manager.io/docs/

• Document and maintain a public roadmap or other forward looking planning document or tracking mechanism.

We've iterated on the old roadmap along with the steering committee and moved it into the community repo: https://github.com/cert-manager/community/blob/fb1a2477406f4ce563b7ec5044049d29dc79e1d6/ROADMAP.md

• Roadmap change process is documented.

This is going through review currently by community members and members of the steering committee: cert-manager/community#28

• Document overview of project architecture and software design that demonstrates viable cloud native use cases, as part of the project's documentation.

Also available in project docs

• Document the project's release process and guidelines publicly in a RELEASES.md or equivalent file that defines:

  • Release expectations (scheduled or based on feature implementation)
  • Tagging as stable, unstable, and security related releases
  • Information on branch and tag strategies
  • Branch and platform support and length of support
  • Artifacts included in the release.
  • Additional information on topics such as LTS and edge releases are optional. Release expectations are a social contract between the project and its end users and hence changes to these should be well thought out, discussed, socialized and as necessary agreed upon by project leadership before getting rolled out.

Our release process is documented on the website and our support schedule is documented on a different page.

The release process has been updated to reflect recent changes which mean that any maintainer can do a release, and that it's no longer a requirement to be a Venafi employee.

It's worth pointing out that there is still one thing which requires approval of a Venafi employee: publishing Helm charts to charts.jetstack.io. This is difficult to change wholesale to an entirely vendor neutral location since that repo is so widely used.

We intend to provide an alternative install mechanism in the near future (see cert-manager/cert-manager#7132) which is entirely vendor neutral, but to continue publishing to charts.jetstack.io to avoid any kind of break for users.

• History of regular, quality releases.

Our supported releases page lists our extensive release history for cert-manager.

Security

Note: this section may be augemented by a joint-assessment performed by TAG Security.

Suggested

• Achieving OpenSSF Best Practices silver or gold badge.

We'll look into this down the road but it's not an immediate priority for us.

Required

• Clearly defined and discoverable process to report security issues.

SECURITY.md file in all relevant repos, with a mailing list we watch for reports.

• Enforcing Access Control Rules to secure the code base against attacks (Example: two factor authentication enforcement, and/or use of ACL tools.)

2FA required for GitHub org members.

• Document assignment of security response roles and how reports are handled.

This applies to all maintainers and our org security policy is quite detailed.

• Document Security Self-Assessment.

Document here. This has been worked on alongside TAG security. We intend to complete this and merge it to the TAG security repo.

• Third Party Security Review.

  • Moderate and low findings from the Third Party Security Review are planned/tracked for resolution as well as overall thematic findings, such as: improving project contribution guide providing a PR review guide to look for memory leaks and other vulnerabilities the project may be susceptible to by design or language choice ensuring adequate test coverage on all PRs.

Completed, see announcement

• Achieve the Open Source Security Foundation (OpenSSF) Best Practices passing badge.

https://www.bestpractices.dev/en/projects/8079 - badge linked in cert-manager/cert-manager

Ecosystem

Suggested

N/A

Required

• Publicly documented list of adopters, which may indicate their adoption level (dev/trialing, prod, etc.)

Quite out of date but we have one: https://github.com/cert-manager/community/blob/main/USERS.md

We're looking to expand this as part of the graduation process

• Used in appropriate capacity by at least 3 independent + indirect/direct adopters, (these are not required to be in the publicly documented list of adopters)

Used by a few orders of magnitude more adopters than 3!

The project provided the TOC with a list of adopters for verification of use of the project at the level expected, i.e. production use for graduation, dev/test for incubation.

• TOC verification of adopters.

We've added 6 users to the internal graduation google doc who've all agreed to be interviewed.

Refer to the Adoption portion of this document.

• Clearly documented integrations and/or compatibility with other CNCF projects as well as non-CNCF projects.

Our docs are clear around Kubernetes integrations (since that's our primary integration). We also document integrations with Hashicorp Vault, Let's Encrypt, tens of other issuers, SPIFFE (via csi-driver-spiffe) and a huge variety of cloud DNS providers.

Adoption

Adopter 1 - $COMPANY/$INDUSTRY

If the Adopting organization needs to remain anonymous, stating the industry vertical is sufficient.
MONTH YEAR

Adopter 2 - $COMPANY/$INDUSTRY

If the Adopting organization needs to remain anonymous, stating the industry vertical is sufficient.
MONTH YEAR

Adopter 3 - $COMPANY/$INDUSTRY

If the Adopting organization needs to remain anonymous, stating the industry vertical is sufficient.
MONTH YEAR

[kgamanji]

On 23/04, cert-manager maintainers and I had the kick-off meeting for the graduation process. Here are the suggested action items:

• [ONGOING] Update the graduation submission as an issue according to the moving levels template (+close the current PR and reference it): https://github.com/cncf/toc/blob/main/.github/ISSUE_TEMPLATE/template-graduation-application.md#project-graduation-application
• Reach out for TAG assessment:
  • TAG-Contrib-Strategy
  • TAG-Security
• Provide an adopter list for interviews, looking for 5-7 across multiple industries( ref: https://github.com/cert-manager/community/blob/main/USERS.md)
• Share the security audit results and resolution
• Community and steering committee(SC):
  • Share SC meetings as stated per governance: “All recordings and minutes will be posted publicly”. Any progress or input from the SC on the project would be helpful too
  • Most of the commits are from Venafi. Reach out to TAG contrib-strategy for an overview and strategy to attract new maintainers (perhaps mentor up SC members)
• Roadmap:
  • Update the Roadmap to reflect the ongoing and planned work
  • Document the Roadmap change process as part of the governance
• Provide an up-to-date list of sub-projects to be included
• Release:
  • Update the release process to reflect the latest shift to CNCF infrastructure
  • Provide a Release.md file

[SgtCoDFish]

@kgamanji I've updated the description of this issue to reflect the status today. I think we've now ticked off everything we needed to, and I think everything in your follow-up comment is addressed as well!

I think we're ready for the next step now!

[anvega]

Thank you @SgtCoDFish and the cert-manager team for your presentation to TAG Security. The project's growth and evolution impressive, and we'd like to provide some feedback:

1. Ecosystem Integrations: The integration capabilities with Istio, SPIFFE, and other systems are noteworthy, enhancing both security and flexibility.

2. Self-Assessment and Joint Assessment: We appreciate the progress on your self-assessment and look forward to the joint assessment with TAG Security reviewers.

3. Security Audit: The completion of the external security audit by Ada Logics and your efforts to address the findings demonstrate a strong commitment to security. All eight identified issues have been addressed, which is commendable.

4. Best Practices and Documentation: Your thorough documentation of build and deployment processes is excellent.

5. Static Analysis: We encourage you to expand on your current use of go vet and implement govulncheck as planned. It would be beneficial to update your OpenSSF Best Practices information if these have been addressed.

6. OpenSSF Best Practices: We recommend completing the silver and gold level criteria, as you likely already meet most of them.

7. Subproject Consideration: While the main repository is typically the focus of audits, we suggest considering security audits for subprojects like trust-manager and csi-driver. Maybe the CNCF can batch these in a follow-up audit.

Overall, TAG Security is supportive of cert-manager's progression to graduated status in the CNCF. The project demonstrates maturity and a strong commitment to security practices.

We look forward to seeing the continued evolution and success of cert-manager. Keep up the excellent work!

[SgtCoDFish]

Sharing latest questions + answers from Katie here at her request. Quoted sections are from Katie, unquoted are our responses.

Security - for the joined assessment the project should file an issue, as per process guidelines: https://github.com/cncf/tag-security/tree/main/community/assessments#process. Did you open an issue?

We haven't raised an issue for joint tag security assessment yet - we're aiming for that in the future but didn't have the bandwidth just yet!

(As far as we're aware right now that's not a requirement for graduation, just a thing we'd like to do in the future)

Contrib-startegy - did you present to the TAG? Any outcomes?

We did! The detail is tracked in here: cert-manager/community#14

I've also added a link to that issue in the original post of this issue.

@maelvls can expand further if needed!

Any updates on the security audit results and resolution? Did you share that with TAG security?

I shared the audit results in the presentation I made to tag security, we'd already addressed everything from it that we were going to as far as I remember

Reviewed the notes and action items from the first Steering Committee meeting. Great work!
Helm chat release has been updated and the including access to GCP project to allow all contributors to lead the release process
Thank you for providing a Release.md file
Roadmap: Has been updated reflect the ongoing and planned work. Great work! Changes to the governance process reflects the involvement of the Steering committee
Release process has been updates to reflect the latest shift to CNCF infrastructure. Well done!

🚀

Are there any timeframes for the work items? Will the roadmap have releases as milestones? e.g. will it be updated to include the work for 1.16 release and further?

Now that 1.15 is released it would be nice to schedule something in for 1.16 from the roadmap. No plans today but we're in planning mode so we should be able to get something in. A few of the roadmap things are mainly going up involve subprojects so won't necc. be tied to a cert-manager release.

Can you confirm that the list of sub-projects to be included under the overall governance is up-to-date?

Confirmed that the list in this issue is up-to-date.

I have also reached out to 3 of the end users to schedule the adopter interview calls.

This is super exciting - great to be getting closer to being complete with this.

[kgamanji]

As I am putting together the Graduation PR, I am going through all the responses filled in the Graduation issue. Here are a few follow up points:

• Governance:

  • “Most of our decisions today are made by maintainers who’re actively involved in the project” - were there any decisions that were not made by the maintainers in the past?
  • “Our maintainer list currently lists 8 maintainers, with the expectation that at least one more will be onboarded soon” - who is the new onboarded member? Does it refer to Erok: Add erikgb as maintainer cert-manager/community#32?
  • “we have at least three people on the path to becoming maintainers.” - ditto - any evidence to support this statement?
  • [Blocker] Update the CoC to use/reference the CNCF CoC
    • Currently the Code of Conduct is adapted from the Contributor Covenant, version 1.4, available at http://contributor-covenant.org/version/1/4. However one of the incubation criteria is to explicitly reference and use the CNCF CoC https://github.com/cncf/foundation/blob/main/code-of-conduct.md e.g. “Document agreement that the project will adopt the CNCF Code of Conduct.” You can and are encouraged to have your own CoCC, but the CoC must be based on the CNCF one and you can add on top and detail your own processes.

• Engineering Principles

  • “one thing which requires approval of a Venafi employee - that’s to publish the Helm chart to charts.jetstack.io.” Is there an issue opened for this work?

Other updates:

• Adopter interview: I have 2/3 secured now. Going through the scheduling process for the last one, but don’t have a specific timeline for it yet
• still waiting on TAG Contrib-Strategy review to be complete.

[SgtCoDFish]

Thanks Katie! Responding to the above:

“Most of our decisions today are made by maintainers who’re actively involved in the project” - were there any decisions that were not made by the maintainers in the past?

I'm not sure I've got anything great to point at - I'll have a think - but non-maintainers have certainly contributed to decision making in our meetings. Erik is a good example - he's contributed with a lot of discussion and decision making in our daily standups, e.g. helping decide which approach to take for solving a problem.

(I'm talking about Erik's past contributions here, before he applied to be a maintainer!)

“Our maintainer list currently lists 8 maintainers, with the expectation that at least one more will be onboarded soon” - who is the new onboarded member? Does it refer to Erik: cert-manager/community#32?
“we have at least three people on the path to becoming maintainers.” - ditto - any evidence to support this statement?

Ah that's actually out of date because this issue was raised a while back, sorry!

I was actually talking about Adam at the time I wrote that (cert-manager/community#23) who's now been a maintainer for a while. Adam is at Venafi, but he was onboarded as a maintainer entirely through the public process which is the same one that Erik (who is not affiliated with Venafi) has just cleared in the PR you linked.

We've also offboarded Irbe as a maintainer since then (at her own request because she didn't have the time).

The people on the path to becoming a maintainer at the time I wrote that would've been people who were on the path through the governance process. There's not really a great list for that since it's a bit decentralised, but it would have included:

• Erik (approver on the trust-manager project since Oct 2023 and a reviewer on cert-manager since Feb)
• Houssem (Venafi affiiliated, has been an approver for istio-csr since January)
• Peter (Venafi affiliated - member of the cert-manager org and incredibly helpful with a lot of non-code tasks like debugging, testing, admin, taking notes in meetings, etc)

I'll update that section to reflect the situation today!

[Blocker] Update the CoC to use/reference the CNCF CoC

This was updated today and is already resolved. As I've written elsewhere, we've been using the CNCF CoC in practice for a long while now (it's what we mention at the start of the biweekly meeting) - this was just an oversight.

“one thing which requires approval of a Venafi employee - that’s to publish the Helm chart to charts.jetstack.io.” Is there an issue opened for this work?

Yeah we have a design proposal which will help to address this: cert-manager/cert-manager#7132 (I'll add a link to the issue description)

The plan here is to evolve away from depending on the chart repo, since most people are moving to OCI registries anyway for charts. We'll keep pushing to the existing repository so we don't break anyone but we envision the primary way of consuming project charts to be pulling from OCI registries (which any maintainer should be able to access!)

EDIT 19:58 UTC+1: I've updated the issue description with some of the details I wrote in this comment 👍