[Process improvement]: Make an easy-to-navigate website to find and query KEPs

[lasomethingsomething]

This issue documents the roadmap/plan for a key 1.20 process-related objective.

What Is the Website

• An up-to-date index with links to tracking issues and KEPs
• Displays KEPs similar to how Python displays PEPs, with more detail
• Would be part of the Contributor website

Why Do We Need It

• Helps fulfil needs raised by @jberkus's Make KEPs transparent issue

Who's Needed, Who's Helping, Who's Informed

Needed:

• SIG contributors for Step 1, to update KEP metadata
• Anyone with experience working with Hugo data templates for steps 2-4

Helping:

• @palnabarun has volunteered to work on this
• @sftim will look for “Good First Issues”

Informed:
/sig docs
/sig contributor-experience
/sig architecture

How Do We Get There: Implementation Steps and Notes

• 1. SIGs get their metadata happy/valid, because the site will read from this metadata
• 2. Programmatically generate KEP metadata dump for Hugo (kepify, jq/yq etc)
  • For Netlify/Kepify, maybe an NPM wrapper that then runs Hugo
  • Kepify = Netlify: will either be Go or bash (if a script to produce it)
  • Once the metadata is consistent, we can render it all in a table quite easily
  • We don't have to surface the KEP directly yet, just the metadata and links back to the actual thing
• 3. Automate updating of data on merge (prow job dump, a JSON blob somewhere, etc)
• 4. Integrate data dump into Hugo
  • Hugo can query for data at site build time, including from public HTTP APIs via Hugo Data Templates

Related Action Item: Preparing for 1.21

• @kikisdeliveryservice will survey not-updated KEPs + their enhancement status and use results to assert that updates should be required for inclusion in a 1.21 release via kepval or something else; we'll make this blocking in 1.21
  • We have the most leverage when people want to be in the milestone
  • people are updating at our request now because they want to be in the milestone

Potential Next Iterations/Milestones

• Using the metadata, capture which release a KEP went into, at what stage
  • Making it so that docs contributors no longer have to use git bisect against k/k to work out this kind of detail
• Linking to Feature Gates
  • SIG Docs contributors would like to use this effort as groundwork for a future enhancement: tracking all Feature Gates and having their status available as an API that can be consumed by https://kubernetes.io/; graduating a feature automatically, updating documentation pages that mention it.
• Supports Social media guidelines for Enhancements (holding this until the website is running)

[palnabarun]

/area enhancements

[gkarthiks]

@LappleApple I am interested in this. Please ping me on Slack.

[jberkus]

A site designed after the PEP page will fulfill most of the transparency requirements. The one additional feature we'll eventually need that PEP doesn't have is to have lists of:

• current work tied to each KEP (issues and PRs)
• planned deliverables for the KEP, by release no.

Copying the PEP page would be a good first step, though, and way more accessible than what we have now.

[lasomethingsomething]

Some user stories drafted in June 2020 for KEPs tooling, based on Make KEPs Transparent and misc. conversations:

• As a Kubernetes contributor, I would like to find useful information (including SIG) about current KEPs in an automated and searchable way, so that I can consider contributing to efforts that the community deems valuable.
• As a SIG Lead, I would like to find out in an automated and searchable way what are the current active KEPs in my SIG and their statuses, so that I can track responses to PR reviews after those PRs have been merged and see which KEPs are making/missing release cycle timelines.
• As a new Kubernetes contributor, I would like to know which KEPs in any SIG touch on, or even subsume, a proposed feature, so that I can dedicate my time effectively.
• As a KEP owner or stakeholder, I would like to find (in a searchable and automated way) issues and PRs that are blocking any current KEPs, so that I can move them along.
• As a member of ContribEx, I would like to find the latest KEPs so that I can promote them for review/commentary.
• As a member of ContribEx, I would like to see active KEPs in a website format to make them more transparent and discoverable.

[kbhawkey]

Hi @LappleApple .
I may have some time to pitch in with the templating piece.

[sftim]

OK, here's a mockup of a future https://k8s.dev/resources/enhancements/:

What do we need to render that? The minimum is that for each KEP we have a bit of metadata.
For the mockup the minimum you need is like this:

---
path: "/keps/sig-contributor-experience/1553-issue-triage"
title: Issue Triage Workflow and Automation
kep-number: 1553

or

---
title: Implement maxUnavailable for StatefulSets
path: "/keps/sig-apps/20190226-maxunavailable-for-statefulsets.md"

(you don't need a KEP number - but not having it makes them hard to sort!)

If those files end up in data/generated/enhancements before site build, the mockup adds them to the KEP list.

kubernetes/contributor-site@master...sftim:20201017_kep_list_mockup shows the Hugo & Markdown changes I made for the mockup.

---

How it could work:

• add a new GitHub Action to this repo that generates a single JSON document listing all KEPs
  • I'm thinking to GitHub Pages
  • doesn't have to be perfect, so long as title and path are available
• use Hugo's ability to fetch remote data, so that https://k8s.dev/ fetches that JSON during build
  and uses it to render a KEP list.
• https://k8s.dev/ builds on a timer, so new KEPs get picked up automatically within a few hours
  • extra credit: have a build to this repo trigger a notification to rebuild https://github.com/kubernetes-sigs/contributor-site/

[sftim]

Questions about that mockup?

[mrbobbytables]

I'd go the table formatting route, but we should be able to get at a minimum from the metadata:

• title
• status
• owning-sig

Once the enhancement structure is in place, we can get the kep # (and associated tracking issue).

I'm thinking to GitHub Pages

GitHub pages would require an additional commit any time a kep is updated =/ I think we could provision a gcs bucket and dump the content there as a postsubmit job.

[palnabarun]

@sftim -- that mockup looks great.

I have a few ideas that I was experimenting with related to preparing the data for the website.

One question I had though, do the KEP markdown files need to have certain Hugo metadata in order to be rendered by Hugo during the site generation phae?

I think we could provision a gcs bucket and dump the content there as a postsubmit job.

@mrbobbytables -- Putting the JSON into a GCS bucket sounds great. 👍🏽

I will get back with more updates by next Monday as I don't have free cycles to work on this during the work week. :(

[mrbobbytables]

One question I had though, do the KEP markdown files need to have certain Hugo metadata in order to be rendered by Hugo during the site generation phae?

We don't want to render the keps themselves in hugo (at this point). We do want to provide a list of them from a single page.

As someone that spent many hours trying to fix rendering of them in hugo (the original goal of the contrib site was to list keps). It very much is not a path forward without significant effort in fixing paths, links and more. =/

[palnabarun]

We don't want to render the keps themselves in hugo (at this point). We do want to provide a list of them from a single page.

Okay. That clears the doubt. Thank you! 😃

[sftim]

provision a gcs bucket and dump the content there as a postsubmit job

Something I've thought of doing as a place to dump generated JSON: use the repo wiki (which is actually a Git repo), and have the GitHub Action update a JSON file in that repo every time someone merges to the main branch in the main repo.

Depends what we think about GitHub Actions though!

[kbhawkey]

👀

[kbhawkey]

Hi. I don't know whether you need more help.
Do you have a program to create the Markdown files from the KEP data?
I'd like to see what a list of KEPs looks like. Is there an existing site?

[palnabarun]

Hi everyone 👋, I was experimenting with a change in kepctl to output the list of KEPs in YAML/JSON. My changes are in this tree and I shared a sample output here.

Broadly, the tasks to complete this look like the following:

• Make kepctl query be able to output YAML/JSON, say keps.yaml. (Add YAML and JSON printers to kepctl query #2139)
• Create a GCS bucket (let's say kubernetes-enhancements-artifacts) to store the keps.yaml.
  This bucket should be writeable only by the workload in the next item. It may be read by anyone at the moment since the only thing we are storing at the moment is keps.yaml which is public information.
• Create a Prow postsubmit job for kubernetes/enhancements to generate the keps.yaml after every merge to master and upload the same to the kubernetes-enhancements-artifacts GCS bucket.
• Implement the KEP table in kubernetes-sigs/contributor-site which will get populated at build time by the contents of keps.yaml.
• Create a Prow periodic job for kubernetes-sigs/contributor-site to rebuild the website.

Let me know what do you think of the plan.

[kbhawkey]

Hello @palnabarun .
Do you want to publish/republish the "chart" to the contributor site every time keps.yaml is changed or
could you batch the updates (small changes)?
How often during a release is this data updated? Does the information remain on the contributor site after the release?
Do you plan on showing historical data from previous releases?

[sftim]

BTW I looked again and JSON is going to be easier to work with with YAML, for the Kubernetes contributor website.

[palnabarun]

@kbhawkey

Do you want to publish/republish the "chart" to the contributor site every time keps.yaml is changed or
could you batch the updates (small changes)?

The contributor website can be published periodically.

How often during a release is this data updated?

The information is updated fairly frequently before the Enhancements Freeze kicks in during each release cycle.

Does the information remain on the contributor site after the release?

Yes. The plan was to have the listing permanent.

Do you plan on showing historical data from previous releases?

The scope of the ask right now is to just show all KEPs somewhere easily visible to whoever wants to see a summary. Tracking them through the releases can come in the next iterations.

@sftim

BTW I looked again and JSON is going to be easier to work with with YAML, for the Kubernetes contributor website.

The kepctl query command supports JSON as well. So, you can use the exported JSON to test out things.

[donmeganotz]

Can ⬆️📲

…

On Wed, Nov 25, 2020, 1:10 AM Nabarun Pal ***@***.***> wrote: @kbhawkey <https://github.com/kbhawkey> Do you want to publish/republish the "chart" to the contributor site every time keps.yaml is changed or could you batch the updates (small changes)? The contributor website can be published periodically. How often during a release is this data updated? The information is updated fairly frequently before the Enhancements Freeze kicks in during each release cycle. Does the information remain on the contributor site after the release? Yes. The plan was to have the listing permanent. Do you plan on showing historical data from previous releases? The scope of the ask right now is to just show all KEPs somewhere easily visible to whoever wants to see a summary. Tracking them through the releases can come in the next iterations. @sftim <https://github.com/sftim> BTW I looked again and JSON is going to be easier to work with with YAML, for the Kubernetes contributor website. The kepctl query command supports JSON as well. So, you can use the exported JSON to test out things. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2095 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACR5HAQ33KNCMA7W64YFQJTSRSUVRANCNFSM4SMR7RZQ> .

[lasomethingsomething]

Next steps, based on conversation during Feb 2, 2021 Enhancements meeting:

• Get the data in place
• Confirm we're happy with the written validation
• Have contributors review it
• Handle execution
• Ask for contributors to help

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[sftim]

kubernetes/contributor-site#222 is still in progress

It would be great to rally some help to get this over the line.
/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[palnabarun]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[sftim]

Still worth doing, and there is (slow) progress.
/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[palnabarun]

/remove-lifecycle stale

[palnabarun]

/assign

[palnabarun]

Remaining Tasks:

• Script to generate manifest and push to bucket add script to generate and push KEP manifest to GCS bucket #3442
• Postsubmit for generating the manifest on every PR merge Add postsbumit for generating KEPs manifest test-infra#26928
• Contributor-site page Add page to list all KEPs contributor-site#324

Already There:

• Periodic job to rebuild contributor-site (https://github.com/kubernetes/contributor-site/blob/master/.github/workflows/netlify-periodic-build.yml)

[sftim]

Periodic job to rebuild contributor-site

Have a look at https://github.com/kubernetes/contributor-site/blob/master/.github/workflows/netlify-periodic-build.yml - I think we already have this.

[palnabarun]

Have a look at https://github.com/kubernetes/contributor-site/blob/master/.github/workflows/netlify-periodic-build.yml - I think we already have this.

This is what I exactly wanted. Thanks @sftim for pointing to that!

[palnabarun]

We would need to increase the frequency of that job.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[sftim]

@palnabarun is this now Implemented? Can we close the issue?