New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create component standard working group #3008

Merged
merged 1 commit into from Jan 10, 2019

Conversation

@mtaufen
Copy link
Contributor

mtaufen commented Dec 4, 2018

As discussed in SIG-Architecture and KEP 0032:
https://github.com/kubernetes/enhancements/blob/master/keps/sig-cluster-lifecycle/0032-create-a-k8s-io-component-repo.md

@luxas @sttts there are still a few TODO items, like setting our regular meeting time and agreeing on a name that isn't just the word "component."

I'll hold off on running make generate and emailing stakeholder SIGS/steering committe with a link to this PR until we settle the name.

@luxas

This comment has been minimized.

Copy link
Member

luxas commented Dec 8, 2018

@mtaufen mtaufen changed the title Create component working group Create component base working group Dec 18, 2018

@mtaufen mtaufen changed the title Create component base working group Create component standard working group Dec 18, 2018

@mtaufen mtaufen force-pushed the mtaufen:wg-component branch 2 times, most recently from 5c3b635 to 3e311b9 Dec 18, 2018

@mtaufen

This comment has been minimized.

Copy link
Contributor Author

mtaufen commented Dec 18, 2018

/hold
for stakeholders and steering approval

/sig architecture
/committee steering

@luxas

luxas approved these changes Dec 18, 2018

Copy link
Member

luxas left a comment

Thanks!

LGTM (might wanna add slack when created tho)

* Michael Taufen (**[@mtaufen](https://github.com/mtaufen)**), Google

## Contact
* [Slack](https://kubernetes.slack.com/messages/)

This comment has been minimized.

@luxas

luxas Dec 18, 2018

Member

TODO for slack admins?

Develop a standard foundation (philosophy and libraries) for core Kubernetes components to build on top of.
Areas to standardize include configuration (flags, ComponentConfig APIs, ...), status endpoints (healthz, configz, ...), integration points (delegated authn/z, ...), and logging.
Details are outlined in KEP 0032: https://github.com/kubernetes/enhancements/blob/master/keps/sig-cluster-lifecycle/0032-create-a-k8s-io-component-repo.md.
charter_link:

This comment has been minimized.

@luxas

luxas Dec 18, 2018

Member

Do we need this as a WG?

This comment has been minimized.

@luxas

luxas Jan 10, 2019

Member

you could also omit this from the YAML to not confuse anybody

@pwittrock

This comment has been minimized.

Copy link
Member

pwittrock commented Dec 18, 2018

@luxas @mtaufen

Couple questions:

  • Working groups don't own code. Instead code is owned by subprojects of SIGs. Where (which SIg / subprojects) will the development be done out of?
  • Working groups are designed to facilitate cross-SIG work with a clear beginning and end (after which the WG is disbanded). Does this initiative have a clear end state?

Do we need this and a subproject for shared kubernetes infrastructure? We also have a number of k8s libraries for things like sets and retries. Do you envision these as being part of that initiative?

@mtaufen

This comment has been minimized.

Copy link
Contributor Author

mtaufen commented Dec 18, 2018

@mtaufen

This comment has been minimized.

Copy link
Contributor Author

mtaufen commented Dec 18, 2018

Offline chat with @pwittrock:

  • Main concerns are ownership and clear end state.
  • We should definitely find a SIG to own this repo and the building blocks we create. @luxas I saw you created the KEP under sig-cluster-lifecycle, but since this is about APIs, maybe sig-apimachinery is a better place since a lot of this relates to APIs?
  • Regarding end state:
    • The main reason to have a WG is that this discussion cuts across many sigs and we need a forum for that; sig-architecture is too high level to get into the nitty gritty details of how to implement these building blocks.
    • This should be a working-group at least until we can move the common code to a common place; at that point we can revisit the structure and decide how to collaborate going forward.
@pwittrock

This comment has been minimized.

@mtaufen

This comment has been minimized.

Copy link
Contributor Author

mtaufen commented Dec 18, 2018

What is the difference between server-frameworks and server-sdk?

github: mtaufen
company: Google
meetings:
- description: Regular WG Meeting

This comment has been minimized.

@luxas

luxas Dec 19, 2018

Member

We should add zoom link as soon as we create it as well

@luxas

This comment has been minimized.

Copy link
Member

luxas commented Dec 19, 2018

@pwittrock I'm happy to put this under server-frameworks. In large, this is a code split of the k8s.io/apiserver repo, along with newly-extracted code from k8s.io/kubernetes. This isn't an sdk, so it doesn't belong in server-sdk.

The main reason to have a WG is that this discussion cuts across many sigs and we need a forum for that; sig-architecture is too high level to get into the nitty gritty details of how to implement these building blocks.
This should be a working-group at least until we can move the common code to a common place; at that point we can revisit the structure and decide how to collaborate going forward.

Agree on these. I'd also want to fill in that the end state is that our components behave the way we want them to (the WG will create a dedicated KEP for what standards we should have for our core components), and the code that made those standards happen are conveniently refactored into k8s.io/component-base

@luxas I saw you created the KEP under sig-cluster-lifecycle, but since this is about APIs, maybe sig-apimachinery is a better place since a lot of this relates to APIs?

Yeah, just because I have authority to put things there and by the time I created the KEP initially I hadn't got SIG API Machinery/Arch approval (which we now have). Happy to move the KEP around, but think it could make sense to put this in a dedicated wg-component-standard folder as well...

For me, as said, I'm happy to put the code under server-frameworks, i.e. API Machinery.
That said, SIG Cluster Lifecycle will be part of this effort in various ways still.

Do you envision these as being part of that initiative?

We envision this playing well together with all the other initiatives.

@mtaufen

This comment has been minimized.

Copy link
Contributor Author

mtaufen commented Dec 20, 2018

I'd also want to fill in that the end state is that our components behave the way we want them to

I think @pwittrock's concern is that this is a moving target, potentially a perpetual motion machine :).
But for now a WG works for our purposes, and if it turns out this should be a longer lasting structure, we can revisit that later.

@mattfarina

This comment has been minimized.

Copy link
Member

mattfarina commented Dec 20, 2018

I still have a few open questions...

  1. Who will own the repo k8s.io/component-base? A WG cannot own the code. If it is SIG API Machinery, shouldn't their ownership of the subproject be updated as well?
  2. A working group will help with cross SIG collaboration. What SIGs are planned on being apart of this? I ask to make sure they are looped in, announcements made at those SIGs, report out (using the new templates) a the community meeting for those SIGs should talk about it, etc.
  3. SIG Architecture has a new subproject for code organization. How does this relate to that?
  4. Where will any conventions be documented for contributors and who will own that (A WG can't own these things)?

These are all logistical concerns for pulling this off effectively within the structure we have.

@pwittrock

This comment has been minimized.

Copy link
Member

pwittrock commented Dec 20, 2018

I don't intend to block this. It is easy to unmake a WG, and I agree with the goals and execution. I do want to probe a bit deeper - since the code will be done in a subproject of apimachinery anyway (proposed). Why not just do the work out of the subproject. It is fine for individuals to be working in multiple SIGs and for sub projects to have the collaboration tools of WG's. The server-sdk project that owns kubebuilder is owned by @droot, @pwittrock (sig cli tl) and @DirectXMan12 (sig autoscaling).

Put another which SIGs should own the repos under development, and what value does the WG + subproject provide over just the subproject itself? I assume it is to get buy in for where the code is being moved from and that a WG facilitates this in some way?

FWIW, If you just go the subproject route, you don't need steering committee approval, you just need AM approval for the subproject.

@mtaufen

This comment has been minimized.

Copy link
Contributor Author

mtaufen commented Dec 20, 2018

@mattfarina

Who will own the repo k8s.io/component-base? A WG cannot own the code. If it is SIG API Machinery, shouldn't their ownership of the subproject be updated as well?

Yeah. It sounds like folks are leaning towards putting this in a SIG API Machinery sub-project. That could work, but I also want to make sure we have a public way of organizing this effort (regular open meetings, notes, etc.), because it broadly affects the project, and SIGs and WGs seem like the only two "official" ways to do this right now.

A working group will help with cross SIG collaboration. What SIGs are planned on being apart of this? I ask to make sure they are looped in, announcements made at those SIGs, report out (using the new templates) a the community meeting for those SIGs should talk about it, etc.

I think this depends on each item. For example, ComponentConfig probably involves a mix of SIG Cluster Lifecycle and SIG API Machinery. Standardizing delegated auth implementation obviously involves SIG Auth. Standardizing endpoints like /configz, /healthz, etc. probably involves SIG Instrumentation. Our current plan is to write a KEP for each sub-area, and identify the relevant SIGs in those KEPs.

Thinking about this, maybe we want to separate "overall repo ownership" from ownership of the various pieces - which might align more directly with the involved SIGs for each sub-area.

SIG Architecture has a new subproject for code organization. How does this relate to that?
Where will any conventions be documented for contributors and who will own that (A WG can't own these things)?

That's an interesting idea. Based on the description "Overall code organization, including github repositories and branching methodology, top-level and pkg OWNERS of kubernetes/kubernetes, vendoring," I'm not sure if it's a good fit, since our effort involves standardizing the implementation of various building-blocks, whereas the code organization subproject sounds higher-level (maybe it's not?).

These are all logistical concerns for pulling this off effectively within the structure we have.

Yeah, totally agree these need to get figured out. Thanks for your clear list :)

@pwittrock

I assume it is to get buy in for where the code is being moved from and that a WG facilitates this in some way?

Buy-in and transparent discussions, since it has a broad impact. Maybe KEPs are already good enough for that?

I'm definitely not against a sub-project, as there is less administrative work involved :P. But I want this work to be open and to our community's standard for collaboration.

@pwittrock

This comment has been minimized.

Copy link
Member

pwittrock commented Dec 20, 2018

Following up on a few points of clarification.

For collaboration:
In addition to a repo, you can get a Slack Channel, Email Discussion Group, Zoom Channel, Calendar Meeting, etc as a subproject. Kubebuilder has a Slack channel that is relatively active and we have found to be a great way to collaborate. It also has a email group that we used to setup a collocated BoF at Kubecon North America 2018.

A WG may be great if you have a large group of individuals that need to be informed, and a smaller group of individuals that are responsible. The WG would give you a way to broadcast updates, but keep the subproject channels more focussed. If the group of folks responsible is effectively the same as the group to be informed, the WG would probably just be the same folks within the subproject.

Regardless of if you have a WG or not, you will need a subproject to how the code and don't need to block on getting a WG to get a subproject. You will need to get the support of the SIG TL's on the proposal KEP (e.g. @deads2k and @lavalamp for AM), and want to present it at a SIG meeting to be able to field questions.

@mtaufen

This comment has been minimized.

Copy link
Contributor Author

mtaufen commented Jan 2, 2019

@luxas @sttts @deads2k @lavalamp what do you guys think?
Shall we focus on making this a subproject of apimachinery for the time being?

@sttts

This comment has been minimized.

Copy link
Contributor

sttts commented Jan 3, 2019

Shall we focus on making this a subproject of apimachinery for the time being?

If it helps to move forward, why not.

I have my doubts though whether a subproject is the right tool, following @pwittrock characterization above. It's not about having a slack channels, email groups or zoom channels in the first place, but merely about refactoring existing code into a central place. If the efforts turn into a project of its own at some point, that's fine and we can create a formal subproject out of it. But for now in my view we have these goals:

  1. move (existing) code to a central place (component-base repo) from places where they don't belong (mostly k8s.io/apiserver+apimachinery, some code in kubelet)
  2. refactor that code to make it more reusable and refactor k/k components to use that shared code

Then, with a far far smaller priority:

  1. make that code work for other projects in the ecosystem.

In other words, if Golang allowed to restrict the use of a repository to k/k components only, we would do that. Unfortunately it doesn't. Does this mean we should make this effort bigger than goal 1 and 2 for now, with design docs, a subproject, slack channel? IMO we shouldn't.

Let's get 1 and 2 done quickly (for 1.14+15), explicitly scope it down to those two, not the 3rd. And then later it should be feasible to turn the result into something bigger for wider adoption, possibly with further, larger refactorings, documentation, slack channel etc.

To avoid confusion and to avoid premature adoption, maybe we should reflect that in the repo or package names for now (I like the "experiment" term in Golang), or at the very least big and bold in the README.md. Until now it was enough to hide such code in k/k itself and claiming "we don't support it".

We need a sig as an owner for our new component-base repo (sig-arch, sig-api-machinery would both work IMO). But that's a different question. The work is cross-sig, through-out the whole code base. In that sense a working group probably is the right tool.

@sttts

This comment has been minimized.

Copy link
Contributor

sttts commented Jan 4, 2019

I have chatted with @luxas and @mtaufen about the "experimental" package idea I mentioned above to avoid the tension between the goals of

  1. building something for the wider ecosystem to consume
  2. versus moving fast with refactoring the vast amount of code we have out into component-base and re-using it in our in-tree k/k component.

The former will certainly require follow-up KEPs for some of the work (further generalization of componentconfigs comes to mind). The later should improve code quality in 1.14 and 1.15 and would make componentconfig implementation inside k/k feasible in the first place. The later is also about learning what we need in order to reach the first goal.

In order to allow rapid refactoring for the second goal, we propose to

  1. create the component-base repository as soon as possible
  2. but move unfinished, factored out code into k8s.io/component-base/experimental/<sub-package> first.

We will then promote packages to the top-level k8s.io/component-base/ when we think they are ready for public consumption.

We will try to minimize the use of those experimental packages in k8s.io/apiserver (which is already used by 3rdparties) or at best avoid it completely. On the other hand, kube-controller-manager, kube-scheduler, kubelet, kube-apiserver are all in-tree completely and can serve as test beds while the new code becomes mature and ready for promotion.

That having said, the working group should serve goal 2 and prepare for goal 1. @pwittrock does that match your conception of WG vs. sub-project?

@luxas

This comment has been minimized.

Copy link
Member

luxas commented Jan 7, 2019

Hi all, are there still outstanding issues/comments/concerns or can we proceed with the WG creation?

Simply put, we want WG + API Machinery subproject because:
a) The WG will work on cross-cutting issues wrt Component Standards (exactly as the name suggests), and will focus on the "docs-first" approach and develop KEPs wrt how a k8s component should work, as well as how to graduate existing (or non-existent to date) ComponentConfig schemas to beta and higher, in collaboration with SIG Architecture and others.
b) The k8s.io/component-base subproject will focus on moving code out from code in the proposed and approved fashion ("code-first", if you like). The WG will help keep the subproject on the right track, and involve the right stakeholders when more complex changes are made (essentially help with communication to the rest of the community)

If this makes sense, can we please proceed with this? We intend all the best for the project and if it later turns out this way of working wasn't the ideal one, it is easy to unmake the WG later, as you said.

@pwittrock @mattfarina @thockin does this make sense?

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Jan 7, 2019

@luxas

This comment has been minimized.

Copy link
Member

luxas commented Jan 8, 2019

@mtaufen when updating this with the final feedback, please include k8s.io/component-base as a AM subproject as per kubernetes/org#330 (comment)

@mtaufen mtaufen force-pushed the mtaufen:wg-component branch from fea66f2 to 96387cc Jan 8, 2019

@mtaufen

This comment has been minimized.

Copy link
Contributor Author

mtaufen commented Jan 9, 2019

Chatted w/ @pwittrock offline, he's fine with a WG too, just didn't want us to block on it.

@mtaufen

This comment has been minimized.

Copy link
Contributor Author

mtaufen commented Jan 9, 2019

@luxas @sttts I updated this to add the component-base subproject to sig-api-machinery.

Are there any additional updates folks would like to see before we merge this?
Are we set on Tuesday at 8:30AM PST as the meeting time?

@luxas

luxas approved these changes Jan 10, 2019

Copy link
Member

luxas left a comment

LGTM :)

Left some comments as follow-ups once we have all the other communication machinery set up

frequency: weekly
url: https://docs.google.com/document/d/18TsodX0fqQgViQ7HHUTAhiAwkf6bNhPXH4vNVTI7GwI
contact:
mailing_list: https://groups.google.com/forum/#!forum/kubernetes-wg-component-standard

This comment has been minimized.

@luxas

luxas Jan 10, 2019

Member

Follow-ups: add slack (#wg-component-standard, once created), and teams (@kubernetes/wg-component-standard, once created)

Develop a standard foundation (philosophy and libraries) for core Kubernetes components to build on top of.
Areas to standardize include configuration (flags, ComponentConfig APIs, ...), status endpoints (healthz, configz, ...), integration points (delegated authn/z, ...), and logging.
Details are outlined in KEP 0032: https://github.com/kubernetes/enhancements/blob/master/keps/sig-cluster-lifecycle/0032-create-a-k8s-io-component-repo.md.
charter_link:

This comment has been minimized.

@luxas

luxas Jan 10, 2019

Member

you could also omit this from the YAML to not confuse anybody

@@ -2340,3 +2344,31 @@ workinggroups:
contact:
slack: wg-security-audit
mailing_list: https://groups.google.com/forum/#!forum/kubernetes-wg-audit
- name: Component Standard
dir: wg-component-standard

This comment has been minimized.

@luxas

luxas Jan 10, 2019

Member

consider also adding label: wg/component-standard here. can be followed up

@luxas

This comment has been minimized.

Copy link
Member

luxas commented Jan 10, 2019

Are we set on Tuesday at 8:30AM PST as the meeting time?

Yep, I think so

Are there any additional updates folks would like to see before we merge this?

I left comments but please consider them non-blocking. I think we can merge now.

/hold cancel
/lgtm

Please hit the /approve button @thockin @bgrant0607 or @jdumars :). Thanks!

@jdumars

This comment has been minimized.

Copy link
Member

jdumars commented Jan 10, 2019

/approve

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Jan 10, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jdumars, luxas, mtaufen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@dims

This comment has been minimized.

Copy link
Member

dims commented Jan 10, 2019

LGTM 👍

@k8s-ci-robot k8s-ci-robot merged commit e060786 into kubernetes:master Jan 10, 2019

3 checks passed

cla/linuxfoundation mtaufen authorized
Details
pull-community-verify Job succeeded.
Details
tide In merge pool.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment