Creating a Nodeless working group #2215
Conversation
k8s-ci-robot
added
size/M
|
/hold |
k8s-ci-robot
added
the
do-not-merge/hold
|
We took this to sig-arch, and there's a good amount of interest on the topic for creating a nodeless kubernetes design. |
|
@rbitia if you have any links to the discussions, that would be helpful. @bgrant0607 @jdumars Can you comment from the sig-arch side? |
|
@cblecker SIG Architecture hasn't agreed that we should form a working group now. We have higher-priority issues we need to apply more effort to, such as developing an API review process and identifying features that need to be covered by conformance tests. The project in general and SIG Architecture in particular don't have enough people chopping wood and carrying water. It doesn't make sense to launch a huge new initiative when we have critical fundamentals missing, such as test coverage of most pod features. We also do not agree on the focus of the working group. |
|
And it doesn't make sense to undertake a new kubelet implementation while we don't have remotely sufficient tests to ensure compatibility. Please work with SIG Node, SIG Network, SIG Storage, SIG Arch., and SIG Testing on plugging the egregious gaps in pod functionality/behavior coverage. |
|
The working group construct is designed to accommodate efforts that require coordination across multiple SIGs (with a primary SIG sponsor) so a defined problem set can be solved and the group dissolved. I don't feel the deliverables of the working group are clearly defined, nor is the condition of dissolution. Since it appears the primary driver of the group is discussing possible implementation strategies, and better defining the work, I think an informal "birds of a feather" structure would make much more sense and require less commitment of scarce time and resources. |
|
It was directly asked at the end of sig architecture when we discussed this if there were objections, none were raised. Lazy consensus indicates that this is approved. We don't need to take any additional time from sig architecture to consider this. I don't think your objections have merit in this case. |
|
And specifically the outcome of this working group is the design proposal not the implementation, so your claims about reimplementing the kubelet are also without merit. |
|
And finally you can not lay claims on people's time. If people are interested in working on something, it is not yours (not anyone's) to judge the relative priority. That's open source. |
|
A part of Brian's concern that I'll echo is that there's no promise or guarantee of this becoming part of Kubernetes (an issue not resolved in the architecture call). A statement like "we have a working group designing a way to run Kubernetes without nodes" wouldn't be accurate, while a statement like "the working group, having encountered a lot of skepticism from the impacted SIGs who also pushed back on the need to do this now, is deeply committed to overcoming the suggested concerns". But that's where the time issue comes in to play - that puts a burden back on the people who raised those concerns to continue to be involved. I'm ambivalent on what we actually call the effort. I am usually not against forming working groups to explore areas that matter to contributors. But I think it's really, really important to clarify that the bar on this particular WG is high in terms of up front design work in order to conserve that time. |
|
There's a balance to be found here that requires clear terms of reference. I would agree that "running Kubernetes without nodes" suggests major architectural impact that I'm not sure is warranted. Kubernetes will always need nodes. The question is - what is the business value in providing a consumption model that abstracts the consumer away from the nodes in a highly multi-tenant CaaS world and what is the right manifestation of that? I would suggest that the WG should start by reaching agreement on the problem domain, the use-cases and the high-level business value. There were clearly different approaches mooted in the sig-architecture meeting that can and should be explored - in particular the potential overlap with other efforts. The goal should be to reach consensus about what nodeless is, what it buys us and what we should do about it. |
|
I agree with Ben. @jdumars and @bgrant0607 do you also agree that the following can be the goal for the nodeless working group?
|
|
I have a suggestion on pharasing. What if we change the following. to something like this, Maybe the sentence can be worded more elegantly, but the idea is that one of the working group's charter is to explore various alternatives. This should reduce any time commitments needed to think about nodeless from folks not directly involved in the working group... until we have a final proposal that includes the pros/cons of various designs and justification on why the final design is the best one. @rbitia's goals is just to create a working group under kubernetes that would allow participants to come, discuss, and contribute. For some potential participants holding out on participating, having a standards committee under an industry umbrella helps allay any fears of vendor lock in or proprietary, non-standard solutions. Does this help reduce the fears that some in this thread may have about time commitments? The working group will take on the time commitment and come back with a proposal once it has explored all alternatives and a final solution. |
|
Thank you all for the comments. I think my concerns about a defined scope/terminus have been concretely answered, and it would be great to just document that explicitly in the WG doc. I apologize that this process is still inchoate, and hopefully we can do a much better job in the future. Architecture is the place where passions and ideals intersect with implementation, and that presents an enhanced challenge for us to work even more collaboratively. I would encourage the WG to be as inclusive as possible since the implementation will depend on consensus building widely across the project. It would also be nice to have some regular cadence (monthly?) of updates to SIG Architecture's meetings to discuss progress and identify places where the SIG can help. |
|
Yes that sounds wonderful, Jaice! I need to update the wg mission statement and we can develop a monthly cadence with sig-arch. Appreciate all the help. |
|
Okay, so I went back and reviewed notes from the agenda and listened to the sig-arch meeting (5/24). There was varying opinions on this topic, but near the end of the call (56:31), @brendandburns asked for closure on the topic, and requested a working group be formed, not for Virtual Kubelet, but for addressing Nodeless topics. @bgrant0607 and I believe @quinton-hoole (trying to recognize voices is difficult without cameras on the recording) noted that scope and output of the working group would be important. Then @brendandburns and @rbitia were asked to come back with a charter/proposal for the new working group, and agreed to that. I don't know if that next step happened, or where (I don't see it on the k-sig-arch mailing list). If it hasn't happened yet, that scope/proposal/charter sounds like the next step. From what I can gather from the notes and call, consensus from sig-arch was dependant on that scope/proposal being documented first. @bgrant0607 @jdumars @brendandburns @rbitia: Does this sound reasonable? |
|
Hey so I did send out a proposal to sig-arch's mailing list but here it is again: https://drive.google.com/open?id=1Y1GEKOIB1u5P06YeQJYl9WVaUqxrq3fO8GZ7K6MUGms |
|
Okay, I missed that.. I re-searched it was sent out as a part of @jessfraz's thread here: https://groups.google.com/d/msg/kubernetes-sig-architecture/y7eGXxX9-lo/CeprNlOIAQAJ My next question is: should this document then get PR'd into |
|
Sure, yes I can add that in too! |
|
What if another goal of the working group was to come up with a name that doesn't end with "less"
|
|
The request from sig-architecture was to submit a document clarifying the
goals, non-goals (i.e. scope) and outputs of the working group.
The current document does not address these requirement, as evidenced by
many of the comments on the doc.
I think that the document needs to be re-aligned with the above goals
before being presented to sig-arch for approval.
In case it helps, here are a few example proposals by other working groups
that might help to give you a flavor of what we're after.
==============
CNCF Storage working group:
Phase 1:
Goals:
1. Clarify the terminology currently in use in the storage space, and
the relationships between the various terms. Essentially a taxonomy of the
storage landscape.
2. This to include anything reasonably within scope of “storage”,
including block stores, key value stores, databases, object stores,
volumes, file systems etc.
3. Provide some general information as to how these things are currently
being used in production in public or private cloud environments.
4. Compare and contrast the various technology areas w.r.t. the primary
properties of availability, scalability, consistency, durability,
performance, API, etc.
Non-goals:
a. Define what’s in-scope and out of scope for the CNCF.
b. Provide any recommendations regarding preferred storage approaches or
solutions.
During phase 2 we plan to tackle the non-goals, on the basis of phase 1,
specifically in light of understood production use, and comparisons w.r.t.
primary properties.
============
CNCF IoT/Edge working group:
Goals
1. Provide reference architectures for various IoT/Edge environments.
2. Create and maintain conformance tests tailored towards performance
and reliability requirements of the most popular IoT/Edge use cases.
3. Build end-to-end PoCs to validate overall design and provide examples
to system integrators.
4. Evaluate and possibly extend k8s federation and network
infrastructure to better suite IoT/Edge use cases over bandwidth
constrained and unreliable WAN interconnects.
5. Evaluate and possibly improve connectivity and data ingestion options
to better support various field protocols.
6. Evaluate and extend existing CLI tools to manage k8s clusters running
in remote edge locations.
===========
…On Thu, Jun 7, 2018 at 1:26 PM, Jess Frazelle ***@***.***> wrote:
What if another goal of the working group was to come up with a name that
doesn't end with "less"
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2215 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AJ6NAaCZ6Ij2hVN6_dM9yjSiGjqXjUFQks5t6YyMgaJpZM4UZ7OW>
.
--
Quinton Hoole
quinton@hoole.biz
|
|
@quinton-hoole Yes, thanks! I need to update the doc with those. Will send out a revision shortly. |
|
@rbitia: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
| 2. Design a user experience around nodeless | ||
| 3. Create reference architectures for a nodeless Kubernetes and define clear behaviors for how a nodeless Kubernetes should act | ||
|
|
||
| Phase 2 |
There was a problem hiding this comment.
Can this be omitted since it presupposes the outcome of phase one? If the goal is inquiry into a solution, that should be the deliverable. Phase 2 cannot be accomplished by a working group. The design from phase one will require consensus building in the affected sub-projects OWNERS in order to get those code changes in (if applicable). WGs do not own code, do not have any vested authority, and are intended to problem solve cross-SIG concerns. The problem is undefined until the work of phase one is complete by design of the WG itself.
There was a problem hiding this comment.
@jdumars are you asking us to delete phase two?
There was a problem hiding this comment.
I like the new summary sentence Ria.
The output of the working group should be documentation that explores the domain and comes to detailed recommendations about the alternatives that present themselves. In order to do that, the group will need to:
- define the problem space
- examine and document user stories in terms of business value
- explain the limitations of the status quo Eg. Why is namespace support not enough?
- state clear assumptions about existing CaaS APIs and what limitations that puts on a design
- investigate the possible solutions, propose potential designs and speculate on the impact
In that respect, I think (5) is important to keep in scope because any analysis of approaches needs to account for limitations and user impact. (7) matters because a solution can't be credible unless all assumptions are made explicit, but could be rephrased as "document assumptions" rather than a detailed API definition. (4) and (6) seem unnecessary, because I would expect that any upstream changes or compliance questions would be dependent on the solution being accepted.
There was a problem hiding this comment.
I actually don't mind this section. My reading is that "propose" != "implement". Proposals from a WG are fine.
I would drop 7, though. Kubernetes is concerned with the UX and API - any platform that can meet those needs is fine. It's not our place to define what providers need to do.
|
Ok I’ve thought about this a lot and there is something about this that is just not right, but I would like to discuss if that’s okay. And it’s not the name I was totally joking you can call it whatever you want. There doesn’t have to be a working group for everything right and wasn’t this just fine being it’s own community project as virtual-kubelet. I completely agree with you Brendan it’s open source so people should work on what they want. Well then why aren’t they working on virtual kubelet. What’s going to change by it being a working group? More visibility? Wouldn’t having virtual kubelet meetings be the same thing? Or is it that the kubernetes project has a wider audience and so making it a working group would allow more people to potentially join? Also it was my impression that this was working for customers as virtual kubelet right? So why reinvent it again in a different way... just a bit confused I guess Is the fear that someone else will come and do this in a different way and k8s won’t be first? That happens with everything. Look at skydns then coredns... openvz then lxc then docker etc... like this always happens so there's not really much to fear right... innovation is good and if people want to do things like Brendan said... then they will, that’s the best part about open source. But no one outside of the people who already do things w virtual kubelet are wanting this in k8s so that’s making me a little :\ Anyways please feel free to share the multitenancy working group as you see fit for this. We are super chill and if there's not something to discuss we just cancel til the next time so you can have our stage if you want it :) |
|
So a couple of issues I see right now. First off this process is way too difficult. I've been going in circles for months, and unfortunately it's not an exaggeration. I wish it was. This is JUST a working group and as far as I know it's a group of people who want to work within the K8s community to talk about different interesting concepts, and that's all nodeless is. The readme says that as long as we have interest we should be allowed to create it. If you want users of kubernetes to overflow this PR with support we could do that but that's not actually going to help the PR move forward. We have enough interest and rather than figuring this out on our own we thought working with sig-arch and relating members was the correct way to go about creating a nodeless concept. Jess, I totally agree we should start wrapping VK into multi-tenancy, so we can start that immediately :) That being said I have the upmost respect for everyone on this PR and I appreciate your inputs so much. I just need clarity on what needs to happen to get the space we want. I don't want to continue to waste your time or mine. I think this PR shows a larger problem in the wg structure, which I will help to improve with @jdumars. :) |
|
I wasn't saying bring VK into multi-tenancy. I was saying bring the technical discussion of making a "node interface" to multi-tenancy. There are overlaps and we are more than happy to discuss ways to do this or if it is even worth it. |
|
No one is saying you don't have the freedom to do this. It's open source if you are passionate enough about something anything is possible. The back and forth on this PR is not worth anyone's time. Time that we could be spending actually talking about the technical details about what this entails (which are non-trivial). I'm saying skip this mess and just come to a place that already exists and has deeply technical architecture discussions. Our stage is also yours, we are more than willing to share. |
|
I might ban the word "nodeless" tho but I think we can come up with a better name :P |
|
YMMV but what I would personally do here if this was my thing is I'd come up with some designs (and you are more than welcome to brainstorm with multi-tenancy about this), go propose the designs and discuss with sig-node and Dawn Chen (who is awesome by the way :) but you definitely need sig-node here as this touches the kubelet profusely, then open the proposal, then code it. And boom. Done. I'm unsure where the time spent discussing "nodeless" the concept and a working group comes into play in those steps. |
|
My super blunt take on this PR is that the reviewers are interested in exploring all routes to "nodeless" k8s, but the author wants to propose a specific solution and isn't interested in the exploration (that's not a dig--no shame in owning that, IMO). From having actually been at the SIG Arch meeting where this was discussed I think the reviewers here are more or less asking for what SIG Arch agreed on. If @rbitia and crew believe the exploration/requirements gathering phase done or not necessary, then I don't really see a working group being useful or productive; maybe it's time instead to make KEPs for architecture / API / implementation changes. However, I expect that to go even more poorly than this PR, given that there's no agreed on set of requirements. As multiple people have pointed out, in OSS one can't control what other people do with their time. That is a very sharp double-edged sword. If one wants the community to come along for the ride, everyone has to go together through the steps where we honestly examine the needs of users, formulate design alternatives, and choose the best one(s). It is 100% fine if folks don't want to do that and just want to build their tech out. But it is also fine for projects not to merge PRs constructed that way. Sorry for being super blunt here but maybe this will go a little better if we end the proxy war that seems to be happening in the comments. [edit: and just to be 100% clear, this is 100% my own personal opinion and I'm not speaking on behalf of any sig or company] |
|
I'm not feeling very good about this being positioned as SIG-Architecture stifling discussion or innovation. On the contrary, we're asking for visibility of design that can be broadly discussed and agreed upon, which is exactly the collegial spirit that open source is built on. As a community, our highest duty is to preserve the trust of those who choose our project to power their innovation and success. And, transparency is table stakes for that to remain true. The concern that keeps coming up over and over here is that there appears to be an agenda behind the scenes driving this effort. That inherently creates an allergic reaction in the community, especially among those who have poured countless hours of blood, sweat, and tears into this project over the course of years. Yes, the guidance around working agreements needs attention, and we'll get that sorted. But that's not really the core issue here. I don't see anyone saying "don't do this" - I see everyone saying "start with a community-reviewed design, and then talk about what construction might look like". We're offering this advice because we want new ideas and thinking to be successful. It just can't come at the cost of transparency, end-user trust, or expediency. |
|
We've been very transparent on why we want this and we are bringing along anyone who wants to join us and been working as a community from the start. Pods as a service like platforms have been around for some time, from Hyper.sh and VMWare (VIC). Now Microsoft and Amazon have joined the effort to abstract above the VM layer. This isn't new. As our own community we've been very explicit for why we want this and we've gone ahead and done a lot of the initial work for designs, we believe implementations could change. We are 100% open to that but we do need to extend K8s to include containers as a service platforms, we've been transparent to include it as a goal. We aren't hiding anything but if we can't agree that containers as a service things are thing then the creation of this group isn't going to be helpful :/ Here's some context for what we believe: https://thenewstack.io/the-future-of-kubernetes-is-serverless/ We aren't hiding anything and we as a group want to explore the space, and the implementation is completely up for discussion. I hope this makes sense. |
|
I signed up to help Ria facilitate this working group and I hear people's frustrations, so let me try to find some common ground here. Community concerns about Virtual Kubelet were legitimate and I think we all agree Ria did the right thing in coming forward to start a wider conversation. We can argue about the right forum for that conversation, but until we have consensus about the scope of the problem domain, we need to be open about where any effort may ultimately belong or what its impact might be. What matters here (to quote Daniel above) is, "If one wants the community to come along for the ride, everyone has to go together through the steps where we honestly examine the needs of users, formulate design alternatives, and choose the best one(s)". I think that's what everyone is looking forward to doing here. Where I think we have to be careful is in pre-judging motives. I understand there's politics. But if the output of the working group or even the interim feedback showed implicit bias towards a solution or corporate agenda, fair enough - that should be called out. However, this is still just a conversation about a conversation. Ria has taken early initiative to solve customer problems, she's listened to feedback and she's come to the table. Let's judge the effort so far on that basis. I think we can have success if we ensure that the WG has balanced representation, finish the task of polishing the scope in an accountable manner and begin a collaborative technical discussion. We started it here and for the credibility of the effort, I think it would be better if we continue it here. |
|
@jessfraz If the only option on the table was CaaS integration via the Kubelet interface, then sig-node would be the right forum and they're where the initial VK effort was aligned to. I think what folks are asking for here is an exploration of whether there are alternatives that keep the existing node model and its assumptions completely intact, but still allows for a consumption model that presents virtual compute in the abstract. This is where I think there's potentially significant overlap with the multi-tenancy work. |
I think this is at the heart of some of the discontent. I'm 100% open to understanding and adapting to new platforms, but this is such a substantially different platform that it really demands that we step back and contemplate it from many angles. I know there's working code, but many people have raised concerns with the design. If we're going to have a productive discussion, we need to start with no presuppositions. I do not presuppose that "CaaS platforms" is the correct way to position the evolutionary driver here. CaaS platforms are an example of a larger problem, which is what we should be discussing. Looking at the doc, it seems many edits have gone in. I apologize for not keeping up. I have assigned thsi to myself so that it actually comes through my email filters (not because I own the issue). I'll do another read-through right now. |
You are not required to be perfectly aligned with our wg to speak there. We are very open welcoming people. We also have times we just cancel the meetings since no one has anything to talk about which means you have so much space to discuss all this. What I don't understand is the strong desire from those involved here to play politics on this thread and making a new buzz word. When you want to execute and design software our door is open. It's always been open. You don't need a working group to do that :) And look, if you really want the responsibility of leading a working group, please lead ours. I absolutely despise doing the planning tasks. I'D LOVE THAT |
k8s-ci-robot
added
the
needs-rebase
|
@rbitia: PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
k8s-ci-robot
added
the
lifecycle/stale
|
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
k8s-ci-robot
added
lifecycle/rotten
|
Rotten issues close after 30d of inactivity. Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
|
@fejta-bot: Closed this PR. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
15 participants
No description provided.