Submit k3s for inclusion to CNCF as a sandbox project #447
Conversation
cjellick
changed the title
|
+1 for one of the coolest and reliable kuberneres distribution |
|
I don't usually +1 any PRs via comment, but definitely +1 here. Happy to see k3s be a part of CNCF in this way (would bring some light to 2020). |
caniszczyk
added
new project
|
This sets a very weird precedent IMO, because k3s is essentially a distribution, where several of the patches could be pushed into upstream k/k. Where would this end, and what is the policy? @kubernetes/steering-committee |
|
+1 i have nothing against k3s but it is a distro, or a feature, of k8s. |
|
To expand - imho it is ok for the foundation to support distribution projects, provided that
I worry that 1-2 are in conflict, as are 3-4. |
Not that I have any important status in the project or the CNCF... but my two cents would be that k3s, while it can seem to be just a distribution at first, it is more than that (at least in its current form). It's a project with entirely different goals than k8s at the moment, and it's hyperfocus ends up creating mini projects that may or may not end up being contributed back to upstream (depending on the overarching goals and decisions for upstream, which again doesn't necessarily match the goals of this project.) Wether that's for or against the sandbox status goals... I can't say, but I think labelling it as purely a distribution isn't the right call. |
|
@timothysc Regarding patches, there's no change we do to k/k code that we wouldn't want to push upstream. Overtime we have drastically reduced the amount of changes we do to core k8s. We are slowing working through the right approaches to getting everything upstream. Most of this work happens indirectly though. Any large change we've done to k8s has usually been just doing what upstream was going to do already, so external cloud providers, move to external storage drivers, and similar efforts. Most of the work we do to align with k8s upstream is not very visible but I will say it has always been a clear goal to get to the point of not modifying k8s at all. Getting k3s adopted into CNCF will only better help us achieve that goal. @monadic The term distribution is not a really well defined in the k8s world. We call k3s a distribution because that's a term people kind of get but I can't really say it's the best term. k3s represents a significant amount of work to enable Kubernetes to be used in a simple and lightweight fashion. k3s enables a use case not easily achieve with pure k8s and that has clearly been reflected by it's adoption. k3s is really not too far off from KubeEdge which is already a CNCF project. |
| ### Anticipated use cases | ||
| With hundreds of thousands of downloads and a large and growing community of users, k3s's use cases are well proven. These use cases have already been stated: edge, IoT, ARM devices, and software appliances. One major use case that has not yet been addressed is that of fleet management. This is tied directly to the "edge" and IoT use cases, but is worth calling out explicitly. k3s's ease of deployment, operations, and upgrades makes the managing thousands or even millions of small Kubernetes clusters a possibility. | ||
|
|
||
| Alignment with SIG Reference Model |
There was a problem hiding this comment.
This part is missing?
There was a problem hiding this comment.
Yes, could use some guidance on how to answer this. To be honest, I started with it blank based on seeing similar in the PARSEC pr https://github.com/cncf/toc/pull/441/files#diff-ad0d260064d69088bf8419b00db477c1R156
|
I realise that the whole PR contains a lot of info, but adding an intro / summary as a PR description would be helpful if you're open to adding it? |
|
There is a lot of history here and I think it makes sense to have more folks from the k8s community share their impressions. Some thoughts:
But -- overall -- I echo the concerns of Tim and Alexis. This is either a fork of kubernetes or it is an integration/distribution project.
I'd love to see this happen before joining the CNCF. If I were on the TOC still this is something I'd be looking at very carefully. |
|
As someone coming from IoT, I find inclusion as a separate project quite appealing. Viewing k3s as a stripped down k8s distribution would be short-sighted. But making it separate would encourage contributions focused on making this work well within the typically resource-constrained and hostile (eg unreliable power, internet) IoT/edge environments. (I can't speak to broader CNCF issues, and if there are specific tasks that k3s needs to do before joining the CNCF then I'd encourage the authors to perform those tasks.) |
| Yes. | ||
|
|
||
| ### Project and Code Quality | ||
| _Are there any metrics around code quality? Are there good examples of code reviews? Are there enforced coding standards?_ |
There was a problem hiding this comment.
Will add an answer later today or this evening.
|
@jbeda Thanks for your feedback, it's much appreciated and you bring up some great points probably shared by many working on core Kubernetes.
The idea that k3s is a fork does come up and I think it's important to clarify this point. k3s is not a fork and I'm open to hear any data points to indicate it is, because it's not the intention. Today k3s modifies the core Kubernetes and caries a patch set we keep in sync with all community maintained version of Kubernetes. The patch set is about 300 lines and deals primarily with how we package k8s, not functionality. This is a huge improvement as the first version of k3s had a diff of literally millions of lines. So I would in no way call what we do a fork as it's fairly straight forward for any distribution of X to carry some patches that can't or haven't yet been upstreamed. Regarding the idea of not contributing to Kubernetes. Again, the patch set we maintain mostly has to do with how we integrate and package k8s, not features. So there really isn't a single feature to upstream and I'd love to hear if there is something in k3s that one would like to see upstream as I'd gladly do that. I'll be honest in that the majority of the patches we carry do not have enough technical merit to be upstreamed, they can be viewed more as technical debt where we did things wrong and there are better known approaches. This is why this patch set is dwindling overtime without having to upstream much of anything beyond bug fixes. One of the biggest changes we did to Kubernetes was that k3s supports SQL backends. The first version of this code modified k8s storage backend. Based on feedback we got from upstream the recommended approach was to create a etcd shim. As a result we rewrote what was previously called kvsql into a new component kine. The great thing about this project is that it works with any k8s distribution, not just k3s and has seen a lot of adoption already. Kine is a project I think should be donated to kubernetes, but we just haven't had enough time to do that yet. I 100% understand the impression that k3s hasn't worked enough with k8s upstream. The biggest reason is that the majority of what we do with k3s has nothing to do with upstream k8s. This also justifies why we it exists as a separate project. k3s builds upon k8s.
While the goals of k3s for sure overlap with many projects in the ecosystem it's not really complete to say its basically the same as hyperkube. It doesn't take much to see that k3s is far beyond the scope of hyperkube. At this point hyperkube has basically been dropped from k/k so there really isn't much point belaboring this point.
k3s has been way more successful of a project than we could have possibility imagined or expected. You are absolutely right in that we probably up to this point have not been as organized of a project as one would hope. But that is basically the entire reason we want to join CNCF. We have the users and adoption, we need help better running this project. |
|
Brief note regarding vendor-neutral governance: this has typically not been a requirement for projects to be accepted into the CNCF sandbox. It has certainly been a requirement for graduation but it's thus far been expected that projects can use their "sandbox time" to attract contributors from outside the initial org. |
|
@lucperkins Agree with you -- but we did look to see if a project was on the trajectory and trying to bring on other contributors and create a community driven project. This is clearly subjective and something that the TOC will have to consider. |
Even if it is a fork, I see nothing in the charter that would prevent a fork of k8s from being accepted into the CNCF. I see several bylaws that would suggest just the opposite, that inclusion is warranted - maybe even especially if it was a fork. |
|
AFAIK, we used to have an agreement that CNCF will not accept a distribution :) Agree with Timothy & Joe, k3s is more like a distribution of k8s; anyway, how about raising to k8s steering committee/sig-architecture/sig-node avoiding any confusion to TOC? |
|
@raravena80 , this PR should catalog to sig-runtime, WDYT? |
|
k3s is a distro of Kubernetes in the same way CoreOS is a distro of Linux - i.e. it adds value for use cases that will either conflict with upstream, dilute existing use cases or upstream maintainers will just not be able to prioritize adequately. If Kubernetes is the platform of platforms that we imagined it would be then projects like k3s should be welcome in the CNCF community. There is real end-user value in k3s judging by its popularity and production deployments. |
|
@k82cn yes, I believe this is within the scope of SIG-Runtime. |
A distinction that I would add is that KubeEdge is built on top of Kubernetes and uses standard or any other flavor of Kubernetes whereas k3s is a specific flavor of Kubernetes primarily targeted for the edge (i.e single binary instead of separate components/binaries) with some additional components (containerd, Flannel, CoreDNS, CNI, Traefik, etc) |
|
k3s is a opinionated distro optimized for a set of user stories, and their documentation states that outright. My point earlier is on policy and governance. There are a lot of opinionated k8s distros; gke, eks, aks, openshift, k3s, to name only a few. The question is really, "What is the CNCF policy on housing distros?" What effects would this have on the ecosystem? It's a slippery slope... and requires a well thought out policy to spell out the rules and clauses. For example, we saw in the early days of kubernetes that opinionated deployment tools raced to become part of the k8s org and it was misused as leverage in an effort to try and land grab on market-share. I'm not saying that is the intent is here, but it could be easily misconstrued, which is why policy is so important. |
|
@timothysc These are all good points and I look forward to hearing the TOC's approach to distributions. My comment previously that a distribution is a not a well defined term in the k8s space is highlighted by your examples of GKE, EKS, AKS, OpenShift and k3s. GKE, EKS, AKS are managed offerings. OpenShift is a fully integrated end to end container platform and product. While all these can be loosely labeled as a distributions they are clearly in different categories. It will be interesting how the TOC chooses to classify the space. At the end of the day k3s' rate of adoption identifies there is something unique about it and therefore will probably be difficult to easily characterize it. I have a hard time seeing k3s as a CNCF project would somehow be a net negative for CNCF or the Kubernetes ecosystem but I do understand this is something new and therefore deserves consideration. |
|
k3s self-describes as a distribution, and I don’t think distributions in general would make good CNCF projects, because we don’t want to compete against our own community. But is k3s just a distribution? Is it a really distribution in the same sense as commercial or vendor-specific offerings like GKE or OpenShift or Rancher? It also solves the genuine end-user problem of running cloud-native applications in constrained environments like edge. So I don’t think it would be serving our community well to dismiss it simply because of the word “distribution” without further consideration. I’d very much like to hear the view of the Kubernetes steering committee about whether and how something like k3s could / should co-exist with Kubernetes, and how they would envisage the CNCF supporting the adoption of K8s in Edge / IoT environments. (This comment is my own, it’s not based on a TOC discussion, but I wanted to reply publicly since I’ve been messaged about it in private - not, I might add, by anyone directly connected with the project.) |
|
I really appreciate everyone's input on this thread. As to @cjellick question, I just want to clarify that according to https://github.com/cncf/toc/blob/master/process/project_proposals.adoc it seems we have hit "Step 2 - TOC Triage." From my understand that's mostly a sit and wait from the PR creators perspective. We are just asking if there is anything we need to do move the process forward. Any comment from TOC members as to what we should be expecting or are required to do would be helpful. Thanks. |
|
FWIW -- I know that this is a spoof account but this tweet shows that there is legitimate confusion over the name k3s. https://twitter.com/morecobol/status/1280877846567751680
|
|
In what way @jbeda? What is the confusion on the thread or the linked blog post by Civo who have built a commercial service upon it? |
|
The fact that someone has to ask the question implies, to me, that they are confused and see them as overlapping. The closeness of naming implies a connection that isn't there. See also: https://twitter.com/dustinmoris/status/1280886236723503104 |
I'm not sure how I see one random person's confusion as being connected or representative. We can always find some subjective evidence to support whatever opinion we have... There's a lot of confusion, in some circles, about what is Kubernetes to Chef, or to Ansible, the list goes on. We don't consider those to be the same, because they fundamentally... aren't. While I, personally, think it's good to highlight there is potential confusion, I just think that maybe we should make sure it's a larger sample size. |
|
I'm sorry but no matter the many merits of K3s its name is unambiguously confusing. By all means sample the real world but other than for a handful of super in the know people, k8s and k3s are identical. |
Not sure we have enough evidence to make such a statement. Also, it doesn't really have anything to do with including it or not. If k3s has an ambiguous name, rename it, its not really k8s fault that other projects chose a name that makes differentiation hard. |
|
Yes exactly k3s should have a different name and release path, or be a subproject of k8s. |
|
Question:
In my opinion, highlighting a random person tweeting that they don't know what K3S is does not indicate that there is a naming problem. That person doesn't know what K3S is and was not willing to Google it in a follow-up tweet. it's a very small sample size, and quite frankly seems a little bit ridiculous to pull up a tweet like that to prove a point about the naming of this particular project. Nor do I feel like the naming should have anything to do with its validity for inclusion in the CNCF sandbox. It seems like in general there is a very strong consensus to include K3S in the CNCF sandbox, and correct me if I'm wrong but wouldn't including the project in CNCF give the community greater power to be able to make some changes to the project and also to push certain changes upstream? |
|
Hi there! I want to help, so I also tweeted: https://twitter.com/__yajo/status/1281130802848333824. Now the world is balanced again. 🤣 Now, jokes apart, forget the name. Rancher had a project and had to name it somehow. The name is already known, meaning it has a nice brand and website, and some child projects based on it with names based on it. Don't you like the name? Then you should have created it first and named it however you liked. But right now, it's just a name. Some like it, some not. If you change it, some will like it, some will not, so that will make no difference. Forget it and move on! 💪 😉 |
|
@ibuildthecloud Just for transparency. When we were going through the sandbox proposals, this was the next one before we ran out of time in our meeting. This will be discussed at our meeting next week. There is nothing more we need from you or the k3s team at this time. Thanks for your submission. |
|
Thanks for the info @michelleN |
|
if your levenshtein distance is exactly but maybe cobol was just having a weird day yall |
|
+1! |
|
I don't think it needs to be accepted as a sandbox project.
Finally, sandbox should not be abused. |
|
Note: I am not speaking for rancher labs, I just want to inject my own interpretation: Kubernetes is more a specification/abstract system (to me personally) than "a piece of software", k3s implements thus specification/system and optimizes it for low-spec / edge computing, allowing a kubernetes system to exist at the low end of the computing spectrum. Thus, it is not really a k8s fork, but rather a project that implements k8s in a different way than conventional slightly-modified cloud-provided kubernetes engines. And I think I can also slightly answer I think that by accepting k3s as a CNCF sandbox project, it can rise up the project tiers (incubating, graduated) and reach wider recognition, and a "stamp of approval" from the CNCF that k3s is ready for production use, to have another authority validating its development and existence could help with adoption and further recognition across the (admittedly chaotic and fast-paced) cloud landscape. |
|
Ok. I thought this was a no brainer but I'll add my reasoning instead of the generic "fanboy +1" I entered before. "Cloud native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach. These techniques enable loosely coupled systems that are resilient, manageable, and observable. Combined with robust automation, they allow engineers to make high-impact changes frequently and predictably with minimal toil. The Cloud Native Computing Foundation seeks to drive adoption of this paradigm by fostering and sustaining an ecosystem of open source, vendor-neutral projects. We democratize state-of-the-art patterns to make these innovations accessible for everyone." What k3s does is it extends k8s to play well at the "smaller scsale" - just like monta-vista linux introduced linuxfor embedded device in the early 2000 and was the prequel to "linux for mobile"(android) |
|
Questioning the k3s project does not mean questioning the mission of the
cncf.
From the perspective of the development of k3s, from the 1.13 version
contains a lot of k8s code to the current version, there is almost no
relationship with k8s.
Maybe I didn't go into k3s project, Is it appropriate for the current
expression of the relationship between k3s and k8s ?
It is true that Cloud native technologies empower organizations to build
and run scalable applications in modern, dynamic environments such as
public, private, and hybrid clouds. Containers, service meshes,
microservices, immutable infrastructure, and declarative APIs exemplify
this approach.
IMO, a full explanation is needed about the relationship between k3s and
k8s, it is time for cncf to make some changes when accepting applications
for sandbox projects, rather than simply +1.
Thank you !
Lior Kesos <notifications@github.com> 于2020年8月10日周一 上午2:51写道：
… Ok. I thought this was a no brainer but I'll add my reasoning instead of
the generic "fanboy +1" I entered before.
Quoting the CNCF's mission..
"Cloud native technologies empower organizations to build and run scalable
applications in modern, dynamic environments such as public, private, and
hybrid clouds. Containers, service meshes, microservices, immutable
infrastructure, and declarative APIs exemplify this approach.
These techniques enable loosely coupled systems that are resilient,
manageable, and observable. Combined with robust automation, they allow
engineers to make high-impact changes frequently and predictably with
minimal toil.
The Cloud Native Computing Foundation seeks to drive adoption of this
paradigm by fostering and sustaining an ecosystem of open source,
vendor-neutral projects. We democratize state-of-the-art patterns to make
these innovations accessible for everyone."
What k3s does is it extends k8s to play well at the "smaller scsale" -
just like monta-vista linux introduced linuxfor embedded device in the
early 2000 and was the prequel to "linux for mobile"(android)
I've been working with k3s for the past 6 monthes and it brings the
cloud-native practices to form-factors I can work with empowering me ein
the process (Look ma, I can program autonomous drones to do image
recognition on a jetson board using kubeflow on ARM)
This skillset would have been much harder for me if I did not apply my
existing k8s skills to the smaller k8s compliant k3s project.
By donating k3s to the CNCF - rancher is also addressing the vendor lockin
issue and I predict also redhat will leave all kind. of wierd ideas about
"mini or microshift" and will adopt k3s as the standard for "embeded k3s
systems.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#447 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABMEQBME2RVGHA656P6ZDJTR73V43ANCNFSM4M6CG2VA>
.
|
|
@zouyee I shall repeat: The relation between k3s and kubernetes is that (at this stage) k3s fully implements kubernetes' system, but in a more efficient and (slightly opinionated) fashion, ready for low-end devices. The development history of k3s being a source code fork of k8s (which, as you said, has heavily diverged now) should not interject with that above principle. K3s' relation to k8s is to implement it on a low-spec scale. |
|
FYI, we've revamped our README.md to more clearly explain what k3s is and isn't. https://github.com/rancher/k3s/blob/master/README.md |
|
guessing this is related ... https://twitter.com/ibuildthecloud/status/1292497393150078976 |
|
An interesting question here is as well if it wouldn't make more sense to put k3s and its building blocks into the kubernetes{,-sigs} ecosystem rather than directly into the CNCF |
|
sig-low-spec? |
I'd be supportive of this. |
| ### Future Plans | ||
| k3s's roadmap includes: | ||
| - Improving stability and support across operating systems and architectures | ||
| - Enhancing security through SELinux support and CIS compliance | ||
| - Developing HA solutions appropriate for small edge clusters where even three nodes is cost prohibitive | ||
| - Improving support for Kubernetes cloud providers | ||
| - Improved support for database backends, including embedded etcd | ||
|
|
||
| while continuing to maintain release cadence and conformance with upstream Kubernetes. | ||
|
|
There was a problem hiding this comment.
Please could you add this info to the k3s repo so that folks can easily find the Roadmap?
And It would be really helpful to clearly surface the intention to stay conformant with upstream, and to maintain the same release cadence
The TOC had a discussion with the Kubernetes Steering Committee about this but KSC expressed a preference not to go that route at this time (which doesn't mean that it can't move in the future if that were good for both projects). Please see this thread on the TOC mailing list. |
|
+1 Binding: Saad abstains: https://lists.cncf.io/g/cncf-toc/message/5113 |
Let's see where we are with k3s-io/k3s#2245 at the 1 year mark in about a month. |
No description provided.