Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scope of Kubernetes #3180

Merged
merged 1 commit into from Mar 11, 2019

Conversation

@bgrant0607
Copy link
Member

commented Jan 31, 2019

Added document explaining the scope of the Kubernetes project, including
historical rationales and decision criteria.

cc @thockin @smarterclayton @kubernetes/sig-architecture-pr-reviews

Added document explaining the scope of the Kubernetes project, including
historical rationales and decision criteria.
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Jan 31, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bgrant0607

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mattfarina
Copy link
Member

left a comment

@bgrant0607 thanks for putting this together.

Would it make sense to have a section that outlined the process by which this is revised including the body that owns it (sig architecture)?

Would it make sense to highlight areas we want to encourage the ecosystem to build projects alongside the k8s ones? Basically, areas we have but we don't think need to be exclusive to the project. For example, web UIs or local environments. I use these examples because they are already being done.

heapster, which is grandfathered), alerting, application performance
management, tracing, and debugging tools
* General-purpose machine configuration (e.g., Chef, Puppet, Ansible,
Salt), maintenance, automation (e.g., Rundeck), and management systems

This comment has been minimized.

Copy link
@mattfarina

mattfarina Jan 31, 2019

Member

The tools listed her as examples are also used for configuration management of workloads in Kubernetes. For example, I've read about using stuff from Chef to manage applications. It can even handle offline storing of credentials securely. Should we list tools that are used for both general purpose machine configuration and kubernetes workload configuration management?

Salt), maintenance, automation (e.g., Rundeck), and management systems
* Templating and configuration languages (e.g., jinja, jsonnet,
starlark, hcl, dhall, hocon)
* File packaging tools (e.g., helm, kpm, kubepack, duffle)

This comment has been minimized.

Copy link
@mattfarina

mattfarina Jan 31, 2019

Member

The previous several bullets have talked about a bunch of specific areas, which I like, but one part of this workload configuration management. What of that should be listed as out of scope to help provide guidance?

* asynchronous event reporting
* API producer SDK
* API client SDK / libraries in widely used languages
* dynamic, resource-oriented CLI, as a reference implementation for interacting with the API and basic tool for declarative and imperative management

This comment has been minimized.

Copy link
@mattfarina

mattfarina Jan 31, 2019

Member

What is basic? It isn't the same as simple. To grab a definition for basic...

Of, relating to, or forming a base; fundamental: "Basic changes in public opinion often occur because of shifts in concerns and priorities” ( Atlantic).
adj.
Of, being, or serving as a starting point or basis: a basic course in Russian; a set of basic woodworking tools.

Is this meant to be foundational and the starting point for others to build on for declarative and imperative management? Is that in terms of workflows, functionality, or something else?

I ask because this has impacts on both SIG CLI users and on tools that build on top of Kubernetes by needing to look at kubectl as their foundation.

Or, maybe I'm missing something or have it wrong. Just looking for clarification.

This comment has been minimized.

Copy link
@smarterclayton

smarterclayton Jan 31, 2019

Contributor

I think it is within scope to provide:

  1. foundational CLI tooling that works generically with resources
  2. higher level CLI tooling that allows common day to day workflows of applying core API types like config and secrets to the cluster repeatedly
  3. works generically well for all object types that follow the kube declarative API model, and supports day-to-day workflow on the core api objects
  4. can be used as a final step by higher level tooling to avoid recreating common code like apply or create
  5. fits well within scripted cluster-admin and cluster-consumer focused flows that alter objects.

I don't expect kubectl is required to be used to accomplish any of those. I do think that kubectl is the complement to the declarative API model and that it is within scope to do that, as it has been for the entire history of the project.

This comment has been minimized.

Copy link
@mattfarina

mattfarina Feb 1, 2019

Member

can be used as a final step by higher level tooling to avoid recreating common code like apply or create

I would prefer that the final step for higher level tooling be baked into client libraries or the API. This would allow many tools to use those features without having shell out to kubectl or deal with making sure it's installed at a right dependency version.

Why can't these things be baked into the API and client libs?

This comment has been minimized.

Copy link
@mattfarina

mattfarina Feb 1, 2019

Member

higher level CLI tooling that allows common day to day workflows of applying core API types like config and secrets to the cluster repeatedly

Whose higher level workflows and how to we figure out what those are? Since most of the Kubernetes users aren't in meetings or involved in the community how do we determine the workflows to chase after?

If the base level features were built into client libraries and APIs people could more easily create their own workflows as needed. The k8s client would not need to try and figure out the workflows to chase.

This comment has been minimized.

Copy link
@smarterclayton

smarterclayton Feb 1, 2019

Contributor

These tools are, and we do, and we have been for most of the last two years.

But there's a big difference between "some go code that lives in a library" and "go code that is used to solve a problem for a user right now". I'm not interested in the generic "we could build this in so many ways", I'm much more focused on reducing day to day user stress within Kubernetes. All libraries need one major consumer giving them purpose, and our declarative resource management tool kubectl is the major consumer of those libraries.

This comment has been minimized.

Copy link
@mattfarina

mattfarina Feb 1, 2019

Member

Here's an example. Validation of resource config is something that was listed as an area in the SIG CLI scope in their proposed changes. If the project is going to add that should it be at the API level and then the CLI and others can leverage it? If that's the case, is it a CLI feature or an API machinery feature?

If validation were in the API than tools like Helm, ksonnet, kubecfg (the one related to ksonnet), and others could all use it without having to exec out to kubectl or importing some of their internal libraries.

I use validation as an example because it is a common enough workflow that there are multiple ecosystem tools doing it today.

How we structure things has the ability to be a force multiplier. That means app devs can build tools using JS, Java, Python, PHP, C#, C++, Ruby, or one of the many other languages that are popular. I grabbed these from Redmonks last programming language popularity rankings and used them in order.

It would also mean that the CLI would be a reference implementation (as noted in this doc) and that others could use that reference as a way to build this into their own stuff.

This comment has been minimized.

Copy link
@bgrant0607

bgrant0607 Feb 5, 2019

Author Member

kubectl already implements validation using the OpenAPI spec, which https://github.com/garethr/kubeval also uses.

Server-side validation is still something I'd like done (kubernetes/kubernetes#5889), but has additional challenges (backward compatibility, version skew, how to return warnings, how to make error messages understandable and tool-friendly, ...). And we decided diff and server-side dry run were more urgent.

* required
* pluggable
* optional
* usable independently of the rest of Kubernetes

This comment has been minimized.

Copy link
@mattfarina

mattfarina Jan 31, 2019

Member

Not sure the above list provides clear guidance on what is core. For example, Kompose is in the Kubernetes GitHub org but is not core. Should the experiments, on the kubernetes-sigs org, be considered core? I looks these cases fit from the list above.

Would there be another way that is maybe more clear?

containerized workloads and services, that facilitates both
declarative configuration and automation. Workload portability is an
especially high priority. Kubernetes provides a flexible, easy-to-run,
secure foundation for running containerized applications on any cloud

This comment has been minimized.

Copy link
@quinton-hoole

quinton-hoole Jan 31, 2019

nit: public cloud provider

This comment has been minimized.

Copy link
@timothysc

timothysc Feb 1, 2019

Member

disagree, I think cloud provider is more accurate.

This comment has been minimized.

Copy link
@quinton-hoole

quinton-hoole Feb 1, 2019

So you think we're talking about three things here? (Public cloud, private cloud), and "your own systems"? I was thinking "public cloud" and "private cloud/your own systems". I would understand either approach, but I think right now the wording is confusing, and contributes to the existing confusion where many people think that "cloud" implicitly means "public cloud", which in my mind it does not.

This comment has been minimized.

Copy link
@bgrant0607

bgrant0607 Feb 1, 2019

Author Member

"Your own systems" could also be your laptop, pair of workstations under your desk, raspberry pi, or whatever.

This came verbatim out of an email thread, so I'm happy to tweak, but I don't think this particular point is worth a lot of wordsmithing.

* local container image caching
* configuration and secret distribution
* manual and automatic horizontal and vertical scaling
* deployment, progressive (aka rolling) upgrades, and downgrades

This comment has been minimized.

Copy link
@smarterclayton

smarterclayton Jan 31, 2019

Contributor

I would probably add, "and the ability to build more complex workflows on top" to clarify that we consider the full breadth of deployment not in scope.


## Scope domains

Most decisions are regarding whether any part of the project should

This comment has been minimized.

Copy link
@quinton-hoole

quinton-hoole Jan 31, 2019

This whole paragraph is quite difficult to understand. What are you trying to say here?

This comment has been minimized.

Copy link
@timothysc

timothysc Feb 1, 2019

Member

Agreed. This paragraph makes me want to ask more questions.

This comment has been minimized.

Copy link
@bgrant0607

bgrant0607 Feb 1, 2019

Author Member

Should I just delete this section? I pulled it from notes from a past discussion about what is "core", but I'm not sure it's useful, actually.

This comment has been minimized.

Copy link
@dims

dims Feb 1, 2019

Member

yes please (delete the section)

# Kubernetes scope

Purpose of this doc: Clarify factors affecting decisions regarding
what is and is not in scope for the Kubernetes project.

This comment has been minimized.

Copy link
@quinton-hoole

quinton-hoole Jan 31, 2019

It would be useful to explain here what the relevance of "in scope" vs "out of scope" is - i.e. what do those two labels mean, exactly, and therefore why should I read this document. Later on there is a sentence "Most decisions are regarding whether any part of the project should undertake efforts in a particular area. However, some decisions may sometimes be necessary for smaller scopes". I think that may be what you're trying to convey, but it's not at all clear. Also, there have previously been objections to this point of view, in the sense that open source contributors can undertake efforts anywhere they or their sponsors/employers choose to (although if their attempted contributions by necessity need to be reviewed others, they may run into problems). I think you'll need to reword this to make your point more clear.

This comment has been minimized.

Copy link
@timothysc

This comment has been minimized.

Copy link
@bgrant0607

bgrant0607 Feb 1, 2019

Author Member

Ack

This comment has been minimized.

Copy link
@bgrant0607

bgrant0607 Feb 5, 2019

Author Member

This and other comments make me think there are even more "meta" issues to resolve, such as our goals for Kubernetes, the software, and for Kubernetes, the project, why the issue of scope is important for each, how the issue of scope applies to new proposed functionality, and how it applies to existing code.

I was going to make a separate pass to update the list of related documents below, but I'll take another look at a better way to organize this content, so that we don't just end up with 4-5+ overlapping documents with different levels of detail and currentness.

* optional
* usable independently of the rest of Kubernetes

## Other inclusion considerations

This comment has been minimized.

Copy link
@quinton-hoole

quinton-hoole Jan 31, 2019

Is the message intended to be that the answer to all of these questions needs to be 'yes' in order to be considered in scope? Or mostly yes? Or some combination? It seems that for a minority of the questions, an answer of 'no' is required, or at least an answer of 'yes' motivates for exclusion. For example "Is there an acceptable home for the recommended ecosystem solution(s)?"

This comment has been minimized.

Copy link
@quinton-hoole

quinton-hoole Feb 1, 2019

In summary, needs some explanation or rewording to clarify.

This comment has been minimized.

Copy link
@bgrant0607

bgrant0607 Feb 1, 2019

Author Member

Ack

* Has the functionality been provided by the project/release/component
historically?

## Technical scope details and rationale

This comment has been minimized.

Copy link
@quinton-hoole

quinton-hoole Jan 31, 2019

It's worth noting that most of the sections do not yet include a rationale, and therefore come across as simply a statement of decree, i.e. these things are arbitrarily declared to be in scope, with no reasons given. It would be good to have a clear set of guiding principles for inclusion or exclusion (and, for example, cross references to numbered or named principles for each item). In the absence of these, it would be fairly easy to make a counter-argument to many of the inclusions/exclusions, and end up in the kind of arguments that I think this document is attempting to circumvent.

And separately, there is a fair amount of stuff that is declared "grandfathered in", that superficially seems fairly arbitrary. I think it would be useful to be clear about what's:

  1. currently incorrectly treated as being in scope simply because of historical habits (and we haven't yet gotten around to kicking it out of scope), vs
  2. stuff we legitimately cant kick out of scope any more, ever, because that would somehow break backward compatibility, vs
  3. stuff that is not truly in scope, but for some, possibly pragmatic reason, we still care about it.

I think lumping all three of the above together under the label "grandfathered in" creates some confusion and frustration.

This comment has been minimized.

Copy link
@quinton-hoole

quinton-hoole Feb 2, 2019

A few brain-dumped principles that might apply:

  1. "NICE-TO-HAVES" If it's fairy easy to build and run a useful cluster containing useful production cloud-native applications without this, then it's probably out of scope (e.g. istio, helm).
  2. "NOT-REALLY-OPTIONAL-NATIVES" If >80?% of users use this thing with their clusters anyway, and it's not much use anywhere else than with kubernetes (to exclude things like etcd and CoreDNS, which seem self-evidently out of scope) then it's probably in-scope (e.g. kubectl).
  3. "GENERAL-TOOLS" If this thing is one of a general class of stuff (e.g. CI or testing frameworks) that is or could be non-kubernetes-specific, even if we just happen to have decided to build it ourselves (e.g. prow) then it's probably out of scope.
  4. "TABLE-STAKES" Kubernetes is not really considered finished, fully usable, or on par with competitive projects without one of these, and there isn't a good enough open source one out there that we can vendor into upstream by default (e.g. GUI, Multicluster tools) then it's probably in scope.
  5. "PRACTICAL-DE-FACTO-STANDARD" It's useful to have a good standard/default way of doing this, e.g. for documentation, training, automation scripts etc, and no good enough, vendorable, open source alternatives exist, then it's probably in-scope (e.g. kubectl, a cluster build tool, ...)
    ...
    I can dump some more in a more useful place than a PR comment if you think that would be useful?

This comment has been minimized.

Copy link
@bgrant0607

bgrant0607 Feb 5, 2019

Author Member

Ack on "grandfathered" breakdown, though I'd say we need pragmatic reasons to "kick out" existing subprojects. Some past material for consideration includes:
https://github.com/kubernetes/community/blob/master/github-management/kubernetes-repositories.md
https://github.com/kubernetes/community/blob/master/incubator.md

Ack on the need for categories, also. There's some overlap with the earlier layers notion that's worth thinking about.


## Other inclusion considerations

The Kubernetes project is a large, complex effort.

This comment has been minimized.

Copy link
@bgrant0607

bgrant0607 Jan 31, 2019

Author Member

Note to self: Oops. I didn't finish this paragraph

@bgrant0607 bgrant0607 self-assigned this Jan 31, 2019

@timothysc
Copy link
Member

left a comment

IMO the structure and flow of the document needs some TLC.

I think having a set of non-goals in the doc would help.

For example, I don't think we need organizational structure in this doc at all, because it muddles the main intent imo.

containerized workloads and services, that facilitates both
declarative configuration and automation. Workload portability is an
especially high priority. Kubernetes provides a flexible, easy-to-run,
secure foundation for running containerized applications on any cloud

This comment has been minimized.

Copy link
@timothysc

timothysc Feb 1, 2019

Member

disagree, I think cloud provider is more accurate.


While not a full distribution in the Linux sense, adoption of
Kubernetes has been facilitated by the fact that the upstream releases
are usable on their own, with minimal dependencies (e.g., etcd, a

This comment has been minimized.

Copy link
@timothysc

timothysc Feb 1, 2019

Member

I would leave out the "e.g." details in this doc, b/c that list will grow over time, and other documents outline that.

This comment has been minimized.

Copy link
@bgrant0607

bgrant0607 Feb 1, 2019

Author Member

@dims suggested adding those for clarity, I think.

We don't yet have a canonical list AFAIK, though I believe one was requested by the release team, in order to clarify what belongs in release notes, if nothing else.


## Scope domains

Most decisions are regarding whether any part of the project should

This comment has been minimized.

Copy link
@timothysc

timothysc Feb 1, 2019

Member

Agreed. This paragraph makes me want to ask more questions.

@timothysc

This comment has been minimized.

Copy link
Member

commented Feb 1, 2019

/assign @timothysc
/cc @kubernetes/steering-committee - because it relates to the conversation this week

@k8s-ci-robot k8s-ci-robot requested a review from kubernetes/steering-committee Feb 1, 2019

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Feb 1, 2019

@timothysc: GitHub didn't allow me to request PR reviews from the following users: relates, the, conversation, week, to, this, -, because, it.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/assign @timothysc
/cc @kubernetes/steering-committee - because it relates to the conversation this week

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

* Cluster lifecycle tools
* Extensibility to support execution and management in diverse environments
* Multi-cluster management tools and systems
* Project GitHub automation and other process automation

This comment has been minimized.

Copy link
@quinton-hoole

quinton-hoole Feb 1, 2019

I'm not at all convinced that these need to be considered in scope. I would quite imagine spinning these out into a separate autonomous set of projects, and also imagine that they would be very useful to other (e.g. CNCF) projects.

This comment has been minimized.

Copy link
@bgrant0607

bgrant0607 Feb 1, 2019

Author Member

In a perfect world that might be true, but we can't afford to disempower our selves to solve our own (fairly dire) automation needs. If someone were to spin some out, ensure they were adequate for us, and generalize them for others, then, we'd happily adopt them, as with the devstats dashboard, which replaced a previous K8s-specific effort.

@quinton-hoole
Copy link

left a comment

Added comments.

* Extensibility to support execution and management in diverse environments
* Multi-cluster management tools and systems
* Project GitHub automation and other process automation
* Project continuous build and test infrastructure

This comment has been minimized.

Copy link
@quinton-hoole

quinton-hoole Feb 1, 2019

At least parts of this seem to fall into a similar category to above. Relatively kubernetes-specific stuff could certainly stay in-scope, but generic build and test tools and frameworks don't belong in-scope IMO.

This comment has been minimized.

Copy link
@bgrant0607

bgrant0607 Feb 1, 2019

Author Member

We tried several services and Jenkins before deciding to build our own.

This comment has been minimized.

Copy link
@quinton-hoole

quinton-hoole Feb 27, 2019

What I mean is we should spin them out, e.g. to the CNCF, similar to what happened with Helm, and what OpenStack did with Zuul-CI.

### Service discovery, load balancing, and routing

Including:
* endpoint tracking and discovery, including pod and non-pod endpoints

This comment has been minimized.

Copy link
@thockin

thockin Feb 27, 2019

Member

We have affordances for external services today, but I wonder whether that was a mistake. I am not convinced we want to carry that to logical conclusion, as opposed to integrating into higher-level SD systems, including things like Consul but also things like Istio.

I guess, since we have it now it's hard to say it's out of scope, but I am not 100% sure we want to carry it forward.

This comment has been minimized.

Copy link
@bgrant0607

bgrant0607 Feb 27, 2019

Author Member

What I think was a mistake is creating the expectation that Endpoints would be durable -- it should not be the source of record. If we nuked all Endpoints, we should expect controllers to repopulate them.

It's worth thinking about what interfaces we'd want if we intended to integrate with Consul and Istio, but we do need an interface that's accessible by more than just proxies. Monitoring and health-checking systems are obvious examples of other consumers of the information.

Otherwise the ecosystem will fragment and/or lots of things will then track pods directly, and label selectors would need to be propagated by users and implemented in many clients. (We still need an API-based mechanism to convert the schematized form of label selectors to the query param form: kubernetes/kubernetes#3676.)

This comment has been minimized.

Copy link
@bgrant0607

This comment has been minimized.

@jdumars

This comment has been minimized.

Copy link
Member

commented Mar 7, 2019

How do we reach a "good enough" state on this document so we can merge and iterate?

@philips

This comment has been minimized.

Copy link
Contributor

commented Mar 11, 2019

I think it is good enough

/lgtm

@philips

This comment has been minimized.

Copy link
Contributor

commented Mar 11, 2019

/hold

@philips

This comment has been minimized.

Copy link
Contributor

commented Mar 11, 2019

I will let someone else remove the hold. But, it has my LGTM.

@dims

This comment has been minimized.

Copy link
Member

commented Mar 11, 2019

/hold cancel

+1 to merge and iterate.

@k8s-ci-robot k8s-ci-robot merged commit be85a98 into kubernetes:master Mar 11, 2019

3 checks passed

cla/linuxfoundation bgrant0607 authorized
Details
pull-community-verify Job succeeded.
Details
tide In merge pool.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.