Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: add-ons design proposal #242

Closed
wants to merge 10 commits into from

Conversation

@errordeveloper
Copy link
Member

commented Oct 5, 2018

No description provided.

@christopherhein
Copy link
Collaborator

left a comment

Overall 💯 agree with this feature. Added some changes and my thoughts.

What could really help this would be talking about a couple example projects or addons that would scope into this. For example --addons=helm (or whatever implementation looks like)

When we pass helm as an addon eksctl uses the compiled in helm libraries to call helm init from the go SDK. How do we tell it to use RBAC? Did it auto create the SA and ClusterRoleBinding, etc? When you call the sdk does it automatically write the files to the persons home directory?

Or for another --addons=dashboard when this is triggered it uses the referenced github manifest links to apply to the cluster.

Last one, kube2iam when you deploy this is will use the helm chart with params (do we have a dependency graph that we now have to manage?) then it will deploy the IAM Assume Role Cloudformation template which referenced there node group to get the instance role ARN? it is auto-compiled into the nodegroup CFN, if so how would we handle removing it.

Hopefully that will help guide this a little more from the experience perspective. For example if I as a customer still need to manage my own upgrades of these addons I believe that would be acceptable as long as the initial heavy lifting was done. But maybe we strive for the implementation to be idempotent and if you deploy the dashboard and we upgrade the cli with a newer version you could run eksctl apply addon dashboard and it would apply the manifests in place just updating the existing. For helm this could get complicated cause now we need a way to persist the state of the individual addon names that they were deployed as.

Again super excited about this feature, happy to see we're starting with a proposal!


- fully deterministic behaviour for a given version of eksctl, thereby it should have
- strict versioning of built-in add-ons by default
- any externally-sourced config has to be pinned down or vendored *[TBD]*

This comment has been minimized.

Copy link
@christopherhein

christopherhein Oct 5, 2018

Collaborator

Can we get more definition around this? Do you mean a helm repo or chart, or manifest that lives at some url? Trying to understand how we could parameterize these…

This comment has been minimized.

Copy link
@richardcase

richardcase Oct 9, 2018

Collaborator

Good point @christopherhein. How i see us using it where i work, it would be great to support:

  • Chart repo / chart museum (and chart name/version)
  • Explicit pointer to a chart in a repo

An add-on extends functionality of a Kubernetes cluster, it may consist of a workload and/or configuration within the give cluster or the cloud provider. The workload, if present, would may classify as an operator or a controller, however that is not neccessary.

The difference between add-ons and anything user decided to run themselves, is that from user's point of view, add-on is something they don't need to maintain directly. A cluster provider or bootstrap tool (i.e. eksctl), is expected to ensure ease of use, by providing minumum viable (that configuration to ensure end-to-end integration and compatibility), as well as catering for reconfiguration and upgrades.

This comment has been minimized.

Copy link
@christopherhein

christopherhein Oct 5, 2018

Collaborator

grammar. "anything a user decided"

- any externally-sourced config has to be pinned down or vendored *[TBD]*
- there should be an option to use latest version or altenative source URL
- user should be able:
- remove any add-on (including it's dependencies) at any time

This comment has been minimized.

Copy link
@christopherhein

christopherhein Oct 5, 2018

Collaborator

Do you consider this a hard requirement? Scope wise thinking that helm doesn't even fit into this.

This comment has been minimized.

Copy link
@errordeveloper

errordeveloper Oct 8, 2018

Author Member

It's up for discussion really, certainly something we cannot avoid having a conversation about.

This comment has been minimized.

Copy link
@mt-inside

mt-inside Oct 16, 2018

What does minikube do...?

- install add-on at any time
- an add-on may consists exclusively of
- a reference to any Helm chart (or a copy of one)
- a reference to a Ksonnet application definition

This comment has been minimized.

Copy link
@christopherhein

christopherhein Oct 5, 2018

Collaborator

also a standalone manifest?

This comment has been minimized.

Copy link
@errordeveloper

errordeveloper Oct 8, 2018

Author Member

Yes, that's kind of implied as default/built-in method, perhaps it could be augmented with kustomize. Basically whatever could be the simplest thing.

- upgrade an add-on
- install add-on at any time
- an add-on may consists exclusively of
- a reference to any Helm chart (or a copy of one)

This comment has been minimized.

Copy link
@christopherhein

christopherhein Oct 5, 2018

Collaborator

I would consider helm itself an addon, do you think we should allow a chart as a packaging format?

This comment has been minimized.

Copy link
@stefanprodan

stefanprodan Oct 5, 2018

Member

An addon could be packaged as a Helm chart but to apply it you don't necessary need Tiller running on the cluster. eksctl could detect if Tiller is not running in the cluster and do helm template | kubectl apply

This comment has been minimized.

Copy link
@christopherhein

christopherhein Oct 5, 2018

Collaborator

That's a very good point @stefanprodan hadn't considered that implementation. How could we handle setting params in those templates via eksctl?

This comment has been minimized.

Copy link
@Raffo

Raffo Oct 6, 2018

Just my two cents reading this: I would try to add additional options of templating as not everybody likes helm or wants to have it as a dependency. This doesn't need to be in a first implementation, but probably it makes sense to make it in the proposal if you agree.

This comment has been minimized.

Copy link
@errordeveloper

errordeveloper Oct 8, 2018

Author Member

Yes, this proposal in the current form assumes there would a few different options. Helm is a popular choice of course, especially due to the fact that there are so many existing community charts.

This comment has been minimized.

Copy link
@stefanprodan

stefanprodan Oct 8, 2018

Member

@christopherhein I would use a yaml file to provide the list of addons and the setting params for each addon.

eksctl create cluster -f ./config.yaml

apiVersion: eksctl.io/v1alpha1
kind: Config
metadata:
  name: eks-dev
spec:
  addons:
    - name: cert-manager
      namespace: kube-system
      chart:
        repository: https://kubernetes-charts.storage.googleapis.com
        name: cert-manager
        version: v0.5.0
      values:
        createCustomResource: true
        rbac:
          create: true
    - name: kured
      namespace: adm
      chart:
        repository: https://kubernetes-charts.storage.googleapis.com
        name: kured
        version: 0.1.0
      values:
        updateStrategy: OnDelete
        rbac:
          create: true

This comment has been minimized.

Copy link
@christopherhein

christopherhein Oct 8, 2018

Collaborator

I love it, sounds like a great way to handle this.

This comment has been minimized.

Copy link
@errordeveloper

errordeveloper Oct 9, 2018

Author Member

Thanks, Stefan! For now, I'd have distinct add-on objects, as I'd rather leave cluster object undefined for until we get to implementing Cluster API (right now the plan is to implement add-ons first). Ultimately, add-on objects can be inlined in cluster object at a later point. I've added some examples just now, will incorporate what you shown as well.

This comment has been minimized.

Copy link
@richardcase

richardcase Oct 9, 2018

Collaborator

Multiple templating/packing options would be great (helm being one obviously). I like the format that @stefanprodan listed but personally i'd prefer separate add-on resources. This would then allow our teams to each look after the addons they are interested in having (ensuring idempotency).

This comment has been minimized.

Copy link
@mt-inside

mt-inside Oct 16, 2018

I really like this too. +1 for trying to bring this into the Cluster object ASAP though, to avoid forking.

1. discuss and specify add-on definition objects and repository layout
2. discuss and specify how different config management tools would integrate (Helm, K/Jsonnet, kubegen, kustomize)
3. implement ability to
- install a built-in add-on (must ensure upgrades are catered for)

This comment has been minimized.

Copy link
@christopherhein

christopherhein Oct 5, 2018

Collaborator

What does "ensure upgrades are catered for" mean? Are we going to enforce a lifecycle?

This comment has been minimized.

Copy link
@errordeveloper

errordeveloper Oct 8, 2018

Author Member

It's open ended at the moment really. I used "upgrade" for the lack of a better word, but what I meant is version change (which is seems as a distinct operation from config change).

This comment has been minimized.

Copy link
@mt-inside

mt-inside Oct 16, 2018

Do you have a list of initial target addons? Flux? Tiller? Istio?

This comment has been minimized.

Copy link
@Lazyshot

Lazyshot Oct 17, 2018

Collaborator

I think Tiller, Dashboard, Istio and Kiam/Kube2Iam are good first targets.

Secondary may be ingress controllers, external-dns, and cert-manager.

This comment has been minimized.

Copy link
@errordeveloper

errordeveloper Oct 18, 2018

Author Member

The source of inspiration is actually in roadmap section of the docs 😉

There are a some issues, e.g. #178, #248 and #53 (to name a few).

@stefanprodan
Copy link
Member

left a comment

Maybe the proposal could include a sample config file for the addons.

apiVersion: eksctl.io/v1alpha1
kind: Config
metadata:
  name: eks-dev
spec:
  addons:
    - name: cert-manager
      namespace: kube-system
      chart:
        repository: https://kubernetes-charts.storage.googleapis.com
        name: cert-manager
        version: v0.5.0
      values:
        createCustomResource: true
        rbac:
          create: true
    - name: kured
      namespace: adm
      chart:
        repository: https://kubernetes-charts.storage.googleapis.com
        name: kured
        version: 0.1.0
      values:
        updateStrategy: OnDelete
        rbac:
          create: true

dismissing feedback over conversation.

- name (fixed) *[TBD]*
- version tag or revision (has default value that user can override)
- source URL (has default value that user can override)
- set of customisations (no default value, user defined)

This comment has been minimized.

Copy link
@richardcase

richardcase Oct 9, 2018

Collaborator

Perhaps a hash of some sort so the integrity of the addon can be verified.

- Kubernetes workload definitions
- Kubernetes configuration objects - PersistentVolumes, ConfigMaps, ...etc
- cloud provider resource definitons (e.g. CloudFormation sub-resources)
- dependencies (supported Kubernetes versions and corresponding cloud providers, list of references to other add-ons)

This comment has been minimized.

Copy link
@richardcase

richardcase Oct 9, 2018

Collaborator

When we say list of references to other add-ons we mean to a specific version of that addon? If we do mean that, then will we support the NPM (or something else) style of specifiying versions (e.g. 1.2.1, >1.2.1, ~2.2.1, ^1.2.1)?

This comment has been minimized.

Copy link
@errordeveloper

errordeveloper Oct 18, 2018

Author Member

Good question! I am not sure, perhaps to start with we could start

I'd be challenging to support multiple versions of the same add-on, but hopefully that won't be required, especially considering that most of the add-ons provide cluster-wide functionality. Perhaps we should state as part of the definition at the very top that every add-on instance is a singleton. In terms of the API, we may allow multiple AddonInstance objects with different names representing effectively the same add-on, but it'd be up to the actual add-on to implement namespacings of its own (e.g. Flux by default runs cluster-wide, but its responsibility can be limited to a namespace).


- fully deterministic behaviour for a given version of eksctl, thereby it should have
- strict versioning of built-in add-ons by default
- any externally-sourced config has to be pinned down or vendored *[TBD]*

This comment has been minimized.

Copy link
@richardcase

richardcase Oct 9, 2018

Collaborator

Good point @christopherhein. How i see us using it where i work, it would be great to support:

  • Chart repo / chart museum (and chart name/version)
  • Explicit pointer to a chart in a repo
- upgrade an add-on
- install add-on at any time
- an add-on may consists exclusively of
- a reference to any Helm chart (or a copy of one)

This comment has been minimized.

Copy link
@richardcase

richardcase Oct 9, 2018

Collaborator

Multiple templating/packing options would be great (helm being one obviously). I like the format that @stefanprodan listed but personally i'd prefer separate add-on resources. This would then allow our teams to each look after the addons they are interested in having (ensuring idempotency).

- fully deterministic behaviour for a given version of eksctl, thereby it should have
- strict versioning of built-in add-ons by default
- any externally-sourced config has to be pinned down or vendored *[TBD]*
- there should be an option to use latest version or altenative source URL

This comment has been minimized.

Copy link
@richardcase

richardcase Oct 9, 2018

Collaborator

If we allow using the latest version (from a remote location) then do we need to verify that the addon is from the same author and hasn't been tampered with?

This comment has been minimized.

Copy link
@errordeveloper

errordeveloper Oct 10, 2018

Author Member

The idea was to have fully vetted (i.e. vendor) built-in add-ons, but latest would be a free for all basically (i.e. up to the user to vet it).

This comment has been minimized.

Copy link
@richardcase

richardcase Oct 11, 2018

Collaborator

Would latest come from a eksctl curated repo though?

This comment has been minimized.

Copy link
@errordeveloper

errordeveloper Oct 11, 2018

Author Member

TBD really. What I'm thinking is this:

  • eksctl repo has add-on that's sourced from a helm chart and a specific version is vendor
  • user may wish to upgrade to latest or whatever other version (perhaps even a fork), there must be a way to pull and install that helm chart without too much dancing around

Similar is ought to be possible for non-helm cases.

This comment has been minimized.

Copy link
@mt-inside

mt-inside Oct 16, 2018

I think this would need a big big banner explaining that it may not work; that it's

  • untested
  • not signed
  • may not work (I can see a lot of the addon charts / CF having to be vendored and altered in some way to work, meaning upstream will not)
- install a built-in add-on (must ensure upgrades are catered for)
- install an external add-on
- reconfigure customer an installed add-on
- upgrade any installed add-on

This comment has been minimized.

Copy link
@richardcase

richardcase Oct 9, 2018

Collaborator

List installed addons (with versions) and indication if there is a new version of the addon available (either because there is a new version included in eksctl or at a remote location).


- built-in add-ons:
- only one version of each add-on is expected to be built-in, alternative versions have to be sourced externally
- only built-in add-ons may include Go code to create AWS resources (this may change in the future, yet significantly simplifies initial design and ensures security)

This comment has been minimized.

Copy link
@richardcase

richardcase Oct 9, 2018

Collaborator

I guess we are not considering supporting a scripting language (i bit retro i know) in the future? I can't decide if thats a good idea or not.

This comment has been minimized.

Copy link
@errordeveloper

errordeveloper Oct 9, 2018

Author Member

To be honest, I'm just trying to avoid adding too much complexity here. I initially thought that having some kind CloudFormation snippet included in the add-on would be a thing, but then I felt like it maybe not be good idea as one would be able to compromise security that way, so it seemed reasonable to avoid that kind of a pickle (at least initially).

This comment has been minimized.

Copy link
@richardcase

richardcase Oct 9, 2018

Collaborator

I agree. There would be a lot of rope to hang ourselves if we introduced a scripting language.

spec:
source:
repo: https://github.com/weaveworks/eksctl
branch: $HEAD

This comment has been minimized.

Copy link
@richardcase

richardcase Oct 9, 2018

Collaborator

Would tags also be supported?

This comment has been minimized.

Copy link
@errordeveloper

errordeveloper Oct 9, 2018

Author Member

Yes, all the usual things would make sense. We might just rely on https://github.com/hashicorp/go-getter actually, it seems to have all the features we'd want.

This comment has been minimized.

Copy link
@richardcase

richardcase Oct 9, 2018

Collaborator

Nice, that looks good to use

@errordeveloper errordeveloper force-pushed the proposal-001 branch from f16d79b to 5aec468 Oct 12, 2018

- fully deterministic behaviour for a given version of eksctl, thereby it should have
- strict versioning of built-in add-ons by default
- any externally-sourced config has to be pinned down or vendored *[TBD]*
- there should be an option to use latest version or altenative source URL

This comment has been minimized.

Copy link
@mt-inside

mt-inside Oct 16, 2018

I think this would need a big big banner explaining that it may not work; that it's

  • untested
  • not signed
  • may not work (I can see a lot of the addon charts / CF having to be vendored and altered in some way to work, meaning upstream will not)
- any externally-sourced config has to be pinned down or vendored *[TBD]*
- there should be an option to use latest version or altenative source URL
- user should be able:
- remove any add-on (including it's dependencies) at any time

This comment has been minimized.

Copy link
@mt-inside

mt-inside Oct 16, 2018

What does minikube do...?

- user should be able:
- remove any add-on (including it's dependencies) at any time
- customise an add-on in an arbitrary fashion
- upgrade an add-on

This comment has been minimized.

Copy link
@mt-inside

mt-inside Oct 16, 2018

This is going to be hard. In-place, no downtime, no data-loss upgrade in the general case is very hard.

Upgrade to what versions? I was assuming each eksctl would come with one vendored copy of each addon's source (e.g. helm chart), tested with that version of EKS and the supporting TF. Would upgrading the addon require upgrading EKS et al too? Is that possible atm?

This comment has been minimized.

Copy link
@Lazyshot

Lazyshot Oct 16, 2018

Collaborator

I agree this requirement is fairly difficult to conquer and will be an ongoing battle, but I believe the onus of upgradeability should be on the add-on itself being either backwards compatible or document breaking version changes.

Just following semantic versioning could allow us to warn a user of potentially breaking changes.

This comment has been minimized.

Copy link
@errordeveloper

errordeveloper Oct 18, 2018

Author Member

I was assuming each eksctl would come with one vendored copy of each addon's source

Yes, that's the idea. But you'd be able to upgrade to some other version from external (read: untested) source.

Would upgrading the addon require upgrading EKS et al too?

Only if it requires newer version of Kubernetes, but even that I don't think it'd make sense to upgrade automatically.

- remove any add-on (including it's dependencies) at any time
- customise an add-on in an arbitrary fashion
- upgrade an add-on
- install add-on at any time

This comment has been minimized.

Copy link
@mt-inside

mt-inside Oct 16, 2018

Again, lots of code to gracefully handle error cases where the user has done something so that the addon now won't install cleanly. Ideally there would be roll-back code for partially successful installations.

Maybe the first version should only support addon install at cluster creation?

This comment has been minimized.

Copy link
@Lazyshot

Lazyshot Oct 16, 2018

Collaborator

Minikube supports installing addons after initial creation.

I could see the first version only supporting it on initial creation, but I'm not sure exactly any limitations besides ones possibly requiring kubelet configuration changes (which is a non-goal with this proposal).

This comment has been minimized.

Copy link
@errordeveloper

errordeveloper Oct 18, 2018

Author Member

the user has done something so that the addon now won't install cleanly

Good point, perhaps AddonInstance should track configuration checksum of all object it holds, and warn if it breaks on upgrade.

- upgrade an add-on
- install add-on at any time
- an add-on may consists exclusively of
- a reference to any Helm chart (or a copy of one)

This comment has been minimized.

Copy link
@mt-inside

mt-inside Oct 16, 2018

I really like this too. +1 for trying to bring this into the Cluster object ASAP though, to avoid forking.

1. discuss and specify add-on definition objects and repository layout
2. discuss and specify how different config management tools would integrate (Helm, K/Jsonnet, kubegen, kustomize)
3. implement ability to
- install a built-in add-on (must ensure upgrades are catered for)

This comment has been minimized.

Copy link
@mt-inside

mt-inside Oct 16, 2018

Do you have a list of initial target addons? Flux? Tiller? Istio?

@errordeveloper

This comment has been minimized.

Copy link
Member Author

commented Nov 2, 2018

I'll probably get back to this only after I'm back from my vocation, and that'd be in December. Just FYI.

@errordeveloper errordeveloper force-pushed the proposal-001 branch from f8c0ff4 to ce66b57 Nov 6, 2018

cluster provider.

### kubernetes/kops

This comment has been minimized.

Copy link
@errordeveloper

errordeveloper Nov 8, 2018

Author Member

TIL: kops doesn't have a way to easily upgrade only network add-ons (or at least it's not documented); it happens to be part of cluster upgrade... we should verify this, of course :)

@yhvh

This comment has been minimized.

Copy link

commented Nov 26, 2018

I vote against this feature, too much complexity for such a small gain.
I would prefer the project maintained a tighter focus.

@mumoshu

This comment has been minimized.

Copy link
Collaborator

commented Nov 30, 2018

I think I have the same sentiment as @yhvh. The specs looks awesome, but the gain seems so small.

As long as eksctl is able to create correct IAM role/permissions for well-known addons, I don't mind if it doesn't actually install k8s resource objects.

I'd just write some bash/pulumi/helmfile/etc to install them.

@cdenneen

This comment has been minimized.

Copy link

commented Jan 30, 2019

@yhvh @mumoshu Can you elaborate on the small gain here? The result I believe will allow you to create a cluster (new, replacement, standby, etc) and have many of the addons automatically install, configure in a repeatable/consistent fashion. This married with GitOps proposal will allow clusters to be built with completeness. I think this proposal is very useful and has tremendous potential gains.

To build a cluster (minimal) THEN apply pieces to enhance it (cluster autoscaler, hpa, helm, kube2iam, weave-flux, etc) and THEN to deploy your services to it (jenkins, gitlab, artifactory, etc) much less your own helm charts for app1, app2, app3.

With having the addons + GitOps implemented you'll have the ability to build a cluster that has weave networking, IAM policy changes, cluster autoscaler, flux, etc as well as jenkins, gitlab, artifactory and app1/app2 deployed... especially in a disaster scenario you are "back in business". In a non-disaster scenario you stand up replacement cluster and then cut over. If you implement additional pieces like @istio and px-motion (@portworx) you could do so much more.

@prageethw

This comment has been minimized.

Copy link
Contributor

commented Jan 30, 2019

my view also addons gonna be an extra burden and quickly can go out of date or will be a too much hassle maintaining them, a great example is kops addons, I found they are out of date whereas helm moved on a long time ago with new patches, versions etc.

@cdenneen

This comment has been minimized.

Copy link

commented Jan 31, 2019

@prageethw

This comment has been minimized.

Copy link
Contributor

commented Jan 31, 2019

Addons would be helm based and if your using GitOps then any updates you do should be in git and therefore should be whatever version you specified. If you manually install something like cluster auto scaler via helm you would have to update the helm release manually same as via a config for eksctl. The addons aren’t eksctl specific they are using helm charts.

I agree on one thing that if addons introduced to eksctl it has to be based on helm, but what puzzles me where the real value we see on addons if that's the case. To me, kops add-on is an old concept they adopted before helm became mainstream.

One reason I like to keep plain old k8s separate from all other extra such as metrics server, ca, etc because at any time they add-ons can go broke against particular k8s version, kops or even eks. This is something we face in prod every day we never upgrade helm charts until we are highly confident it will work after some extensive testing, so it is often required whether a particular version of the addon is compatible with specific k8s or eks,kops or whatever minor differences on them.

so even we have add-ons we ultimately will end up in helm to manage such situations hence my argument it is not an ideal concept, now if add-ons would be part of EKS itself i find it makes more sense as it will be tested before a major release, (still I found giving full control to individual user what add-ons they wanna install more measured approach)

@mumoshu

This comment has been minimized.

Copy link
Collaborator

commented Jan 31, 2019

@cdenneen Hey! Thanks for the response.

Indeed, the concept of add-ons is great, a tool that declaratively manages a cluster and its apps as a whole is awesome, GitOps is the way to go, I believe too. I think we are on the same page.

It is just that doing it in the scope of eksctl, especially as the form of "add-ons", is just an implementation detail to me. And "add-ons" seemed not a good implementation to me.

Add-ons are great if and only if it is well maintained. And it turned out to be not easy to maintain them. Honestly saying even a big project, like kops, who should have so many users and contributors, seems to have failed maintaining it well. Why repeat the same thing that failed before, without major improvements(I suppose. Sry if I missed that it is covered in this proposal) over the last trial?

Also, add-ons is not a pre-requisite for GitOps. For example, I'm currently trying to apply GitOps to both clusters and apps(helm releases). I'd just use eksctl to declaratively manage clusters, and helmfile to declaratively manage apps.

I'm developing an another high-level command that accepts a config file that covers both clusters and apps, calls eksctl and helmfile. My GitOps pipeline implemented by atlantis/codefresh/brigade/etc the command. Implementation-wise, you can use whatever tools that match your use-case. It is true that the market of such high-level commands is still small, but it doesn't mean eksctl should be one.

So, you do need a tool to declaratively manage everything including clusters and apps, and probably a CD solution(whether it is GitOps flavored or not, it doesn't matter) that supports it. But the tool should not necessarily be eksctl.

@cdenneen

This comment has been minimized.

Copy link

commented Jan 31, 2019

@mumoshu

This comment has been minimized.

Copy link
Collaborator

commented Jan 31, 2019

Thanks again for sharing your idea!

I think having this under one umbrella truly helps keeps things easier to maintain

I completely agree with you :)

You could spin up brand new cluster, ALL your config and apps, run full test suite and determine if everything works as a whole

Yep, this is exactly the same U/X I want to achieve via my own high-level command and GitOps pipelines.

I'd also like to:

  • Deploy my own apps along with eksctl add-ons, so that I don't need to script anything to be run after eksctl
  • Regularly update only my apps, not eksctl add-ons, after the initial cluster bootstrap
  • In case one of add-ons is out-dated, I'd like to update it out of eksctl
  • Include terraform/cloudformation variables in my declarative "clusterr" config to manage cloud-provider resources along with my cluster (on top of the cloud-provider resources) and apps(on top of clusters).
  • Have a reusable GitOps pipeline or its framework implemented as a Kubernetes operator, or something similar that runs on a serverless platform like Lambda or Fargate. Then we don't need to reimplement the whole GitOps pipelines on top of various CI/CD services, platforms, etc.

I wish we could come up with a generic solution someday!

dholbach and others added 2 commits May 10, 2019
@whereisaaron

This comment has been minimized.

Copy link

commented May 12, 2019

My add-ons design proposal it don’t do it 😄 Every project you ‘add-on’ you might as well volunteer to maintain those projects too. If eksctl installs them, eksctl on the hook to patch and upgrade them for every release of those projects too. Every if just firing off helm charts you are still tracking versions and breaking changes to chart values settings.

If you think a project is utterly essential cluster bootstrap material, and can’t be installed and maintained separately by other tools, like maybe EKS AMI, CNI plug-in, core-dns. But anything that can be installed (and upgraded) by your choice(s) of helm/brigade/kubectl/octopus/cfn/terraform/etc. after eksctl bootstrap should IMHO not be part of eksctl.

I could see maybekiam/kube2iam could be integrated in to eksctl bootstrapping, given the AWS IAM policy integration required, which is a legit eksctl bootstrapping role; It would be nice if eksctl handles everything up to point that users could do the rest with their choice of k8s native tools, but no further.

I have used other k8s cluster creation systems that have ‘add ons’, like GKE and kube-aws and every time I have used the add ons I’ve regretted it later due some maintenance drama upgrading or reconfiguring the add-on later.

Also, everyone’s stack of in-cluster add-ons is different and integrated and secured in different ways. Currently supporting everyone’s node/ASG/VPC bootstrapping desires is challenge enough for eksctl without trying to be the kitchen sink of all k8s add-on services combinations.

Totally not my decision but that my personal opinion. I think it somewhat aligns with what @mumoshu is saying, but he is just way more polite 😉

@mumoshu

This comment has been minimized.

Copy link
Collaborator

commented Jun 3, 2019

@errordeveloper @christopherhein @stefanprodan @whereisaaron FYI, there's a relevant discussion in SIG Cluster Lifecycle to form a shared interface for addon installation and overall lifecycle kubernetes-sigs/addon-operators#10

@errordeveloper

This comment has been minimized.

Copy link
Member Author

commented Jun 3, 2019

there's a relevant discussion in SIG Cluster Lifecycle to form a shared interface for addon installation and overall lifecycle kubernetes-sigs/addon-operators#10

Yes, indeed. My original intention was to develop something under eksctl umbrella first, and them propose it upstream. However, the upstream conversation has started before we started to work on anything.

I've been meaning to share an update here and close the PR. Myself and other colleagues of mine will continue to take active part in upstream discussions, and once a clear definition of a spec emerges, we are going to look at how it can be implemented in eksctl.

@vladfr vladfr referenced this pull request Jun 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.