New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add KEP for etcdadm #2835

Merged
merged 2 commits into from Nov 22, 2018

Conversation

@justinsb
Copy link
Member

justinsb commented Oct 22, 2018

No description provided.

@justinsb

This comment has been minimized.

Copy link
Member

justinsb commented Oct 22, 2018

@justinsb

This comment has been minimized.

Copy link
Member

justinsb commented Oct 22, 2018

/assign @roberthbailey
/assign @timothysc

(when github catches up :-) )

@justinsb justinsb force-pushed the justinsb:kep_etcdadm branch from dee32b2 to e74add5 Oct 22, 2018

@justinsb justinsb force-pushed the justinsb:kep_etcdadm branch from e74add5 to 1a7d5d9 Oct 22, 2018

@neolit123
Copy link
Member

neolit123 left a comment

LGTM thanks!

@timothysc
Copy link
Member

timothysc left a comment

/approve
/hold

Wait for feedback and vote, but generally LGTM


etcdadm gives us easy to use CLI commands, which will form the base layer of
operation. Automation should ideally describe what it is doing in terms of
etcdadm commands, though we will also expose etcdadm as a go-library for easier

This comment has been minimized.

@timothysc

This comment has been minimized.

@fabriziopandini
etcd-manager works by coordinating via a shared filesystem-like store (e.g. S3
or GCS) and/or via cloud APIs (e.g. EC2 or GCE). In doing so it is able to
automate the manual commands, which is very handy for running in a cloud
environment like AWS or GCE.

This comment has been minimized.

@timothysc

timothysc Oct 22, 2018

Member

I still think using the bootstrap token to encrypt and store the certs as secrets also makes a lot of sense. It keeps it local to the cluster without adding the dependencies and also expires after a period of 24-hours by default to eliminate several of the security concerns.

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Oct 22, 2018

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: timothysc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jpbetz

This comment has been minimized.

Copy link
Contributor

jpbetz commented Oct 22, 2018

* Cluster backups
* Disaster recovery or restore from backup
* Cluster upgrades
* Cluster downgrades

This comment has been minimized.

@gyuho

gyuho Oct 22, 2018

Member

Currently downgrade is not supported by etcd, but being added to v3.4 etcd-io/etcd#7308. Please ping us if you need any help. Thanks!

This comment has been minimized.

@justinsb

justinsb Oct 22, 2018

Member

It will be great if etcd can support it natively! Currently kopeio/etcd-manager implements it via a backup/restore, with a key-by-key copy. We can obviously be smarter about that, but it should work anywhere (it seems to even work for etcd3 -> etcd2)

This comment has been minimized.

@gyuho

gyuho Oct 22, 2018

Member

It will be great if etcd can support it natively!

Yes, @wenjiaswe is working on it :)

This comment has been minimized.

@wenjiaswe

wenjiaswe Oct 22, 2018

@justinsb yes, I am working on this and here is "etcd downgrad design" documentation. Here is the items planned. I haven't tried kopeio/etcd-manager but I will try it out. I think it's good that it works the way it is and etcdadm could use it before etcd downgrade is supported. Meanwhile, shall we sync on eligibility of integration of etcd native downgrade with etcdadm?

This comment has been minimized.

@justinsb

justinsb Oct 23, 2018

Member

@wenjiaswe thanks & absolutely. I don't think there's any question that when etcd supports downgrade natively we should prefer that option :-) (For expediency in kopeio/etcd-manager all upgrades involve a key/value copy today, but I'll fix that for the upgrades that etcd does support - it's easier to have one code path, but it is very sub-optimal).

But we should definitely sync - for example, today we put etcd into "read-only" mode by switching ports. That lets an HA cluster stay up, but means we know that apiserver won't be writing to it. But ... it's not the cleanest solution, and this is another thing that it would be wonderful to have native support for. But again: not a real blocker.

The real wishlist is for non-voting cluster members - that would make automatic management much safer. But I understand that is coming to etcd as well 🎉


This results in an multi-node ("HA") etcd cluster.

#### Automatic Cluster Creation

This comment has been minimized.

@danbeaulieu

danbeaulieu Oct 22, 2018

I'm interested in hearing where the line is drawn between this KEP and the current functionality provided by kops. For instance kops has a very specific architecture (1 ASG per control plane node AZ with a min=max=desired=1, EBS volumes per control plane node AZ tagged in a specific way, etc). This architecture is optimized for etcd fault tolerance and DR. Does this KEP offer the option to have similar fault tolerance and DR functionality?

This comment has been minimized.

@justinsb

justinsb Oct 22, 2018

Member

So kopeio/etcd-manager was a reimplementation of the kops etcd management functionality. The intention is that this is a clean implementation that any installation tool can use, not just kops.

I'll clarify though that we're assuming that an external installation tool sets up the infrastructure itself if we're using EBS volumes - i.e. I don't think etcdadm should set up the volumes or the AWS ASGs or GCE MIGs that will likely provide the machines on which this runs. (Or that would should be a separate KEP if so!) It will make it very easy to set up those ASGs though, as they can all run the same command. I'll clarify this though, as I don't think I covered it sufficiently... etcdadm should (optionally) support auto-mounting of volumes IMO, but I think setting them up is best done externally.

We could bring this into scope, but I think it's better just to clearly document the requirements as opinions vary so widely here! (e.g. "if you're using volumes, pass the tags using the this flag, you probably want to put them in separate AZs, and you probably want to run in separate ASGs to guarantee equal zonal coverage")

<endpoint>` command
* On each other master machine, copy the CA certificate and key from one of the
other masters, then run the `etcdadm join <endpoint>` command.
* Run kubeadm following the [external etcd procedure](https://kubernetes.io/docs/setup/independent/high-availability/#external-etcd)

This comment has been minimized.

@danbeaulieu

danbeaulieu Oct 22, 2018

Is it possible for this tool to integrate with kubeadm in such a way that it produces "local" etcd clusters? There is something nice about etcd running as pods in the cluster as it allows reuse of k8s based tooling for monitoring, logging, metrics, etc

This comment has been minimized.

@justinsb

justinsb Oct 22, 2018

Member

Yes, I'd imagine kubeadm could easily replace its built-in etcd management with a call-out to etcdadm.

And yes, I agree that pods in the cluster is the only configuration that we test today and so it's the one I personally feel most comfortable with. Hopefully we can add more e2e configurations going forward though!

@justinsb

This comment has been minimized.

Copy link
Member

justinsb commented Oct 23, 2018

I added a commit to try to more clearly express why cloud interaction (volume mounting) should be in scope, but more importantly to bound that most cloud interaction will be out of scope :-)

@fabriziopandini
Copy link
Contributor

fabriziopandini left a comment

Looking forward for this!

* Cluster upgrades
* Cluster downgrades
* PKI management

This comment has been minimized.

@fabriziopandini

fabriziopandini Oct 23, 2018

Contributor

IMO a really useful task will be "pivoting" from kubeadm local etcd to etcdadm managed etcd, thus providing the user a way forward from simplest etcd clusters to something more complex


etcdadm gives us easy to use CLI commands, which will form the base layer of
operation. Automation should ideally describe what it is doing in terms of
etcdadm commands, though we will also expose etcdadm as a go-library for easier

This comment has been minimized.

@fabriziopandini
@philips

This comment has been minimized.

Copy link
Contributor

philips commented Oct 23, 2018

Someone should email etcd-dev@googlegroups.com as a heads up to this effort. There might be other interested parties.

@justinsb

This comment has been minimized.

Copy link
Member

justinsb commented Oct 26, 2018

@timothysc

This comment has been minimized.

Copy link
Member

timothysc commented Oct 29, 2018

/lgtm

I'll let @roberthbailey cancel the hold on after the vote timeout.

@k8s-ci-robot k8s-ci-robot added the lgtm label Oct 29, 2018

### Implementation Details/Notes/Constraints

* There will be some changes needed to both platform9/etcdadm (e.g. etcd2
support) and kopeio/etcd-manager (to rebase on top of etcdadm).

This comment has been minimized.

@detiber

detiber Oct 29, 2018

Member

If this is targeting net-new usage why support etcd2 here? It seems like etcd3+ would be sufficient for new and future usage.

This comment has been minimized.

@justinsb

justinsb Oct 29, 2018

Member

For users that are still on etcd2. We're going to strand them in 1.13 otherwise.


### Implementation Details/Notes/Constraints

* There will be some changes needed to both platform9/etcdadm (e.g. etcd2

This comment has been minimized.

@dlipovetsky

dlipovetsky Oct 30, 2018

Contributor

My original sketch of etcdadm included an upgrade verb. It would be limited to making changes on the host where etcdadm is run. Should we add this as a note here?

@wenjiaswe wenjiaswe referenced this pull request Nov 8, 2018

Closed

REQUEST: New membership for @wenjiaswe #227

6 of 6 tasks complete
@justaugustus

This comment has been minimized.

Copy link
Member

justaugustus commented Nov 20, 2018

REMINDER: KEPs are moving to k/enhancements on November 30. Please attempt to merge this KEP before then to signal consensus.
For more details on this change, review this thread.

Any questions regarding this move should be directed to that thread and not asked on GitHub.

@neolit123

This comment has been minimized.

Copy link
Member

neolit123 commented Nov 21, 2018

/cc
to remind myself about this.

unless @roberthbailey unholds in the next few days i will do that.
this proposal was voted and approved.

@k8s-ci-robot k8s-ci-robot requested a review from neolit123 Nov 21, 2018

@roberthbailey

This comment has been minimized.

Copy link
Member

roberthbailey commented Nov 22, 2018

/hold cancel

I didn't realize that this was blocked on me. It passed the vote, so let's get it merged before the great KEP migration of 2018 commences.

@justaugustus

This comment has been minimized.

Copy link
Member

justaugustus commented Nov 22, 2018

Huzzah!!

@k8s-ci-robot k8s-ci-robot merged commit c5f3779 into kubernetes:master Nov 22, 2018

3 checks passed

cla/linuxfoundation justinsb authorized
Details
pull-community-verify Job succeeded.
Details
tide In merge pool.
Details

justaugustus pushed a commit to justaugustus/community that referenced this pull request Dec 1, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment