Introduce etcd petset

[ingvagabund]

Example of etcd cluster via petsets

---

This change is

[ncdc]

@bprashanth @smarterclayton PTAL

[bprashanth]

it would be great if you could do this in an init container and write the config to some hostPath shared location that etcd reads on startup

[ingvagabund]

You can't. The /var/run/etcd/meta is generated after the cluster is bootstrapped. The file contains a member id which is know after the cluster started.

[bprashanth]

Seems like there're a lot of similarities with this and the zookeeper setup, would be nice to duplicate the pattern if possible (https://github.com/kubernetes/contrib/tree/master/pets/zookeeper). It would also make scaling easier.

[tbg]

I tried running this for a while to no avail (all on 1.3) until I realized that I needed to manually create the PVCs expected by the cluster ahead of time. Is that the expected behavior? I would've expected that the necessary PVCs be auto-created (and automatically filled as long as a PV is available). Instead, creating the PetSet before the PVC does create a "zombie" PVC which is never filled, regardless of what I do.

Sorry about asking here, but I've looked around and wasn't able to find any resources on this. I'm sure they exist somewhere; appreciate a pointer.

[bprashanth]

@ingvagabund if you actually want to invoke the dynamic provisioner you need to specify https://github.com/kubernetes/kubernetes.github.io/blob/release-1.3/docs/user-guide/petset.yaml#L43, the way you currently have it, the petset is asking for a specific pvc

@tschottdorf see 4 in https://github.com/kubernetes/kubernetes.github.io/blob/release-1.3/docs/user-guide/petset.md#alpha-limitations and https://github.com/kubernetes/kubernetes.github.io/blob/release-1.3/docs/user-guide/petset.md#stable-storage

The provisioners are started based on your environment. From https://github.com/kubernetes/kubernetes/blob/release-1.3/examples/experimental/persistent-volume-provisioning/README.md

No configuration is required by the admin! 3 cloud providers will be provided in the alpha version of this feature: EBS, GCE, and Cinder.

[smarterclayton]

As noted on the cockroach thread you can do:

kind: PersistentVolume
apiVersion: v1
metadata:
  name: pv-host-1
spec:
  hostPath:
    path: /var/lib/random/dir-1

as necessary (you may have to set capacity and length).

[bprashanth]

@ingvagabund did #1295 (comment) work?

[ingvagabund]

@bprashanth I will check that out. Sorry for the delay.

[ingvagabund]

@bprashanth the etcd petset is now scalable. Though, I have hit some issues when scaling up and down:

• When running kubectl patch petset/etcd -p '{"spec":{"replicas": 3}}' from 5 to 3 replicas, pets etcd-4 and etcd-5 were not deleted.
• When running kubectl edit and changing number of replicas from 13 to 5, pets etcd-5 and etcd-6 were not deleted. After removing etcd-6, both pets got deleted.

It does not have to be necessary related to the petset controller. Maybe something wrong in the specification.

[sttts]

kubectl scale?

[ingvagabund]

$ kubectl scale --replicas=5 petset/etcd
error: no scaler has been implemented for {"apps" "PetSet"}

[ingvagabund]

Scaling up from 3 to 13 and back, the middle state of the petset is

NAME                 READY     STATUS    RESTARTS   AGE
etcd-0               1/1       Running   0          3m
etcd-1               1/1       Running   0          3m
etcd-11              1/1       Running   0          3m
etcd-12              1/1       Running   0          3m
etcd-2               1/1       Running   0          3m
etcd-3               1/1       Running   1          3m

Don't think pets etcd-12 and etcd-11 should exist if etcd-10 is gone.

[bprashanth]

Scaling will stop if a pet is unhealthy. It's possible that there are issues that need to be fixed when scaling halts because of an unhealth pet, and then the user scales up/down. The behavior currently is undefined in such situations. Scaling up should happen in order, once the petset is stable, scaling down should happen in reverse order. kubernetes/kubernetes#30082

[bprashanth]

this stuff seems like exactly the type of thing to put in the init container (see zookeeper example:), any reason it should be in entrypoint? https://github.com/kubernetes/contrib/tree/master/pets/zookeeper/init. Putting it in init allows one to maintain both images independently, but I'm ok doing it this way if there are some limitations.

[bprashanth]

putting it in init also gives you a clear signal that bootstrapping/joining cluster failed

[ingvagabund]

this stuff seems like exactly the type of thing to put in the init container (see zookeeper example:), any reason it should be in entrypoint?

There is nothing to install, setup/configure or bootstrap. Initial cluster bootstrapping consists of waiting for initial pets. The same holds for the adding post-bootstrap members. The script "just" adds new member and start etcd.

Putting it in init allows one to maintain both images independently, but I'm ok doing it this way if there are some limitations.

As all the code is basically written over etcd image, there is no need for additional one.

putting it in init also gives you a clear signal that bootstrapping/joining cluster failed

Would not be postStart better? Basically, the main container command itself is to run etcd. The current code "just" collects required envs, params, puts them together and start etcd. So one either add member, remove member or update member. That is the only piece of logic worth putting into initialization part. Unfortunately, all three etcd commands differs in input options which are populated during initialization. So separation of initialization code and executing etcd itself does not make sense.

[lachie83]

Both are valid cases. I guess the question comes down to consistency

[bprashanth]

I'm planning on waiting for consensus on https://groups.google.com/forum/#!topic/kubernetes-dev/MhQl2GBylgM (I think we're close) before doing a next pass. Hoping we can directly check this in there. Let me know if you'd rather check it in here and transition instead and I can do a review.

[ingvagabund]

I am fine with waiting. At the same time (and as was commented in the discussion) we need some sort of CI to at least test 1) correctness and 2) regressions.

[mdshuai]

@ingvagabund Is the etcd pod must run as root and in etcd namespaces?

[ingvagabund]

@ingvagabund Is the etcd pod must run as root and in etcd namespaces?

Etcd process runs as a root by default unless specified otherwise. Not aware of any etcd namespace requirement. What is your usecase here to test?

[mdshuai]

{HOSTNAME}.${PETSET_NAME} can't be resolve by dns. should be {HOSTNAME}.${SERVICE_NAME}, in this example, you set petse-tname equal with svc-name so it can running well. Need add another env SERVER_NAME to replace some fields.

[ingvagabund]

Can you dump entries in your DNS? I have never had this problem.

[mdshuai]

In this design doc https://github.com/bprashanth/kubernetes.github.io/blob/4df7d96369964a267466bdffc73f7d343c77584f/docs/user-guide/petset.md#network-identity As each pet is created, it gets a matching DNS subdomain, taking the form: $(petname).$(governing service domain), where the governing service is defined by the serviceName field on the Pet Set.
If you set petsetname don't equal to svc name, then you can see this issue.

[test-foxish]

recomputing cla status...

[k8s-github-robot]

[CLA-PING] @ingvagabund

Thanks for your pull request. It looks like this may be your first contribution to a CNCF open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://identity.linuxfoundation.org/projects/cncf to sign.

Once you've signed, please reply here (e.g. "I signed it!") and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.

[k8s-github-robot]

[CLA-PING] @ingvagabund

Thanks for your pull request. It looks like this may be your first contribution to a CNCF open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://identity.linuxfoundation.org/projects/cncf to sign.

Once you've signed, please reply here (e.g. "I signed it!") and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.

[k8s-github-robot]

[APPROVALNOTIFIER] The Following OWNERS Files Need Approval:

• pets

We suggest the following people:
cc @bprashanth
You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancelin a comment

[foxish]

Closing this, it should probably exist as an example outside contrib.