Skip to content

Commit

Permalink
Adds a mechanism for Kubernetes StatefulSet, guess id from hostname
Browse files Browse the repository at this point in the history
  • Loading branch information
@solsson
solsson committed Dec 25, 2016
1 parent 3c5e7be commit df9474f
Showing 1 changed file with 6 additions and 0 deletions.
6 changes: 6 additions & 0 deletions 3.4.9/docker-entrypoint.sh
View file
Expand Up @@ -27,6 +27,12 @@ fi

# Write myid only if it doesn't exist
if [ ! -f "$ZOO_DATA_DIR/myid" ]; then
if [ -z "$ZOO_MY_ID" ]; then
ZOO_MY_ID=$(($(hostname | sed s/.*-//) + 1))
echo "Guessed server id: $ZOO_MY_ID"
# Tries to bind to it's own server entry, which won't work with names ("Exception while listening java.net.SocketException: Unresolved address")
sed -i s/server\.$ZOO_MY_ID\=[a-z0-9.-]*/server.$ZOO_MY_ID=0.0.0.0/ "$ZOO_CONF_DIR/zoo.cfg"
fi
echo "${ZOO_MY_ID:-1}" > "$ZOO_DATA_DIR/myid"
fi

Expand Down

9 comments on commit df9474f

@deitch
Copy link

@deitch deitch commented on df9474f Jan 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You really should submit this as a Pull Request for the official zookeeper repo, maybe with a variable flag such as EXTRACT_ID_FROM_ORDINAL (or something with a better name)

@solsson
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@deitch What is your use case for this fix? I thought that maybe there are more generic ways to do this, but that I just didn't know enough about zk.

@deitch
Copy link

@deitch deitch commented on df9474f Feb 13, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trying to think it through.

zk requires a unique ID for each server that starts up. When running three individual servers, it is easy to just set them. But doing it the right way, with a StatefulSet, means you state one spec with however many replicas you want. If the spec is identical, the ZOO_MY_ID will be too, and all three (or however many replicas you have) will have same ID, and zk will fail.

StatefulSets attach an ordinal to each replica (0, 1, 2...). It also gives each container a unique hostname by appending -ordinal. So you can get the sequential ID by extracting it out of the hostname.

The right thing to do, of course, is to have an env var set dynamically for the ordinal. k8s 1.5 does not support it, but I think they are targeting 1.6. See kubernetes/kubernetes#40651

@solsson
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, that was my understanding too. If I remember correctly, the first zookeeper ID must be 1. That means we still need some entrypoint work with valueFrom fieldRef fieldPath, don't we?

@dminkovsky
Copy link

@dminkovsky dminkovsky commented on df9474f Feb 13, 2017 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@deitch
Copy link

@deitch deitch commented on df9474f Feb 13, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That means we still need some entrypoint work with valueFrom fieldRef fieldPath, don't we?

Once they have the sets in place, we don't need the above patch. We can set the ordinal to an env var. I would do something like this:

        env:
        - name: ZOO_MY_ID
          valueFrom:
            fieldRef:
              fieldPath: metadata.annotations['spec.pod.beta.kubernetes.io/statefulset-index']

@deitch
Copy link

@deitch deitch commented on df9474f Feb 13, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I discovered kubernetes/helm and kubernetes/charts a few days ago

Perhaps. They still need an image that behaves nicely when run in the k8s environment. Those are just config packaging, but they still reference the image itself.

@solsson
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That means we still need some entrypoint work with valueFrom fieldRef fieldPath, don't we?

Once they have the sets in place, we don't need the above patch. We can set the ordinal to an env var.

But you'd get ZOO_MY_ID=0 which is invalid.

@deitch
Copy link

@deitch deitch commented on df9474f Feb 13, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But you'd get ZOO_MY_ID=0 which is invalid.

Oh yeah. Nice catch.

Please sign in to comment.