Proposal: API endpoint for listing nodes in cluster

[ahmetb]

Question: Is there a plan for building an API endpoint (specific to Swarm) that would list all the active/healthy nodes in the cluster? This could help a great deal in building tools/functionality on top of Swarm. I'm not sure if this is discussed before but it sounds like a having this would be nice-to-have yet harmless to me.

[garagatyi]

Would it help?

https://docs.docker.com/swarm/

Once the manager is running, check your configuration by running docker info as follows:

docker -H tcp://<manager_ip:manager_port> info

$ docker -H tcp://0.0.0.0:2375 info
Containers: 0
Nodes: 3
agent-2: 172.31.40.102:2375
└ Containers: 0
└ Reserved CPUs: 0 / 1
└ Reserved Memory: 0 B / 514.5 MiB
agent-1: 172.31.40.101:2375
└ Containers: 0
└ Reserved CPUs: 0 / 1
└ Reserved Memory: 0 B / 514.5 MiB
agent-0: 172.31.40.100:2375
└ Containers: 0
└ Reserved CPUs: 0 / 1`

└ Reserved Memory: 0 B / 514.5 MiB

[ahmetb]

OK, it looks like /info provides through some hacks (like pushing info down to DriverStatus) and thus making the output look like what you pasteed above. It didn't cross my mind that I could find this at /info endpoint.

➜  ~  curl -s http://localhost:2375/info | jq '.'
{
  "Containers": 149,
  "DriverStatus": [
    [
      "\bStrategy",
      "random"
    ],
    [
      "\bFilters",
      "affinity, health, constraint, port, dependency"
    ],
    [
      "\bNodes",
      "2"
    ],
    [
      "hostname1",
      "hostname1:2376"
    ],
    "..."],
  "NEventsListener": 0,
  "Debug": false
}

It is not documented at https://docs.docker.com/swarm/API/#endpoints-which-behave-differently –it probably should be, because it's essentially behaving quite differently than any normal docker engine.

Also it looks like this endpoint is somewhat hacky and designed to make human-readable docker info look nice.

Something like GET /nodes still would be nice and easily parseable¹ by programs and could potentially report more info (e.g. health, or other stuff stored about the node in swarm) in a well-presented way.

¹: for instance in the json response above, it's really hard to get <hostname, IP:port> pairs from the DriverStauts array, this is far too hacky and can easily break.

[garagatyi]

I agree that it's is not well parsable. Actually I had some issues with that because addition labels (I suppose not only labels) changes this output and I have to go though all items and search for Nodes entry.

+1 to add swarm specific API.

[vieux]

@ahmetalpbalkan no, we want to abstract nodes from the API, for example when using swarm with the mesos scheduler backend, there is no concept of nodes.

[ahmetb]

@vieux I see. Is that the case because the Mesos framework for Swarm would advertise the available resources (i.e. nodes) from Swarm to Mesos (instead of Mesos asking Swarm).

I understand the concern that Swarm must abstract the nodes in "Docker Remote API", but how about a Swarm-specific API (not a superset/subset of Docker Remote API) that provides info about the Swarm cluster itself and its nodes?

If someone was building another sort of tool on top of Swarm, and they need list of nodes, is there any way they could get that?

[yoanisgil]

As @ahmetalpbalkan mentioned this is a feature which could be very useful for tools looking to integrate with Swarm. As an example I am working on heapster source implementation for Swarm (see here for details: kubernetes-retired/heapster#367) and right now we have not way of getting the list of nodes other than running swarm list.

@vieux if the concept of node does not exist when using swarm with the mesos scheduler backend, then what would be the output of a call to swarm list (on a Mesos cluster of course)?

[chanwit]

I also would like to have an API for selecting and returning a node (not placing the new container just selecting).

This is ver useful for the cluster-wide build process.

[yoanisgil]

Maybe we should put together a list of pros and cons so we can reach a
consensus and decide whether this something should be not or not. A few use
case might help as well.

Bests,

Yoanis

Le vendredi 26 juin 2015, Chanwit Kaewkasi notifications@github.com a
écrit :

I also would like to have an API for selecting and returning a node (not
placing the new container just selecting).

This is ver useful for the cluster-wide build process.

—
Reply to this email directly or view it on GitHub
#735 (comment).

[chanwit]

Following these. I +1 for

• GET /nodes for listing nodes returning <host names, addresses>.
• GET /candidate for selecting the most appropriate candidate with all filter applied. This makes Swarm act as the front-end scheduler for client such as docker-compose. GET /candidate should accept parameters similar to docker run. Basically, this endpoint is the dry version of the run command.

[ahmetb]

Sorry for not following up on this discussion I started earlier. :-)

@vieux what is being asked here is not related to Docker Remote API at all. And Mesos might be working without this piece of info. What we're proposing a different set of API endpoints (that look like Remote API of course) which provide various info about cluster. This could be simply for building GUIs/visualizations on top of Swarm, collecting data about Swarm, or orchestrating/extending Swarm further.

At the end, Swarm API will be a superset of Docker Remote API, and will include a few extra endpoints. Based on @chanwit's suggestion, I would like to propose having GET /cluster/nodes as well, and include various info (addr, hostname, cpu usage, mem usage, ..., NContainers, NImages).

What do maintainers think about this?

[chanwit]

@vieux these kinds of change are really important for us to unlock the barrier between platforms using Swarm.

For example we can GET /candidate to select the node, then issue the build command with the build file, say Dockerfile.windows_amd64, if the node runs Windows OS.

[yoanisgil]

Following up with what @ahmetalpbalkan and @chanwit endpoints such as /nodes, which provides the list of nodes on the cluster alone with the information describing some of the nodes characteristics (CPU, RAM, etc) will ease the integration with monitoring tools such as heapster.

For instance I'd like to implement a heapster metrics source which integrates with Swam. The very first step towards this is to get the list of nodes available in the cluster. This endpoint will be called on a regular basis since nodes can come and go. Right now there is no reliable/consistent/standard way of doing this other than parsing the output from docker info on a cluster Swarm.

Also I don't think adding and endpoint to get the list of nodes will do any harm, nor I think its something entirely new, specially when you consider that the command docker info already provides this sort of information. So why not make it available through an API endpoint? This is also inline, I think, with the way Docker itself works since all communication between the docker client and the docker daemon is through a RESTful API.

[aluzzardi]

I tend to agree with this.

We had a bunch of conversations at DockerCon around APIs with many contributors (including @icecrime and @shykes).

One of the ideas we've been toying around (not sure everyone agrees) is that the current Docker API could become the Container Management API, which both the Engine and Swarm implement.

Swarm would also implement the Cluster Management API, which could provide among other things /nodes. The Engine would not implement such API.

Thoughts?

[yoanisgil]

@aluzzardi Not sure what exactly you mean by Container Management API nor by Cluster Management API. Is this some sort of documentation which already exists? I am sorry if I'm missing something obvious.

Generally speaking, it would be nice to have such APIs. I imagine what you call the Cluster Management API will be a superset, as many others has mentioned here, of the Container Management API. I guess some fair amount of work will need to be done to define those APIs, though it seems to me must of the work will be on the Cluster Management API.

Seems to me a fair set of questions to ask is:

• What is the Cluster Management API about?
• What endpoints, alone with the expected response for each one of them, will be exposed through the Cluster Management API?

[chanwit]

@aluzzardi thank you for the info. It seems that we can focus on defining the CMA then.

How should we start? I think all extra information provided by docker info via swarm could be a good starting point.

Actually, I did hack dockerclient (https://github.com/chanwit/dockerclient/blob/dev/types.go#L222) to add Node struct to the container info for these a couple days ago too.

[aluzzardi]

TL;DR Summary: Let's do this. I'm for /nodes or something similar (pending other maintainers approval), but let's take a step back for a moment and think about the big picture first.

@yoanisgil Container Management API is basically the Docker API - pretty much all it does is expose endpoints to manage containers (creation and operation).

Cluster Management is basically what is proposed here: listing nodes for starter and something like @chanwit mentioned to get container positioning.

But this could evolve in many ways: Moving/migrating containers, draining nodes (as in, moving all containers elsewhere for instance if the node has to be brought down for maintenance), adding/removing nodes, manipulating node labels, ...

Technically speaking, it doesn't change anything - it's just a matter of nomenclature. Instead of saying "the engine defines the Docker API which Swarm implements along with a super set", we'd say: here's the Docker API. It contains primitives to manage containers (current Docker API), clusters (future Swarm API), machines (Machine API?) and stacks (Compose API?). Swarm speaks both the Container and Cluster APIs while the Engine only the Container API.

In that world, maybe Swarm could eventually speak the Stack Composition API, meaning that it'd be able to digest Compose files directly and things such as docker-compose scale web=2 would be handled natively by Swarm in an atomic way.

I realize the conversation is quite meta but I promise it's going to materialize into short-term doable stuff.

/cc @ehazlett @aanand @nathanleclaire

[chanwit]

Yes, let's do this 👍

[abronan]

Too bad no one can see me behind my keyboard because I'm +1'ing this like crazy. Even though this should be something worth of a Proposal I guess?

[ehazlett]

+1 from me as well :)

On Tue, Jun 30, 2015 at 12:40 AM, Alexandre Beslic <notifications@github.com

wrote:

Too bad no one can see me behind my keyboard because I'm +1'ing this like
crazy. Even though this should be something worth of a Proposal I guess?

—
Reply to this email directly or view it on GitHub
#735 (comment).

[yoanisgil]

I am all in for this. Personally, I'd love to see Swarm be able to natively handle Compose ;), given that I am currently working on a deployment procedure based on Compose.

I think at the very least the Container Management API should provide node discovery and management. Just wondering what's next ?

[shykes]

Hi everyone, sorry for not being able to participate more (currently on
vacation) but please don't forget an additional parameter: that we should
start thinking of the docker daemon as a special case of the swarm daemon.

Both daemons perform the same fundamental task: orchestrate
various available resources (the "backends") through a standardized
frontend interface.

In the single-host daemon, the backends are processes, directories, network
interfaces and other resources exposed by a local Linux (and soon Windows)
kernel. In the swarm daemon the backends are an aggregate of single-host
daemons.

In both cases there is the need to express desired state vs actual state,
assign globally unique identifiers to resources, resolve scheduling
constraints (eg. only 1 container may receive traffic on port 80 on
internet-facing $IP).

I completely support the idea of a new cluster-aware API. I also would like
us to use that API in both daemons. Even better, what we today know as "the
docker daemon" and "the swarm daemon" should become a single daemon, with
different backend configurations.

To support this use case, I would also like to request first-class support
for aggregation. The API should expose all resources (containers, volumes,
networks, hosts etc) as a flat-list of globally unique IDs. This means that
aggregation of arbitrary depth becomes much more useful. For example
multi-datacenter deployments.

Sorry for the wall of text.

On Tuesday, June 30, 2015, Yoanis Gil notifications@github.com wrote:

I am all in for this. Personally, I'd love to see Swarm be able to
natively handle Compose ;), given that I am currently working on a
deployment procedure based on Compose.

I think at the very least the Container Management API should provide
node discovery and management. Just wondering what's next ?

—
Reply to this email directly or view it on GitHub
#735 (comment).

[yoanisgil]

@shykes there quite a few interesting things in your comment:

• docker daemon as a special case of the swarm daemon: This will help a lot to unify Swarm's and Docker`s API into a single API. Both project's could greatly benefit from it since this could be refactored out into some sort of common lib (which might already by the case ;)). However I do think there is one thing to keep an eye on: this should not add any complexity to the existing procedure/flow to spin off a Docker Daemon in a single machine. One of the things I love above Docker is that it takes no time to have a working environment, which could be reduced to downloading the binary and firing up the daemon. Swarm is a different beast, even with the help of Machine which takes care of most of the heavy lifting. Once you start working with Swarm one cannot bypass some key concepts like manager, nodes, etc. So I believe this should be abstracted away from a client point of view.
• In both cases there is the need to express desired state vs actual state: Not sure what you meant by this but I believe there are better tools to handle this, like Kubernetes for instance with their concepts of pods, replicas and services.

I am quite new to Docker but I would really love to see this work. Maybe we could start by creating a Draft of what endpoints are required to render the Swarm API more cluster aware. This document could define some sort of roadmap which in turn will help people know how to help and contribute to this effort.

[ahmetb]

Since Swarm now supports multi-manager set up, the managers also have a knowledge of leader and followers. Therefore I'm extending the proposal:

• GET /cluster/managers: Returns a list of nodes along with their addr and leader/follower status.

Also can we mark this issue as proposal rather than question? It's been nearly 4 months and we can probably bring this issue to closure either by accepting the proposal or by declining it.

[Pensu]

+1 for adding specific stuff to swarm API. I think that would help manager make smarter decision as the scheduler becomes more mature.

[batmat]

Sorry if I rehash discussions here, or am not on the right issue. Please tell me if so. I hope to help clarify things for potential newcomers here later who will, like me, arrive here after searching/googling ;-)

Hi, currently trying to monitor how full my swam cluster is. Where now & in the future should I try to retrieve that information? My goal is to be able to be alerted of potential issues with stale containers filling the cluster, or be able to react to that event by adding nodes to the swarm cluster.

I would like to avoid monitoring the nodes themselves because it seems to me like a kind of waste: swarm already has to know if there's still room because I use binpack [hence, that would be rewritting an existing logic].

My impression is that I should currently try and parse the /info URL (or the docker info output). Is the plan that in the future there would more machine-parseable informations for that under a new /nodes IIUC?

Thanks

[shenshouer]

+1

[dvenza]

+1
We are building on top of Swarm and we need to have access to the data about nodes (how many, health), reserved memory and labels.

[raarts]

Any progress on this?

[ViBiOh]

👍

[ahmetb]

Finally some serious stuff happened in v1.12 so we're friends again. ❤️

[abronan]

@ahmetalpbalkan Thanks for taking care of closing the now irrelevant issues, much appreciated. ❤️