Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE REQUEST] Be able to use all the Portainer built-in functionalities in all the containers running in a swarm cluster #461

Closed
bvis opened this issue Jan 4, 2017 · 48 comments

Comments

Projects
None yet
@bvis
Copy link

commented Jan 4, 2017

Description

Currently when running under Docker Swarm mode portainer has visibility of services and their tasks, but from the task you cannot get the logs/ssh session/etc.

In the container tab you have only access to the containers running in the host where Portainer is connected. In case you want to get this functionality on containers running on other nodes of the cluster you need to add manually the connection details, which is not a valid solution in an elastic swarm cluster where nodes are added and removed continuously.

Proposal: From the service tags, when going to the tasks list, get the same functionalities than we currently have in the containers section.

Probably in the future Docker will add more built-in functionalities to support this natively, but I have not seen this feature in their roadmap. It shouldn't be so hard to get a working solution with not so much effort.

How to do it?

Thanks to the "global service" concept in Swarm mode we can run a service ensuring that each node is running a service task. In case the cluster adds more nodes is Swarm the responsible of launch a new task in the node, it's totally automatic.

Thanks to this and the possibility to expose the docker socket via a service it shouldn't be so hard to Portainer to connect to each task of this service.

Technical details:

Being more concrete, if I launch this configuration:

docker network create --driver overlay portainer

# It exposes the docker socket in the "portainer" network, you can reach each of them individually with the DNS name "tasks.docker-proxy".
docker service create --name docker-proxy \
    --mode global \
    --network portainer\
    --mount "type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock" \
    rancher/socat-docker

# Portainer with the proposed feature
docker \
  service create \
  --name portainer \
  --network portainer \
  --publish 9000:9000 \
  --constraint 'node.role == manager' \
  --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \
  portainer/portainer \
  --docker-proxy-service-name docker-proxy

This version of portainer is aware of the docker-proxy service, then internally doing a DNS call like:

nslookup tasks.docker-proxy

Name:      tasks.docker-proxy
Address 1: 10.0.0.3 docker-proxy.0.1qcw37fl32f57c8m8r05h0z6c.portainer
Address 2: 10.0.0.2 docker-proxy.1.aiof234849jfklasdfjlfjal2a0fja.portainer
...
Address 10: 10.0.0.24 docker-proxy.9.fjaoifj92fkjfoa9fjqka84rhra.portainer

Once knowing the endpoint you can connect to each docker daemon in the same way you connect to the local socket daemon in the addresses 10.0.0.3:2375, 10.0.0.2:2375, etc.

  • Target Docker version (the host/cluster you manage): 1.12+
@deviantony

This comment has been minimized.

Copy link
Member

commented Jan 4, 2017

Thanks for the proposal ! This is a major evolution, at least at a functional level, we'll be able to discuss it here.

@deviantony

This comment has been minimized.

Copy link
Member

commented Feb 13, 2017

Update on this one, starting from Docker 1.13 the /nodes API exposes the IP addresses of all the nodes. Portainer can then query all the nodes and aggregate the response to get all the containers and images in the cluster.

@WTFKr0

This comment has been minimized.

Copy link
Contributor

commented Mar 1, 2017

If you use external IP of all nodes in the cluster, you have to get TLS certificates too for requesting API on port 2376
Or, like @bvis proposed, create a global service in same network of portainer with a docker-proxy and access them internally

@deviantony

This comment has been minimized.

Copy link
Member

commented Mar 16, 2017

An extra thought on using the IP addresses of the nodes exposed by the API:

A swarm mode enabled cluster can be created with only the manager exposing the port 2375 to manage the cluster, all workers can run and expose only the Linux socket. So this approach cannot be used.

@man4j

This comment has been minimized.

Copy link

commented Mar 23, 2017

"all workers can run and expose only the Linux socket"
What if manually configure worker to open TCP/IP port? For example in docker.socket:
[Socket]
ListenStream=0.0.0.0:2376

@deviantony

This comment has been minimized.

Copy link
Member

commented Mar 23, 2017

Well yes, but what's the difference with configuring your daemon here?

We want this integration to be simple and transparent for the user, no configuration required.

@man4j

This comment has been minimized.

Copy link

commented Mar 23, 2017

I dont understand. How can I see all containers in cluster now? And why dont implement this feature based on manually worker configuration? Yes, it's not very easy, but at least it probably will work.

@deviantony

This comment has been minimized.

Copy link
Member

commented Mar 23, 2017

@man4j that's simple, at the moment you can't see all the containers in the cluster because the Docker API does not expose that (see it as the equivalent of docker ps -a on your Swarm manager, this will only list the containers running on that node).

The motto of Portainer is: simplicity. We should be able to add this feature in Portainer and make it simple and easy to enable for our users.

Besides, even if you enable a worker on TCP/IP (which might be your case only), some users will use TLS setup, others will want to secure their workers by not enabling a TCP socket on them. We need to provide a solution that works for all the cases.

@man4j

This comment has been minimized.

Copy link

commented Mar 23, 2017

Yes, of course users must enable TCP socket + TLS for this feature ) Why not? Or another way - configure servers to access each other across ssh without passwords (using keys). Yes, it's not very simple, but very important for production usage of Portainer.

@WTFKr0

This comment has been minimized.

Copy link
Contributor

commented Mar 23, 2017

I think it's complicated today to detect swarm cluster endpoint list (with access), based on existing API
I think the easiest solution may be to define endpoints with new option, like that :

[
  {
    "Name": "docker1 - manager",
    "URL": "tcp://docker1:2376",
    "TLS": true,
    "TLSCACert": "/certs/docker1/docker1.pem",
    "TLSCert": "/certs/docker1/docker1.pem",
    "TLSKey": "/certs/docker1/docker1.key",
    "SwarmEndPoints": [
      "docker1 - manager",
      "docker2 - manager",
      "docker3 - worker",
      "docker4 - worker"
    ]
  },
  {
    "Name": "docker2 - manager",
    "URL": "tcp://docker2:2376",
    "TLS": true,
    "TLSCACert": "/certs/docker2/docker2.pem",
    "TLSCert": "/certs/docker2/docker2.pem",
    "TLSKey": "/certs/docker2/docker2.key",
    "SwarmEndPoints": [
      "docker1 - manager",
      "docker2 - manager",
      "docker3 - worker",
      "docker4 - worker"
    ]
  },
  {
    "Name": "docker3 - worker",
    "URL": "tcp://docker3:2376",
    "TLS": true,
    "TLSCACert": "/certs/docker3/docker3.pem",
    "TLSCert": "/certs/docker3/docker3.pem",
    "TLSKey": "/certs/docker3/docker3.key"
  },
  {
    "Name": "docker4 - worker",
    "URL": "tcp://docker4:2376",
    "TLS": true,
    "TLSCACert": "/certs/docker4/docker1.pem",
    "TLSCert": "/certs/docker4/docker1.pem",
    "TLSKey": "/certs/docker4/docker1.key"
  },
]

On swarm manager, we can get all linked endpoints toget aggregated view for containers/images/networks, and list them with a new column endpoint to see where object is defined

@WTFKr0

This comment has been minimized.

Copy link
Contributor

commented Apr 21, 2017

Like the mechanism of the endpoints.json file, it could be interesting if portainer can check for DNS-SRV records to auto discover endpoints

Docker swarm by default embed a DNS server for communication between containers.
For example, if you have a global service named cadvisor, you can list all containers with :

nslookup tasks.cadvisor
Server:    127.0.0.11
Address 1: 127.0.0.11

Name:      tasks.cadvisor
Address 1: 10.0.12.4 stack_cadvisor.lqjx3f5bh61o6htsept82b0s2.zhwfgb6yqowzjcxnvbpi2ab71.stack
Address 2: 10.0.12.19 stack_cadvisor.sia1o72aa0esh0r7cpz2zrop6.qjna1uansg2g3n4i04bqoivop.stack

Prometheus, for example, use that to discover targets dynamically

I think it could be great if we can propose a stack (docker-compose v3) with portainer and a socat like global service
In one command, we got portainer auto-discovering our nodes in the swarm cluster

Info on prometheus :
https://prometheus.io/blog/2015/06/01/advanced-service-discovery/#discovery-with-dns-srv-records

@deviantony

This comment has been minimized.

Copy link
Member

commented May 25, 2017

Latest thoughts on the subject.

Functional goals

  • Ability to list all containers running in the cluster when accessing the containers view
  • Ability to execute operations on any container in the cluster from the containers view
  • Ability to inspect and execute operations on any container in the cluster from the container-details view
  • Should be simple to enable: when creating an endpoint, a user will be able to tick a "Swarm cluster" checkbox. This will enable this capability on the endpoint.

Proposition

Implementation proposition.

Global service

Deploy a global portainer-agent service in the cluster using mode=host when publishing the port so that querying each node will actually talk to the agent located on that node.

Each container would bind mount the local Docker socket and expose the Docker API on a port on each node.

Following a discussion with @bvis on Slack, each agent should be actually created inside an overlay network (Portainer should be started into this overlay network too). The communication between the agents and the main Portainer instance would then be done within this overlay network, removing the need to expose the agent on each node.

It implies the following:

  • Use DNS-SRV records to list the IP addresses of all the agents in the overlay network
  • More secured, as it restrict the access to the agent from outside the cluster and does not expose any port on each node.

As authentication and UAC is stored inside the main Portainer instance, any query must be executed against the Portainer instance.

diagram1.png

In this context, the aggregated response would have applied UAC and filter access to resources.

Service creation would be something similar to:

# Requires the creation of an overlay network when deploying Portainer
$ docker network create \
  --driver overlay \
  portainer-network

# Deploy Portainer
$ docker service create \
    --name portainer \
    --network portainer-network
    --publish 9000:9000 \
    --constraint 'node.role == manager' \
    --mount type=bind,src=//var/run/docker.sock,dst=/var/run/docker.sock \
    portainer/portainer \
    -H unix:///var/run/docker.sock

# Automated from Portainer
$ docker service create \
    --name portainer-agent \
    --network portainer-network
    --mount type=bind,src=//var/run/docker.sock,dst=/var/run/docker.sock \
    portainer/portainer-agent

Registering agents

Multiple solutions can be used to register the agents into the main Portainer instance:

* (A): Use the Docker API to list all nodes in the cluster via the /nodes endpoint, IP address of each node is exposed in the Status.Addr field (note: this is only available in Docker API > 1.24 or Docker version >= 1.13)

  • (B): Create a /agent in Portainer API so that an agent can register itself to the main Portainer instance once started. This requires an extra parameter when starting the agent: the location of the Portainer API (IP/URL + port).

  • (C): Use DNS-SRV records as suggested by @WTFKr0

Still need to figure out which way would be the best. Note that I'm not sure how to retrieve the Portainer API location in (B) (should we use the browser URL ? e.g. when browsing http://myportainer.domain:com API should be located at myportainer.domain.com/api).

Security considerations

By default, Swarm workers only expose the Docker engine via the socket. Adding an agent on each node in the cluster would expose the API on a public port, so each node would be exposed publicly. Running agents and Portainer inside an overlay network prevents other containers outside this network to access the agents but a container can be started inside this network and query the agents, so the communication between Portainer and the agents still needs to be secured.

Now, I don't think that anybody would want that. Here is a proposition for a secured setup:

  • Each node (manager/worker) in the cluster should expose the Docker engine on the socket only
  • Portainer should be started as a service on manager using a socket bind mount
  • Communication between the main Portainer instance and agents should be secured within the overlay network

diagram2.png

We could secure the communication between the agents using the following methods:

  • use TCP + SSL to communicate with agents (Docker API response would be encoded on the agent and decoded on the main Portainer instance)
  • secure the HTTP API of each agents with TLS

Extra considerations

A list of thing that we still need to think about:

  • How to refresh the list of agents ? E.g. when nodes are added/removed, should we add the ability to click on a 'refresh' button in the endpoints view for example?
  • When querying an operation such as inspect a container on a specific container, the agent where the container is running should be queried directly (we should not query each agent/node in the cluster). I think that we'll need to decorate the request to handle that.
@rreinurm

This comment has been minimized.

Copy link
Contributor

commented May 25, 2017

I personally prefer to use option (a) as it makes entire swarm cluster deployment more compact, but its not a hard task to deploy global service of Portainer agents.

I propose to consider gRPC protocol for secure and fast communication with Portainer agents. http://www.grpc.io/

@markvr

This comment has been minimized.

Copy link

commented May 25, 2017

I've been following this with interest as it would make Portainer much more useful to us.

Rather than needing to expose each instance with mode=host which has significant security implications, could you use a swarm overlay network for the intra-portainer communication, and then just need to expose the "master" instance on the host?

If it's any use, for another project I used Nginx with SSL certificates to proxy the Docker socket ports. Something similar would mean you would only need to run portainer on the "master", with a "socket-proxy" container on the overlay networks. This proxies all the REST calls OK but I don't know if it would work with the tty/shell remote access.

@deviantony

This comment has been minimized.

Copy link
Member

commented May 25, 2017

@markvr I've updated my post, indeed using an overlay network would actually be a better choice.

@deviantony

This comment has been minimized.

Copy link
Member

commented May 26, 2017

@markvr @rreinurm @bvis extra-thoughts:

Deploying agents inside an overlay network might work well when deploying Portainer inside that network as a service. But Portainer aims to manage different Swarm clusters from the same instance... using this solution won't work in that case.

We might want to be able to support both:

  • Agent deployment inside an overlay network so that users with only one cluster can deploy Portainer as a service and all its agent inside that cluster
  • Agent deployment with port exposed publicy so that Portainer can manage the cluster without being deployed into that cluster

In both cases, it's a simple deployment configuration for the agent but it impacts how the agents will be registered inside the Portainer instance (the DNS-SRV records solution is working for the first case only, so we need to determine a solution to register agents that will support both cases).

@WTFKr0

This comment has been minimized.

Copy link
Contributor

commented May 30, 2017

Hi all
Thinking of that, I think we need a more independent solution, or an other portainer project
Like a swarm aggregator project

The idea is to provide a compose file deploying 2 services

  • a global socat
  • a replicated aggregator

aggregator should discover socat agents by dns-srv
aggregator should handle as many as possible docker api calls and then communicate to agents to do actions and concatenate responses

we have to establish rules on api calls
container list seems a good starts, as it call list on all agents, then append agent name somewhere in response
but what we do with create/delete containers ?

Then we can tell users to start this compose on swarm cluster, and plug portainer as usual to th aggregator endpoint

I think this feature is too big to be just a portainer feature, but i'm kind to work on that in future 😄
WDYT ?

@jfgrissom

This comment has been minimized.

Copy link

commented Jan 25, 2018

Wow... has it been over a month since I looked at this!

Can I send you a 1.5 LTC? If that works please provide a public address and I'll send it out.

@ncresswell

This comment has been minimized.

Copy link
Member

commented Jan 25, 2018

@wernerbarnard

This comment has been minimized.

Copy link

commented Jan 25, 2018

Hi - what is the status on this issue?

@deviantony

This comment has been minimized.

Copy link
Member

commented Jan 26, 2018

@wernerbarnard I've got a agent POC working but I'm having a few difficulties with container console. Also need to tackle the agent security topic, no ETA yet but the work is in progress.

@jfgrissom

This comment has been minimized.

Copy link

commented Jan 26, 2018

@ncresswell - Thanks for sharing that. I'll send it out this evening.

@jfgrissom

This comment has been minimized.

Copy link

commented Jan 27, 2018

@ncresswell - I shot you guys 1.76943 LTC.

Here is the transaction ID.
https://live.blockcypher.com/ltc/tx/d6b249e78462b2bdf585290bf11be464853722a2e7d7a6383b6e9ecd21c55514/

Thanks or all your hard work on this project!

@ncresswell

This comment has been minimized.

Copy link
Member

commented Jan 27, 2018

@aiminickwong

This comment has been minimized.

Copy link

commented Mar 6, 2018

Any status about this issues?

@chrisalex2

This comment has been minimized.

Copy link

commented Mar 8, 2018

I am also eagerly awaiting this feature.

@deviantony

This comment has been minimized.

Copy link
Member

commented Apr 22, 2018

A PR is finally open for this one, have a look at #1828 :-)

@bvis

This comment has been minimized.

Copy link
Author

commented Apr 23, 2018

@deviantony Good job! It seems to work like a charm! 🥁

@bvis bvis closed this Apr 23, 2018

@deviantony deviantony reopened this Apr 23, 2018

@deviantony

This comment has been minimized.

Copy link
Member

commented Apr 23, 2018

I'll close the issue after merging #1828

@deviantony deviantony removed this from Release 1.13.x in Roadmap May 6, 2018

@deviantony deviantony modified the milestones: next, 1.16.x, 1.17.0 May 6, 2018

@Dean-Christian-Armada

This comment has been minimized.

Copy link

commented Jun 14, 2018

Hi should expect to see all the containers of my Swarm Cluster in portainer when I check the "Containers" tab?

@tlaukkan

This comment has been minimized.

Copy link

commented May 12, 2019

Any updates on this one?

@bvis

This comment has been minimized.

Copy link
Author

commented May 14, 2019

AFAIK this is working since 1 year ago. ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.