How to create a docker cluster of pilosa

[chetangehlot]

Hi,

I am creating a docker cluster of pilosa. But it shows only one Node in cluster.

node1.config.toml

data-dir = "node1_data"
bind = "https://localhost:10501"

[cluster]
coordinator = true

[tls]
certificate = "pilosa.local.crt"
key = "pilosa.local.key"
skip-verify = true

[gossip]
seeds = ["localhost:15000"]
port = 15000
key = "pilosa.local.gossip32"

node2.config.toml

data-dir = "node2_data"
bind = "https://localhost:10502"

[tls]
certificate = "pilosa.local.crt"
key = "pilosa.local.key"
skip-verify = true

[gossip]
seeds = ["localhost:15000"]
port = 16000
key = "pilosa.local.gossip32"

node3.config.toml

data-dir = "node3_data"
bind = "https://localost:10503"

[tls]
certificate = "pilosa.local.crt"
key = "pilosa.local.key"
skip-verify = true

[gossip]
seeds = ["localhost:15000"]
port = 17000
key = "pilosa.local.gossip32"

[jaffee]

The configs look OK to me (though I'd have to check the TLS settings to be sure). My gut says this is probably a network configuration issue between the containers, but you might try turning off TLS to simplify things.

Can you post how you're launching the containers, and any output/logs from Pilosa?

[chetangehlot]

docker run -itd --name pilosa -v ~/node1.config.toml:/node1.config.toml
-v ~/pilosa.local.gossip32:/pilosa.local.gossip32
-v ~/pilosa.local.crt:/pilosa.local.crt
-v ~/pilosa.local.key:/pilosa.local.key
-p 10501:10101 pilosa/pilosa:latest

docker run -itd --name pilosa-worker1 -v ~/node2.config.toml:/node2.config.toml
-v ~/pilosa.local.gossip32:/pilosa.local.gossip32
-v ~/pilosa.local.crt:/pilosa.local.crt
-v ~/pilosa.local.key:/pilosa.local.key
-p 10502:10101 pilosa/pilosa:latest

docker run -itd --name pilosa-worker2 -v ~/node3.config.toml:/node3.config.toml
-v ~/pilosa.local.gossip32:/pilosa.local.gossip32
-v ~/pilosa.local.crt:/pilosa.local.crt
-v ~/pilosa.local.key:/pilosa.local.key
-p 10503:10101 pilosa/pilosa:latest

#Logs with TLS
2018/05/15 05:31:32 Pilosa v0.8.8, build time 2018-02-19T22:04:28+0000
2018/05/15 05:31:32 Using data from: /data
2018/05/15 05:31:32 [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:14000
2018/05/15 05:31:32 [DEBUG] memberlist: Stream connection from=127.0.0.1:60672
2018/05/15 05:31:32 [DEBUG] memberlist: Failed to join ::1: dial tcp [::1]:14000: connect: cannot assign requested address
2018/05/15 05:31:32 Listening as http://0.0.0.0:10101
2018/05/15 05:31:32 Pilosa is currently configured to send small diagnostics reports to our team every hour. More information here: https://www.pilosa.com/docs/latest/administration/#diagnostics
2018/05/15 05:31:32 holder sync monitor initializing (10m0s interval)

[yuce]

@chetangehlot Below is a docker compose file that runs a 2 node Pilosa cluster. The command line switches override environment variables, so I had to override the entrypoint but we may change that in the future. We will be adding a Docker Cluster Tutorial to the documentation soon:

version: '2'
services: 
    pilosa1:
        image: pilosa/pilosa:v0.9.0
        ports: 
            - "10101:10101"
        environment:
            - PILOSA_CLUSTER_COORDINATOR=true
            - PILOSA_GOSSIP_SEEDS=:15000
            - PILOSA_GOSSIP_PORT=15000
        network_mode: "host"
        extra_hosts:
          - "moby:127.0.0.1"
        entrypoint:
          - /pilosa
          - server
          - --bind
          - ":10101"
    pilosa2:
        image: pilosa/pilosa:v0.9.0
        ports: 
            - "10102:10102"
        environment:
            - PILOSA_GOSSIP_SEEDS=:15000
            - PILOSA_GOSSIP_PORT=16000
        network_mode: "host"
        extra_hosts:
          - "moby:127.0.0.1"
        entrypoint:
          - /pilosa
          - server
          - --bind
          - ":10102"

After running this configuration with docker-compose up we can check the status of the cluster, and it should look like the following:

$ curl localhost:10101/status

{"state":"NORMAL","nodes":[{"id":"02ce5e01-ee07-436e-bce4-43cac445d71a","uri":{"scheme":"http","host":"localhost","port":10102},"isCoordinator":false},{"id":"5a66d5b6-d13c-4c32-bbd9-323bd0265fd2","uri":{"scheme":"http","host":"localhost","port":10101},"isCoordinator":true}]}

$ curl localhost:10102/status

{"state":"NORMAL","nodes":[{"id":"02ce5e01-ee07-436e-bce4-43cac445d71a","uri":{"scheme":"http","host":"localhost","port":10102},"isCoordinator":false},{"id":"5a66d5b6-d13c-4c32-bbd9-323bd0265fd2","uri":{"scheme":"http","host":"localhost","port":10101},"isCoordinator":true}]}

[chetangehlot]

Yes its working.
But if I am running the docker command on two different machine without TLS. So its not working.

docker run -itd --name pilosa -p 10101:10101 -e PILOSA_CLUSTER_COORDINATOR=true -e PILOSA_GOSSIP_SEEDS=:15000 -e PILOSA_GOSSIP_PORT=15000 --network="host" --add-host="moby:127.0.0.1" pilosa/pilosa:v0.9.0 server --bind "0.0.0.0:10101"

docker run -itd --name pilosa-worker -p 10102:10102 -e PILOSA_GOSSIP_SEEDS=:15000 -e PILOSA_GOSSIP_PORT=16000 --ip="192.168.1.138" --network="host" --add-host="moby:127.0.0.1" pilosa/pilosa:v0.9.0 server --bind "0.0.0.0:10102"

but it shows error and doesn't connect:

docker: Error response from daemon: user specified IP address is supported on user defined networks only.

[yuce]

You can run the containers in a user created network.

Create a new network using:

docker network create pilosanet

Then run container1:

docker run -it --name pilosa1 -p 10101:10101 -e PILOSA_CLUSTER_COORDINATOR=true -e PILOSA_GOSSIP_SEEDS=pilosa1:14000 --network="pilosanet" pilosa/pilosa:v0.9.0 server -b pilosa1

And container2:

docker run -it --name pilosa2 -e PILOSA_GOSSIP_SEEDS=pilosa1:14000 --network="pilosanet" pilosa/pilosa:v0.9.0 server -b pilosa2

I've made the command lines a bit shorter by using the defaults. Also I didn't add the TLS configuration to the command line above, but let me know if you have problems adding or running with that configuration.

The docker compose file that corresponds to the steps above is:

version: '2'
services: 
    pilosa1:
        image: pilosa/pilosa:v0.9.0
        ports: 
            - "10101:10101"
        environment:
            - PILOSA_CLUSTER_COORDINATOR=true
            - PILOSA_GOSSIP_SEEDS=pilosa1:14000
        networks:
          - pilosanet
        entrypoint:
          - /pilosa
          - server
          - --bind
          - "pilosa1:10101"
    pilosa2:
        image: pilosa/pilosa:v0.9.0
        environment:
            - PILOSA_GOSSIP_SEEDS=pilosa1:14000
        networks:
          - pilosanet
        entrypoint:
          - /pilosa
          - server
          - --bind
          - "pilosa2:10101"
networks: 
  pilosanet:

Let me know if this works for you or if there are any problems.

[yuce]

@chetangehlot It would be great to know what platform are you using docker on. Are you on Linux? If not are you using Docker for Mac/Windows or Docker Machine ?

[chetangehlot]

@yuce I am using linux (ubuntu:14.04) OS .

I am running above commands on different machine. Pilosa master is running but worker or pilosa2 doesn't start its shows the error like:

• Failed to resolve pilosa1:14000: lookup pilosa1 on 127.0.0.11:53: no such host
  2018/05/16 12:55:22 [WARN] memberlist: Failed to resolve pilosa1:14000: lookup pilosa1 on 127.0.0.11:53: no such host
  2018/05/16 12:55:24 retrying after error: 1 error occurred:

[chetangehlot]

It runs on only single machine but when I am using a multiple machine then it shows the error.

[yuce]

OK, I thought you were having problems with running a Pilosa cluster on a single server. The problem you are having is not related to Pilosa; you have to run a Docker swarm and create an overlay network: https://docs.docker.com/network/network-tutorial-overlay since otherwise there's no way a Docker daemon running on one server can know about the containers running in the other. After that, the commandline I've pasted before should work (maybe with small modifications).

[chetangehlot]

I have two servers and run the command on first server

docker run -itd --name pilosa1 -p 10101:10101 -p 14000:14000 -e PILOSA_CLUSTER_COORDINATOR=true -e PILOSA_GOSSIP_SEEDS=192.168.1.138:14000 --network="pilosanet" pilosa/pilosa:v0.9.0 server -b "0.0.0.0:10101"

#Second server

docker run -itd --name pilosa2 -p 10101:10101 -p 14000:14000 -e PILOSA_GOSSIP_SEEDS=192.168.1.138:14000 --network="pilosanet" pilosa/pilosa:v0.9.0 server -b "0.0.0.0:10101"

[chetangehlot]

Hi @yuce the cluster is created but don't send the gossip to 0.0.0.0:14000. How? I open all the IP address with 0.0.0.0.

#####On worker
docker logs -f b72e830e99cb
2018/05/17 08:07:27 load NodeID: /.pilosa/.id
2018/05/17 08:07:27 open server
2018/05/17 08:07:27 add node Node: af389371-fd7c-440d-b983-579f15be90b0 to cluster on Node: af389371-fd7c-440d-b983-579f15be90b0
2018/05/17 08:07:27 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.138:14000
2018/05/17 08:07:27 wait for joining to complete
2018/05/17 08:07:27 received NodeJoin event: &{0 Node: 63b7013a-0f40-48ec-8200-a29ae51b110d}
2018/05/17 08:07:28 [ERR] memberlist: Failed to send gossip to 0.0.0.0:14000: write udp [::]:14000->0.0.0.0:14000: sendto: cannot assign requested address
2018/05/17 08:07:28 [ERR] memberlist: Failed to send gossip to 0.0.0.0:14000: write udp [::]:14000->0.0.0.0:14000: sendto: cannot assign requested address
2018/05/17 08:07:28 [ERR] memberlist: Failed to send gossip to 0.0.0.0:14000: write udp [::]:14000->0.0.0.0:14000: sendto: cannot assign requested address
2018/05/17 08:07:28 [ERR] memberlist: Failed to send ping: write udp [::]:14000->0.0.0.0:14000: sendto: cannot assign requested address
2018/05/17 08:07:30 [ERR] memberlist: Failed to send ping: write udp [::]:14000->0.0.0.0:14000: sendto: cannot assign requested address

[yuce]

@chetangehlot Setting gossip seed to 0.0.0.0 won't work, since that's not a real address (other nodes cannot access it). I'm updating our Docker Tutorial with a Docker Swarm example. I'll ask your feedback when it's completed.

[chetangehlot]

@yuce Thanks for your coordination. I am waiting for the docker swarm without YML file. It should work on multiple server where I run docker command on each server and connect to the master.