Cannot create cluster using docker containers

[udbj]

I want to create a 3-server couchdb cluster on provisioned instances running docker, but I can't get the nodes to connect. Here's my code for couchdb setup:

- name: Setup CouchDB
  block:      

  - name: Launch couchDB container
    shell: docker run -e ERL_FLAGS="-setcookie \"{{erl_cookie}}\"" -e COUCHDB_USER={{couchdb_name}} -e COUCHDB_PASSWORD={{couchdb_pass}} -e NODENAME={{inventory_hostname}} -e COUCHDB_SECRET={{erl_cookie}} -p 5984:5984 -d --name my-couchdb couchdb

  - debug: var=shell_output.stdout_lines

  - pause:
      minutes: 1

  - name: Bind clustered interface to all IP addresses on machine
    shell: curl -X PUT http://{{couchdb_name}}:{{couchdb_pass}}@127.0.0.1:5984/_node/_local/_config/chttpd/bind_address -d '"0.0.0.0"'
    register: shell_output
    args:
      warn: false

  - debug: var=shell_output.stdout_lines

  - name: Set up nodes
    shell: curl -X POST -H "Content-Type:application/json" http://{{couchdb_name}}:{{couchdb_pass}}@127.0.0.1:5984/_cluster_setup -d '{"action":"enable_cluster", "bind_address":"0.0.0.0", "username":"{{couchdb_name}}", "password":"{{couchdb_pass}}", "node_count":"3"}'
    register: shell_output
    args:
      warn: false

  - debug: var=shell_output.stdout_lines

  - name: Enable cluster for nodes
    shell: curl -X POST -H "Content-Type:application/json" http://{{couchdb_name}}:{{couchdb_pass}}@127.0.0.1:5984/_cluster_setup -d '{"action":"enable_cluster", "bind_address":"0.0.0.0", "username":"{{couchdb_name}}", "password":"{{couchdb_pass}}", "port":"5984", "node_count":"3", "remote_node":"{{item}}", "remote_current_user":"{{couchdb_name}}", "remote_current_password":"{{couchdb_pass}}" }'
    register: shell_output
    args:
      warn: false
    with_items: "{{ansible_play_hosts[1:]}}"
    when: inventory_hostname == ansible_play_hosts[0]

  - debug: var=shell_output.stdout_lines
  
  - name: Add nodes to cluster
    shell: curl -X POST -H "Content-Type:application/json" http://{{couchdb_name}}:{{couchdb_pass}}@127.0.0.1:5984/_cluster_setup -d '{"action":"add_node", "host":"{{item}}", "port":"5984", "username":"{{couchdb_name}}", "password":"{{couchdb_pass}}"}'
    register: shell_output
    args:
      warn: false
    with_items: "{{ansible_play_hosts[1:]}}"
    when: inventory_hostname == ansible_play_hosts[0]

  - debug: var=shell_output.stdout_lines

  - name: Finish cluster setup 
    shell: curl -X POST -H "Content-Type:application/json" http://{{couchdb_name}}:{{couchdb_pass}}@127.0.0.1:5984/_cluster_setup -d '{"action":"finish_cluster"}'
    register: shell_output
    args:
      warn: false
    when: inventory_hostname == ansible_play_hosts[0]

  - debug: var=shell_output.stdout_lines


  tags: ['never', 'setup_couchdb']

And here's the output:

TASK [Launch couchDB container] ************************************************
changed: [ec2-3-26-52-207.ap-southeast-2.compute.amazonaws.com]
changed: [ec2-13-236-147-64.ap-southeast-2.compute.amazonaws.com]
changed: [ec2-3-106-130-82.ap-southeast-2.compute.amazonaws.com]

TASK [debug] *******************************************************************
ok: [ec2-13-236-147-64.ap-southeast-2.compute.amazonaws.com] => {
    "shell_output.stdout_lines": "VARIABLE IS NOT DEFINED!"
}
ok: [ec2-3-106-130-82.ap-southeast-2.compute.amazonaws.com] => {
    "shell_output.stdout_lines": "VARIABLE IS NOT DEFINED!"
}
ok: [ec2-3-26-52-207.ap-southeast-2.compute.amazonaws.com] => {
    "shell_output.stdout_lines": "VARIABLE IS NOT DEFINED!"
}

TASK [pause] *******************************************************************
Pausing for 60 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [ec2-13-236-147-64.ap-southeast-2.compute.amazonaws.com]

TASK [Bind clustered interface to all IP addresses on machine] *****************
changed: [ec2-3-26-52-207.ap-southeast-2.compute.amazonaws.com]
changed: [ec2-3-106-130-82.ap-southeast-2.compute.amazonaws.com]
changed: [ec2-13-236-147-64.ap-southeast-2.compute.amazonaws.com]

TASK [debug] *******************************************************************
ok: [ec2-13-236-147-64.ap-southeast-2.compute.amazonaws.com] => {
    "shell_output.stdout_lines": [
        "\"any\""
    ]
}
ok: [ec2-3-106-130-82.ap-southeast-2.compute.amazonaws.com] => {
    "shell_output.stdout_lines": [
        "\"any\""
    ]
}
ok: [ec2-3-26-52-207.ap-southeast-2.compute.amazonaws.com] => {
    "shell_output.stdout_lines": [
        "\"any\""
    ]
}

TASK [Set up nodes] ************************************************************
changed: [ec2-13-236-147-64.ap-southeast-2.compute.amazonaws.com]
changed: [ec2-3-106-130-82.ap-southeast-2.compute.amazonaws.com]
changed: [ec2-3-26-52-207.ap-southeast-2.compute.amazonaws.com]

TASK [debug] *******************************************************************
ok: [ec2-13-236-147-64.ap-southeast-2.compute.amazonaws.com] => {
    "shell_output.stdout_lines": [
        "{\"error\":\"bad_request\",\"reason\":\"Cluster is already enabled\"}"
    ]
}
ok: [ec2-3-106-130-82.ap-southeast-2.compute.amazonaws.com] => {
    "shell_output.stdout_lines": [
        "{\"error\":\"bad_request\",\"reason\":\"Cluster is already enabled\"}"
    ]
}
ok: [ec2-3-26-52-207.ap-southeast-2.compute.amazonaws.com] => {
    "shell_output.stdout_lines": [
        "{\"error\":\"bad_request\",\"reason\":\"Cluster is already enabled\"}"
    ]
}

TASK [Enable cluster for nodes] ************************************************
skipping: [ec2-3-106-130-82.ap-southeast-2.compute.amazonaws.com] => (item=ec2-3-106-130-82.ap-southeast-2.compute.amazonaws.com) 
skipping: [ec2-3-106-130-82.ap-southeast-2.compute.amazonaws.com] => (item=ec2-3-26-52-207.ap-southeast-2.compute.amazonaws.com) 
skipping: [ec2-3-26-52-207.ap-southeast-2.compute.amazonaws.com] => (item=ec2-3-106-130-82.ap-southeast-2.compute.amazonaws.com) 
skipping: [ec2-3-26-52-207.ap-southeast-2.compute.amazonaws.com] => (item=ec2-3-26-52-207.ap-southeast-2.compute.amazonaws.com) 
changed: [ec2-13-236-147-64.ap-southeast-2.compute.amazonaws.com] => (item=ec2-3-106-130-82.ap-southeast-2.compute.amazonaws.com)
changed: [ec2-13-236-147-64.ap-southeast-2.compute.amazonaws.com] => (item=ec2-3-26-52-207.ap-southeast-2.compute.amazonaws.com)

TASK [debug] *******************************************************************
ok: [ec2-13-236-147-64.ap-southeast-2.compute.amazonaws.com] => {
    "shell_output.stdout_lines": "VARIABLE IS NOT DEFINED!"
}
ok: [ec2-3-106-130-82.ap-southeast-2.compute.amazonaws.com] => {
    "shell_output.stdout_lines": "VARIABLE IS NOT DEFINED!"
}
ok: [ec2-3-26-52-207.ap-southeast-2.compute.amazonaws.com] => {
    "shell_output.stdout_lines": "VARIABLE IS NOT DEFINED!"
}

TASK [Add nodes to cluster] ****************************************************
skipping: [ec2-3-106-130-82.ap-southeast-2.compute.amazonaws.com] => (item=ec2-3-106-130-82.ap-southeast-2.compute.amazonaws.com) 
skipping: [ec2-3-106-130-82.ap-southeast-2.compute.amazonaws.com] => (item=ec2-3-26-52-207.ap-southeast-2.compute.amazonaws.com) 
skipping: [ec2-3-26-52-207.ap-southeast-2.compute.amazonaws.com] => (item=ec2-3-106-130-82.ap-southeast-2.compute.amazonaws.com) 
skipping: [ec2-3-26-52-207.ap-southeast-2.compute.amazonaws.com] => (item=ec2-3-26-52-207.ap-southeast-2.compute.amazonaws.com) 
changed: [ec2-13-236-147-64.ap-southeast-2.compute.amazonaws.com] => (item=ec2-3-106-130-82.ap-southeast-2.compute.amazonaws.com)
changed: [ec2-13-236-147-64.ap-southeast-2.compute.amazonaws.com] => (item=ec2-3-26-52-207.ap-southeast-2.compute.amazonaws.com)

TASK [debug] *******************************************************************
ok: [ec2-13-236-147-64.ap-southeast-2.compute.amazonaws.com] => {
    "shell_output.stdout_lines": "VARIABLE IS NOT DEFINED!"
}
ok: [ec2-3-106-130-82.ap-southeast-2.compute.amazonaws.com] => {
    "shell_output.stdout_lines": "VARIABLE IS NOT DEFINED!"
}
ok: [ec2-3-26-52-207.ap-southeast-2.compute.amazonaws.com] => {
    "shell_output.stdout_lines": "VARIABLE IS NOT DEFINED!"
}

TASK [Finish cluster setup] ****************************************************
skipping: [ec2-3-106-130-82.ap-southeast-2.compute.amazonaws.com]
skipping: [ec2-3-26-52-207.ap-southeast-2.compute.amazonaws.com]
changed: [ec2-13-236-147-64.ap-southeast-2.compute.amazonaws.com]

TASK [debug] *******************************************************************
ok: [ec2-13-236-147-64.ap-southeast-2.compute.amazonaws.com] => {
    "shell_output.stdout_lines": [
        "{\"error\":\"setup_error\",\"reason\":\"Cluster setup timed out waiting for nodes to connect\"}"
    ]
}
ok: [ec2-3-106-130-82.ap-southeast-2.compute.amazonaws.com] => {
    "shell_output.stdout_lines": "VARIABLE IS NOT DEFINED!"
}
ok: [ec2-3-26-52-207.ap-southeast-2.compute.amazonaws.com] => {
    "shell_output.stdout_lines": "VARIABLE IS NOT DEFINED!"
}

The required ports are open:

rules:
            - proto: tcp
              ports: 22
              cidr_ip: 0.0.0.0/0
              rule_desc: allow all on ssh port
            
            - proto: tcp
              ports:
                - 5984
                - 4369
              cidr_ip: 0.0.0.0/0
              rule_desc: allow all on couchdb port

            - proto: tcp
              from_port: 9100
              to_port: 9200
              cidr_ip: 0.0.0.0/0
              rule_desc: allow all on erlang ports

It's ansible code, but that isn't relevant here, since the shell commands throw the same error even when I run them on the terminal. Adding the nodes appends their address to the 'cluster nodes' list, but not the 'all nodes' list, and the same time out error is thrown when I try to finish the cluster setup.

I am guessing that it's got something to do with the environment variables. I did set the erlang cookie and the shared secret, but the vm.args file did not have a -setcookie command in it. It had the proper name (couchdb@aws_address), but the cookie wasn't in it. and There was no .erlang.cookie file in the /opt/couchdb folder either, but I am not sure if that's the reason for the failure, since it was set as an environment variable.

[udbj]

Nvm, I hadn't published ports 4369 and 9100-9200 from docker to host. Also had to set the the same hashed password on all nodes.