Add CONSUL_RETRY_JOIN_WAN env #48

tjcelaya · 2017-12-08T00:20:38Z

Resolves #47. Will need to be rebased once #46 is merged.

Wanted some feedback about how best to show the multi-DC example. Options I'm considering:

new multi-datacenter-compose.yml defining 2 Consul services
new partial test-dc2-compose.yml which just has consul-dc2 and gives an example with multiple usages of -f, e.g.

docker-compose -f local-compose.yml -f test-dc2-compose.yml up --scale consul=3 --scale consul-dc2=3

The naming of the second-datacenter consul service definitely needs some improvement though.

jwreagor

During my upgrade PR I didn't break out the compose files into separate examples directories because I wanted to focus on the upgrade itself.

Other blueprints are starting to organize with an examples/ directory and I feel like that is prudent now. We could have examples/compose/, examples/triton/, and examples/multidc/. We could also combine multi-DC with the Triton deployment example as well, if that fits.

There are differences between running locally and running in a staging/prod type environment, just like there is a difference between running single versus multi-DC.

Thoughts? Is that too much re-organization for what you wanted to add in this PR?

Also, I think the usability of the multi-compose file behavior could workout well. Maybe we add a separate multi-dc compose file as an option alongside our triton example?

jwreagor · 2017-12-11T17:29:43Z

README.md

+    ```
+    ==> Error parsing /etc/consul/consul.hcl: ... unexpected token while parsing list: IDENT
+    ```
+


👍 for pointing out the error message.

jwreagor · 2017-12-11T17:42:58Z

I need to try this out myself, hoping to get to that before we finalize.

tjcelaya · 2017-12-13T18:17:16Z

So the setup script works fine, I've successfully gotten clusters in us-sw-1 and us-east-1 talking to each other but...

While consul members -wan lists all nodes and both data centers are listed in the web UI, kv operations and attempts to navigate to the remote data center from the UI always fail with 500: No path to datacenter or some other rpc error.

tjcelaya · 2017-12-13T23:03:40Z

Finally sorted this all out. The issue comes from the way the RPC and Serf servers are decoupled.
If you specify all of the following:

bind_addr = private
adverties_addr = private
advertise_wan_addr = public
serf_lan(_bind) = private
serf_wan(_bind) = public
translate_wan_addrs = true

You'll get the following very helpful error when doing anything across data centers:

rpc error getting client: failed to get conn: dial tcp $PRIVATE:0->$PUBLIC:8300: i/o timeout

Logging onto the remote node and running netstat -ltun reveals that the RPC server is listening on the bind_addr so it will reject connection attempts that reach it by its public address (actual IPs have been changed to their source fields):

bash-4.3# netstat -ltun
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 bind_addr:8300          0.0.0.0:*               LISTEN
tcp        0      0 serf_wan:8302           0.0.0.0:*               LISTEN
tcp        0      0 serf_lan:8301           0.0.0.0:*               LISTEN
tcp        0      0 :::53                   :::*                    LISTEN
tcp        0      0 :::8500                 :::*                    LISTEN
udp        0      0 serf_wan:8302           0.0.0.0:*
udp        0      0 serf_lan:8301           0.0.0.0:*
udp        0      0 :::53                   :::*

Not sure how we could improve this other than making users aware of this pitfall and recommending they use tunnelling or add firewall rules to only allow communication from the other cluster. The addition of a cross-datacenter-but-not-internet-public network would allow us to avoid setting bind_addr = 0.0.0.0, but I don't think that is available in Triton Cloud yet.

…ack into the CONSUL_RETRY_JOIN_WAN block

tjcelaya · 2017-12-14T00:33:50Z

One remaining question (unless I remember the other question):

What to do about the other parameters we need to set to get all of this wired up. I've structured preStart to allow users to set those environment variables if desired, but it might make more sense move them into the if [ -n "$CONSUL_RETRY_JOIN_WAN" ] block so they only appear in consul.hcl if the user has specified they want to try to join other nodes over the WAN. CONSUL_BIND_ADDR is a particularly problematic one given the above caveats, it needs to be set to 0.0.0.0, unless the user has specified it explicitly.

tjcelaya · 2017-12-14T23:38:48Z

Admittedly the setup-multi-datacenter.sh script could be simplified into instructions for how to clone the single-dc docker-compose.yml + _env but I'm leaving that up to review.

jwreagor · 2017-12-15T23:58:19Z

This looks ready in terms of CR. Monday I can run all the scripts and fire up a multi data center Consul cluster, then we can get the functional bits approved.

jwreagor

Nits and a few other things when I started working with this branch. make build definitely needs to be fixed (not a huge issue though).

jwreagor · 2017-12-18T21:32:37Z

README.md

+    - If not specified, automatic detection of the datacenter will be attempted. See [issue #23](https://github.com/autopilotpattern/consul/issues/23) for more details.
+    - Consul's default of "dc1" will be used if none of the above apply.
+
+- `CONSUL_BIND_ADDR`: Explicitly set the corresponding Consul configuration. This value will be set to `0.0.0.0` if not specified and `CONSUL_RETRY_JOIN_WAN` is provided. Be aware of 


Sentence ends at Be aware of.

jwreagor · 2017-12-18T21:37:20Z

examples/compose/docker-compose-multi-datacenter.yml

@@ -0,0 +1,37 @@
+version: '2.1'


Personally, I don't see a good reason to run multi-DC on the same local host and would rather not have this file here. Unless you see a reason to iterate on multi-DC locally and that being something many users of this pattern might do.

jwreagor · 2017-12-18T21:38:00Z

examples/compose/docker-compose.yml

  consul:
-    build: .
+    build: ../../
+    image: autopilotpattern/consul:${TAG:-latest}


What is the point of both build and image? This file was more helpful without image and just building the container locally. I'm not sure what that is buying us.

I was using the docker-compose file to build the image as well. I'll remove it.

jwreagor · 2017-12-18T21:40:20Z

examples/triton/docker-compose-multi-datacenter.yml.template

@@ -0,0 +1,23 @@
+version: '2.1'


I'd rather have this file within examples/multi-dc/docker-compose.yml.tmpl and not under examples/triton so that users can understand how to run Consul on Triton without getting DC involved. examples/multi-dc can also host all the templates you are rendering.

jwreagor · 2017-12-18T21:43:10Z

setup.sh

@@ -42,9 +42,11 @@ check() {
    # make sure Docker client is pointed to the same place as the Triton client


Thoughts on moving setup.sh into examples/triton? It doesn't seem like a requirement for examples/compose or examples/multi-dc.

The multi-dc example's setup script calls setup.sh once per datacenter so there is a dependency between the two. Nothing preventing us from referencing it as ../triton/setup.sh but it would lead to some confusion.

If the multi-dc example were named triton-multi-dc, or lived in examples/triton/multi-dc it might make the script dependency more obvious. Thoughts?

Since this isn't a code library you can copy the check() function into your own shell script. Your script is superseding that setup anyway so owning it might help extend things later, like automatic setup across multiple profiles instead of leaving that up to the end user (removing those extra export steps over each DC). The latter isn't something we need to be concerned with ATM though.

check was copied into ./setup-multi-dc.sh and renamed to generate_env

jwreagor · 2017-12-18T21:43:50Z

setup-multi-datacenter.sh

@@ -0,0 +1,103 @@
+#!/bin/bash


Move this into examples/multi-dc/setup.sh so we keep all example assets together.

jwreagor · 2017-12-18T21:49:29Z

makefile

+
+# ------------------------------------------------
+# Multi-datacenter usage
+clean/triton:


This line isn't an issue but make build is broken due to file path issue in test/Dockerfile. We'll also want to make sure make test still works. Local tests passed for me when I patched.

tjcelaya · 2017-12-19T00:47:33Z

Just realized we don't actually need use triton profile set at all and could instead eval "TRITON_PROFILE=$profile triton env -d" within the script so the user doesn't have to worry their own environment variables. I'll be fixing this as well

tjcelaya · 2017-12-19T02:25:43Z

Discovered the command for getting the current account ID wasn't passing down the triton profile (since it doesn't accept that as a parameter) but can be fixed by passing it as an env on the triton account get invocation. That was preventing every consul instance I created from joining any other since the UUID from a different account was being used from my "current" profile. It should be possible to the the script to generate correct _env files again.

tjcelaya · 2017-12-22T23:51:58Z

This PR should be ready to merge now that all the documentation typos have been fixed and #50 is nearing completion.

jwreagor · 2018-01-04T15:13:38Z

@tjcelaya Feel free to squash and merge. I'm ok with supporting this.

tjcelaya added 3 commits December 7, 2017 15:24

Resolve autopilotpattern#23 by adding CONSUL_DATACENTER_NAME env

ea04c0f

Mention the environment variable in the README

cc21275

Resolves autopilotpattern#47 by adding CONSUL_RETRY_JOIN_WAN

189414c

tjcelaya changed the title ~~Add~~ Add CONSUL_RETRY_JOIN_WAN env Dec 8, 2017

tjcelaya added 2 commits December 7, 2017 16:24

Remove debugging output

23c8c60

Fix the environment value now that we've collected the error message

abad90e

jwreagor reviewed Dec 11, 2017

View reviewed changes

tjcelaya added 5 commits December 11, 2017 13:54

Environment variable documentation and moving examples.

b15de52

Triton datacenter autodetection

ee400d6

README update for envs

5c4c2d4

Merge updates to autopilotpattern#46

33ceafc

more things

32ed8fc

jwreagor mentioned this pull request Dec 13, 2017

Add CONSUL_DATACENTER_NAME env #46

Merged

it was bind_addr all along

5bec8ff

tjcelaya added 2 commits December 13, 2017 15:54

Always set the alternate addresses for now, might move all of these b…

25de3dd

…ack into the CONSUL_RETRY_JOIN_WAN block

Fix the CONSUL_BIND_ADDR check

0c3db58

Add self-anti-affinity

b44294b

tjcelaya added 2 commits December 14, 2017 16:26

Fix silly config error

3ebb138

Move env vars section

24ae977

Resolve merge conflicts

2f2893d

jwreagor suggested changes Dec 18, 2017

View reviewed changes

tjcelaya added 3 commits December 18, 2017 16:01

Split up triton from multi-dc triton examples

ac0d6e2

Remove fake multi-dc docker-compose example

a84a860

Fix test Dockerfile paths

667f0e2

tjcelaya added 2 commits December 18, 2017 16:38

Fix multi-dc clean target

5ab7825

Fix template name

efb1e9b

tjcelaya added 4 commits December 18, 2017 16:52

Try a new multi-dc layout

ff95e50

Separate the triton and triton-multi-dc setup scripts

db96dd7

make the check function easier to use

eb360fc

Fix account id query command

3c6f336

tjcelaya added 2 commits December 19, 2017 11:03

Path and script name change in README

8b27a61

Clarify README

06bc9ea

jwreagor approved these changes Dec 19, 2017

View reviewed changes

tjcelaya mentioned this pull request Dec 19, 2017

Add TLS configuration #49

Open

tjcelaya added 3 commits December 19, 2017 17:36

Fix duplicate env vars from merge conflict resolution

6759213

Fix usage output description

9ea5646

Fix description again

ecc75ed

tjcelaya mentioned this pull request Dec 21, 2017

Add HTTPS/RPC/gossip encryption and setup script #50

Open

tjcelaya added 3 commits December 20, 2017 16:44

Remove the build key from examples/compose/docker-compose.yml

658d4e7

Revert ./setup.sh change

4cf52d9

Typo

dc5018f

tjcelaya merged commit e4f9031 into autopilotpattern:master Jan 5, 2018

tjcelaya deleted the enhancement/47-retry-join-wan-in-config-file branch January 5, 2018 23:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CONSUL_RETRY_JOIN_WAN env #48

Add CONSUL_RETRY_JOIN_WAN env #48

tjcelaya commented Dec 8, 2017

jwreagor left a comment

jwreagor Dec 11, 2017

jwreagor commented Dec 11, 2017

tjcelaya commented Dec 13, 2017 •

edited

Loading

tjcelaya commented Dec 13, 2017 •

edited

Loading

tjcelaya commented Dec 14, 2017

tjcelaya commented Dec 14, 2017

jwreagor commented Dec 15, 2017

jwreagor left a comment

jwreagor Dec 18, 2017

jwreagor Dec 18, 2017

jwreagor Dec 18, 2017

tjcelaya Dec 19, 2017

jwreagor Dec 18, 2017

jwreagor Dec 18, 2017

tjcelaya Dec 19, 2017

jwreagor Dec 19, 2017

tjcelaya Dec 19, 2017

jwreagor Dec 18, 2017

jwreagor Dec 18, 2017

tjcelaya commented Dec 19, 2017

tjcelaya commented Dec 19, 2017

tjcelaya commented Dec 22, 2017

jwreagor commented Jan 4, 2018

		@@ -42,9 +42,11 @@ check() {
		# make sure Docker client is pointed to the same place as the Triton client

Add CONSUL_RETRY_JOIN_WAN env #48

Add CONSUL_RETRY_JOIN_WAN env #48

Conversation

tjcelaya commented Dec 8, 2017

jwreagor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jwreagor commented Dec 11, 2017

tjcelaya commented Dec 13, 2017 • edited Loading

tjcelaya commented Dec 13, 2017 • edited Loading

tjcelaya commented Dec 14, 2017

tjcelaya commented Dec 14, 2017

jwreagor commented Dec 15, 2017

jwreagor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tjcelaya commented Dec 19, 2017

tjcelaya commented Dec 19, 2017

tjcelaya commented Dec 22, 2017

jwreagor commented Jan 4, 2018

tjcelaya commented Dec 13, 2017 •

edited

Loading

tjcelaya commented Dec 13, 2017 •

edited

Loading