Support for Triton discovery #18

misterbisson · 2017-05-07T04:38:01Z

Carries from #17:

Update Prometheus to 1.5.2

Update ContainerPilot to 2.7.2

Update to Consul 0.7.2

Add support for Triton metrics

Re-organize repo to bring it in line with other Autopilot blueprints

Add CI tests

Fixes #16, #8, and #5

This addresses code review questions and issues in addition to the above.

Next steps:

Autodetect the Triton DC, per Data center confusion #19
Update /README.md with more details, including:
Add a note about needing to init/update the submodule when pulling it down
Update /examples/triton/README.md with more detail, including a note about copying the user's TLS key
Update /test/tests.sh so that it sets all the correct env vars
Resolve makefile bugs with setting vars for local docker and docker-compose commands

… container names by test harness

misterbisson · 2017-05-07T04:43:12Z

Makefile

+image := $(namespace)/prometheus
+testImage := $(namespace)/prometheus-testrunner
+
+#dockerLocal := DOCKER_HOST= DOCKER_TLS_VERIFY= DOCKER_CERT_PATH= docker


This does not work on my Mac with Docker Machine. Replacing it with dockerLocal := docker does.

Error is:

$ make build DOCKER_HOST= DOCKER_TLS_VERIFY= DOCKER_CERT_PATH= docker build -f test/Dockerfile -t=autopilotpattern/prometheus-testrunner:branch-triton-support . Cannot connect to the Docker daemon. Is the docker daemon running on this host? make: *** [test-runner] Error 1

Hm, that's unfortunate and is because Docker Machine uses DOCKER_HOST, etc. and Docker for Mac does not. The reason we're doing this is because we want the person (or Jenkins job) running the tests to be able to set their credentials to point to the environment-under-test but then run the tests from the local machine so that the keys never leave the machine running the tests.

This will effect every blueprint where we're doing testing this way. I'm certainly open to new suggestions on how to tackle it though.

This is now fixed by using https://github.com/joyent/triton-docker-cli, though the code here hasn't been updated.

misterbisson · 2017-05-07T04:44:36Z

test/Dockerfile

+COPY examples/triton/docker-compose.yml /src/triton/docker-compose.yml
+
+# install tests
+COPY test/testing/testcases.py /src/testcases.py


Step 7 : COPY test/testing/testcases.py /src/testcases.py lstat test/testing/testcases.py: no such file or directory make: *** [test-runner] Error 1

Yeah, I'm an idiot. That was a submodule and I need to fetch its contents.

misterbisson · 2017-05-07T05:13:12Z

docker-compose.yml

@@ -1,31 +0,0 @@
-# Prometheus demonstration of the autopilot pattern


The new pattern is to put this in /examples/triton

misterbisson · 2017-05-07T05:13:54Z

local-compose.yml

@@ -1,19 +0,0 @@
-prometheus:


The new pattern is to put this in /examples/docker/docker-compose.yml

misterbisson · 2017-05-07T05:15:06Z

Dockerfile

+RUN set -ex \
+    && export CONSUL_VERSION=0.7.5 \
+    && export CONSUL_CHECKSUM=40ce7175535551882ecdff21fdd276cef6eaab96be8a8260e0599fadb6f1f5b8 \
+    && curl --retry 7 --fail -vo /tmp/consul.zip "https://releases.hashicorp.com/consul/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip" \


Adding Consul as a coprocess, per #8

misterbisson · 2017-05-07T05:16:23Z

etc/containerpilot.json

  "services": [
    {
      "name": "prometheus",
      "port": 9090,
-      "health": ["curl", "-so", "/dev/null", "http://localhost:9090/metrics"],
+      "health": ["curl", "-fso", "/dev/null", "http://localhost:9090/metrics"],


We expect if the request fails that curl will return a non-zero exit, yes?

misterbisson · 2017-05-07T05:16:41Z

etc/containerpilot.json

      "poll": 10,
      "ttl": 25
    }
+  ],
+  "coprocesses": [{{ if .CONSUL_AGENT }}


Consul co-process, per #8

misterbisson · 2017-05-07T05:17:08Z

etc/prometheus.yml.ctmpl

+       dns_suffix: 'cmon.{{env "TRITON_DC"}}.triton.zone'
+       endpoint: 'cmon.{{env "TRITON_DC"}}.triton.zone'
+       tls_config:
+        - ca_file: '{{env "TLSCA_PATH"}}'


These var names are incorrect and need to be fixed.

These must match the new vars added to https://github.com/autopilotpattern/prometheus/pull/18/files#diff-86aea33913dc22bc9ba97b5f7b8c956e

misterbisson · 2017-05-07T05:18:39Z

examples/triton/docker-compose.yml

+
+
+  # Consul acts as our service catalog and is used to coordinate global state among 
+  # our Redis containers


misterbisson · 2017-05-07T05:19:59Z

test/tests.sh

@@ -0,0 +1,4 @@
+echo CONSUL=prometheus-consul.svc.${TRITON_ACCOUNT}.${TRITON_DC}.cns.joyent.com > /src/triton/_env


This will need updating to include the changes in /examples/triton/setup.sh

- Make the config file (appear to) work

misterbisson · 2017-05-07T23:07:33Z

Dockerfile

@@ -4,55 +4,83 @@ FROM alpine:3.4
 # artisanally hand-rolling curl and the rest of our stack we'll just use
 # Alpine so we can use `docker build`.

-RUN apk add --update curl
+RUN apk add --update \
+    bash \


It turns out the -n test in the prestart.sh is bash-specific. Of all the choices I had to fix it, I decided to add bash.

misterbisson · 2017-05-07T23:09:06Z

etc/containerpilot.json

+      "name": "consul-agent (host:{{ .CONSUL }})",
+      "command": ["/usr/local/bin/consul", "agent",
+                  "-data-dir=/var/lib/consul",
+                  "-config-dir=/etc/consul",


The alignment of these two dirs with the dirs created in the Dockerfile is frustratingly important.

misterbisson · 2017-05-07T23:10:18Z

etc/prometheus.yml.ctmpl

@@ -1,8 +1,8 @@
 # my global config
 global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
-  evaluation_interval: 15s # By default, scrape targets every 15 seconds.
-  # scrape_timeout is set to the global default (10s).
+  evaluation_interval: 13s #


The examples show the evaluation_interval slightly shorter than the scrape_interval, so...

Fixes #19

misterbisson · 2017-06-16T19:02:24Z

bin/prestart.sh

+
+# Are we on Triton? Do we _not_ have a user-defined DC?
+# Set the DC automatically from mdata
+if [ -n ${TRITON_DC} ] \


@geek did you spot this one?

misterbisson · 2017-07-05T17:38:07Z

Closing in favor of #21

jasonpincin and others added 13 commits April 11, 2017 16:30

Added scripts for prestart and start which configure TLS

e3768af

Updated ContainerPilot, Prometheus, and Consul Template

e4b5c97

Call prestart.sh

2e6112a

Add conditional Triton scrape configuration

49d9b13

Write triton account/dc to _env for use in prometheus config

609a61e

Tests scaffolded, Triton example added

e9f02da

Add compose example and Makefile

a4a031d

corrected consul namespace

ca2c2a8

linked and identified consul for compose example

6137dfd

shorten project name to overcome 23 character limit imposed on docker…

43aa0d3

… container names by test harness

Update setup scripts to namespace consul CNS

1b92dd8

Corrected consul CNS namespace

fc462ae

env vars and creds

9ebfd59

misterbisson mentioned this pull request May 7, 2017

Triton support #17

Closed

misterbisson commented May 7, 2017

View reviewed changes

local-compose.yml

@@ -1,19 +0,0 @@

prometheus:

Copy link

Contributor Author

misterbisson May 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new pattern is to put this in /examples/docker/docker-compose.yml

misterbisson commented May 7, 2017

View reviewed changes

fix creds paths and varnames

c02a952

- Make the config file (appear to) work

misterbisson force-pushed the triton-support branch from de150b6 to c02a952 Compare May 7, 2017 22:26

misterbisson commented May 7, 2017

View reviewed changes

misterbisson mentioned this pull request May 8, 2017

Data center confusion #19

Closed

autodetect triton dc

fe89909

Fixes #19

misterbisson mentioned this pull request May 18, 2017

Update Nginx, add makefile autopilotpattern/wordpress#48

Merged

misterbisson mentioned this pull request Jun 16, 2017

Support latest versions and consul agent #20

Merged

geek mentioned this pull request Jun 16, 2017

Support triton metrics #21

Merged

misterbisson commented Jun 16, 2017

View reviewed changes

misterbisson closed this Jul 5, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for Triton discovery #18

Support for Triton discovery #18

misterbisson commented May 7, 2017 •

edited

Loading

misterbisson May 7, 2017

tgross May 10, 2017

misterbisson Jun 16, 2017

misterbisson May 7, 2017

misterbisson May 7, 2017

misterbisson May 7, 2017

misterbisson May 7, 2017

misterbisson May 7, 2017

misterbisson May 7, 2017 •

edited

Loading

tgross May 10, 2017

misterbisson May 7, 2017

misterbisson May 7, 2017 •

edited

Loading

misterbisson May 7, 2017

misterbisson May 7, 2017

misterbisson May 7, 2017

misterbisson May 7, 2017

misterbisson May 7, 2017

misterbisson Jun 16, 2017

misterbisson commented Jul 5, 2017

		@@ -1,31 +0,0 @@
		# Prometheus demonstration of the autopilot pattern



		# Consul acts as our service catalog and is used to coordinate global state among
		# our Redis containers

		@@ -0,0 +1,4 @@
		echo CONSUL=prometheus-consul.svc.${TRITON_ACCOUNT}.${TRITON_DC}.cns.joyent.com > /src/triton/_env

Support for Triton discovery #18

Support for Triton discovery #18

Conversation

misterbisson commented May 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

misterbisson May 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

misterbisson May 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

misterbisson commented Jul 5, 2017

misterbisson commented May 7, 2017 •

edited

Loading

misterbisson May 7, 2017 •

edited

Loading

misterbisson May 7, 2017 •

edited

Loading