a gentle touch of Clojure to Hashicorp's Consul
Clojure
Latest commit 6e45e64 Jan 10, 2017 @tolitius committed on GitHub [docs]: note about merging exiting configs

README.md

Diplomatic rank

source:

The rank of Envoy was short for "Envoy Extraordinary and Minister Plenipotentiary", and was more commonly known as Minister. For example, the Envoy Extraordinary and Minister Plenipotentiary of the United States to the French Empire was known as the "United States Minister to France" and addressed as "Monsieur le Ministre."

Clojars Project

How to play

In order to follow all the docs below, bring envoy in:

$ boot repl
boot.user=> (require '[envoy.core :as envoy :refer [stop]])
nil

Map to Consul

Since most Clojure configs are EDN maps, you can simply push the map to Consul with preserving the hierarchy:

boot.user=> (def m {:hubble
                    {:store "spacecraft://tape"
                     :camera 
                      {:mode "color"}
                     :mission
                      {:target "Horsehead Nebula"}}})

boot.user=> (envoy/map->consul "http://localhost:8500/v1/kv" m)
nil

done.

you should see Consul logs confirming it happened:

2016/11/02 02:04:13 [DEBUG] http: Request PUT /v1/kv/hubble/mission/target? (337.69µs) from=127.0.0.1:39372
2016/11/02 02:04:13 [DEBUG] http: Request GET /v1/kv/hubble?recurse&index=2114 (4m41.723665304s) from=127.0.0.1:39366
2016/11/02 02:04:13 [DEBUG] http: Request PUT /v1/kv/hubble/camera/mode? (373.246µs) from=127.0.0.1:39372
2016/11/02 02:04:13 [DEBUG] http: Request PUT /v1/kv/hubble/store? (1.607247ms) from=127.0.0.1:39372

and a visual:

Consul to Map

In case a Clojure map with config read from Consul is needed it is just consul->map away:

boot.user=> (envoy/consul->map "http://localhost:8500/v1/kv/hubble")
{:hubble
 {:camera {:mode "color"},
  :mission {:target "Horsehead Nebula"},
  :store "spacecraft://tape"}}

you may notice it comes directly from "the source" by looking at Consul logs:

2016/11/02 02:04:32 [DEBUG] http: Request GET /v1/kv/hubble?recurse (76.386µs) from=127.0.0.1:54167

Watch for key/value changes

Watching for kv changes with envoy does not require to run a separate Consul Agent client or Consul Template and boils down to a simple function:

(watch-path path fun)

fun is going to be called with a new value each time the path's value is changed.

boot.user=> (def store-watcher (envoy/watch-path "http://localhost:8500/v1/kv/hubble/store"
                                                 #(println "watcher says:" %)))

creates a envoy.core.Watcher and echos back the current value:

#'boot.user/store-watcher
watcher says: {:hubble/store spacecraft}

it is an envoy.core.Watcher:

boot.user=> store-watcher
#object[envoy.core.Watcher 0x72a190f0 "envoy.core.Watcher@72a190f0"]

that would print to REPL, since that's the function provided #(println "watcher says:" %), every time the key hubble/store changes.

let's change it to "Earth":

once the "UPDATE" button is clicked REPL will notify us with a new value:

watcher says: {:hubble/store Earth}

same thing if it's changed with envoy/put:

boot.user=> (envoy/put "http://localhost:8500/v1/kv/hubble/store" "spacecraft tape")
watcher says: {:hubble/store spacecraft tape}
{:opts {:body "spacecraft tape", :method :put, :url "http://localhost:8500/v1/kv/hubble/store"}, :body "true", :headers {:content-length "4", :content-type "application/json", :date "Wed, 02 Nov 2016 03:22:41 GMT"}, :status 200}

envoy.core.Watcher is stoppable:

boot.user=> (stop store-watcher)
"stopping" "http://localhost:8500/v1/kv/hubble/store" "watcher"
true

Watch Nested Keys

In case you need to watch a hierarchy of keys (with all the nested keys), you can set a watcher on a local root key:

boot.user=> (def hw (envoy/watch-path "http://localhost:8500/v1/kv/hubble"
                                      #(println "watcher says:" %)))

notice this watcher is on the top most / root /hubble key.

In this case only the nested keys which values are changed will trigger notifications.

Let's say we went to hubble/mission and changed it from "Horsehead Nebula" to "Butterfly Nebula":

watcher says: {:hubble/mission Butterfly Nebula}

It can be stopped as any other watcher:

boot.user=> (stop hw)
"stopping" "http://localhost:8500/v1/kv/hubble?recurse" "watcher"
true 

Watching the Watcher

There is a more visual example of envoy watchers that propagate notifications all the way to the browser:

Notification listner is just a function really, hence it can get propagated anywhere intergalactic computer system can reach.

Consul CRUD

Adding to Consul

The map from above can be done manually by "puts" of course:

boot.user=> (envoy/put "http://localhost:8500/v1/kv/hubble/mission" "Horsehead Nebula")
{:opts {:body "Horsehead Nebula", :method :put, :url "http://localhost:8500/v1/kv/hubble/mission"}, :body "true", :headers {:content-length "4", :content-type "application/json", :date "Wed, 02 Nov 2016 02:57:40 GMT"}, :status 200}

boot.user=> (envoy/put "http://localhost:8500/v1/kv/hubble/store" "spacecraft")
{:opts {:body "spacecraft", :method :put, :url "http://localhost:8500/v1/kv/hubble/store"}, :body "true", :headers {:content-length "4", :content-type "application/json", :date "Wed, 02 Nov 2016 02:58:13 GMT"}, :status 200}

boot.user=> (envoy/put "http://localhost:8500/v1/kv/hubble/camera/mode" "color")
{:opts {:body "color", :method :put, :url "http://localhost:8500/v1/kv/hubble/camera/mode"}, :body "true", :headers {:content-length "4", :content-type "application/json", :date "Wed, 02 Nov 2016 02:58:36 GMT"}, :status 200}

Reading from Consul

boot.user=> (envoy/get-all "http://localhost:8500/v1/kv/hubble")
{:hubble/camera/mode "color",
 :hubble/mission "Horsehead Nebula",
 :hubble/store "spacecraft://tape"}

boot.user=> (envoy/get-all "http://localhost:8500/v1/kv/hubble/store")
{:hubble/store "spacecraft"}

in case there is no need to convert keys to keywords, it can be disabled:

boot.user=> (envoy/get-all "http://localhost:8500/v1/kv/" {:keywordize? false})
{"hubble/camera/mode" "color",
 "hubble/mission" "Horsehead Nebula",
 "hubble/store" "spacecraft://tape"}

Deleting from Consul

boot.user=> (envoy/delete "http://localhost:8500/v1/kv/hubble/camera")
{:opts {:method :delete, :url "http://localhost:8500/v1/kv/hubble/camera?recurse"}, :body "true", :headers {:content-length "4", :content-type "application/json", :date "Wed, 02 Nov 2016 02:59:26 GMT"}, :status 200}

boot.user=> (envoy/get-all "http://localhost:8500/v1/kv/hubble")
{:hubble/mission "Horsehead Nebula", :hubble/store "spacecraft://tape"}

Merging Configurations

Often there is an internal configuration some parts of which need to be overridden with values from Consul. Envoy has merge-with-consul function that does just that:

(envoy/merge-with-consul config "http://localhost:8500/v1/kv/hubble")

will deep merge (with nested kvs) config with a map it'll read from Consul.

In case a Consul space is protected by a token, or any other options need to be passed to Consul to read the overrides, they can be added in an optional map:

(envoy/merge-with-consul config
                         "http://localhost:8500/v1/kv/hubble"
                         {:token "7a0f3b39-8871-e16e-2101-c1b30a911883"})

Options

All commands take an optional map of options.

For example, in case keys are protected by ACL, you can provide a token:

boot.user=> (envoy/consul->map "http://localhost:8500/v1/kv"
                               {:token "4c308bb2-16a3-4061-b678-357de559624a"})
{:hubble {:mission "Butterfly Nebula", :store "spacecraft://ssd"}}

or any other Consul options.

License

Copyright © 2016 tolitius

Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.