The rank of Envoy was short for "Envoy Extraordinary and Minister Plenipotentiary", and was more commonly known as Minister. For example, the Envoy Extraordinary and Minister Plenipotentiary of the United States to the French Empire was known as the "United States Minister to France" and addressed as "Monsieur le Ministre."
- How to play
- Map to Consul
- Consul to Map
- Watch for key/value changes
- Consul CRUD
- Clone and Teleport
- Merging Configurations
- Sessions and Locks
- Options
- License
In order to follow all the docs below, bring envoy in:
$ make repl
dev=> (require '[envoy.core :as envoy :refer [stop]])
nil
Since most Clojure configs are EDN maps, you can simply push the map to Consul with preserving the hierarchy:
dev=> (def m {:hubble
{:store "spacecraft://tape"
:camera
{:mode "color"}
:mission
{:target "Horsehead Nebula"}}})
dev=> (envoy/map->consul "http://localhost:8500/v1/kv" m)
nil
done.
you should see Consul logs confirming it happened:
2016/11/02 02:04:13 [DEBUG] http: Request PUT /v1/kv/hubble/mission/target? (337.69µs) from=127.0.0.1:39372
2016/11/02 02:04:13 [DEBUG] http: Request GET /v1/kv/hubble?recurse&index=2114 (4m41.723665304s) from=127.0.0.1:39366
2016/11/02 02:04:13 [DEBUG] http: Request PUT /v1/kv/hubble/camera/mode? (373.246µs) from=127.0.0.1:39372
2016/11/02 02:04:13 [DEBUG] http: Request PUT /v1/kv/hubble/store? (1.607247ms) from=127.0.0.1:39372
and a visual:
In case a Clojure map with config read from Consul is needed it is just consul->map
away:
dev=> (envoy/consul->map "http://localhost:8500/v1/kv/hubble")
{:hubble
{:camera {:mode "color"},
:mission {:target "Horsehead Nebula"},
:store "spacecraft://tape"}}
you may notice it comes directly from "the source" by looking at Consul logs:
2016/11/02 02:04:32 [DEBUG] http: Request GET /v1/kv/hubble?recurse (76.386µs) from=127.0.0.1:54167
You may also read from consul at a certain :offset
by specifying it in options.
Let's say we need to get everything that lives under the hubble/mission
:
dev=> (envoy/consul->map "http://localhost:8500/v1/kv" {:offset "hubble/mission"})
{:target "Horsehead Nebula"}
Specifying an offset is really useful for multiple environments or teams living in the same consul / acl.
(!) One thing to note is that an offset
should start from the root and should be used with no data prefix in URL:
i.e. this is good:
(envoy/consul->map "http://localhost:8500/v1/kv" {:offset "/hubble/mission"})
but this is not:
(envoy/consul->map "http://localhost:8500/v1/kv/hubble" {:offset "mission"})
this has to do with the fact that Consul always returns data with a prefix e.g.: {:hubble {:mission {:target "Horsehead Nebula"}}}
, hence just "mission" would not be enough to strip it, but "hubble/mission" would.
Watching for kv changes with envoy does not require to run a separate Consul Agent client or Consul Template and boils down to a simple function:
(watch-path path fun)
fun
is going to be called with a new value each time the path
's value is changed.
dev=> (def store-watcher (envoy/watch-path "http://localhost:8500/v1/kv/hubble/store"
#(println "watcher says:" %)))
creates a envoy.core.Watcher
and echos back the current value:
#'dev/store-watcher
watcher says: {:hubble/store spacecraft}
it is an envoy.core.Watcher
:
dev=> store-watcher
#object[envoy.core.Watcher 0x72a190f0 "envoy.core.Watcher@72a190f0"]
that would print to REPL, since that's the function provided #(println "watcher says:" %)
, every time the key hubble/store
changes.
let's change it to "Earth":
once the "UPDATE" button is clicked REPL will notify us with a new value:
watcher says: {:hubble/store Earth}
same thing if it's changed with envoy/put
:
dev=> (envoy/put "http://localhost:8500/v1/kv/hubble/store" "spacecraft tape")
watcher says: {:hubble/store spacecraft tape}
{:opts {:body "spacecraft tape", :method :put, :url "http://localhost:8500/v1/kv/hubble/store"}, :body "true", :headers {:content-length "4", :content-type "application/json", :date "Wed, 02 Nov 2016 03:22:41 GMT"}, :status 200}
envoy.core.Watcher
is stoppable:
dev=> (stop store-watcher)
"stopping" "http://localhost:8500/v1/kv/hubble/store" "watcher"
true
In case you need to watch a hierarchy of keys (with all the nested keys), you can set a watcher on a local root key:
dev=> (def hw (envoy/watch-path "http://localhost:8500/v1/kv/hubble"
#(println "watcher says:" %)))
notice this watcher is on the top most / root /hubble
key.
In this case only the nested keys which values are changed will trigger notifications.
Let's say we went to hubble/mission
and changed it from "Horsehead Nebula" to "Butterfly Nebula":
watcher says: {:hubble/mission Butterfly Nebula}
It can be stopped as any other watcher:
dev=> (stop hw)
"stopping" "http://localhost:8500/v1/kv/hubble?recurse" "watcher"
true
There is a more visual example of envoy watchers that propagate notifications all the way to the browser:
Notification listner is just a function really, hence it can get propagated anywhere intergalactic computer system can reach.
The map from above can be done manually by "puts" of course:
dev=> (envoy/put "http://localhost:8500/v1/kv/hubble/mission" "Horsehead Nebula")
{:opts {:body "Horsehead Nebula", :method :put, :url "http://localhost:8500/v1/kv/hubble/mission"}, :body "true", :headers {:content-length "4", :content-type "application/json", :date "Wed, 02 Nov 2016 02:57:40 GMT"}, :status 200}
dev=> (envoy/put "http://localhost:8500/v1/kv/hubble/store" "spacecraft")
{:opts {:body "spacecraft", :method :put, :url "http://localhost:8500/v1/kv/hubble/store"}, :body "true", :headers {:content-length "4", :content-type "application/json", :date "Wed, 02 Nov 2016 02:58:13 GMT"}, :status 200}
dev=> (envoy/put "http://localhost:8500/v1/kv/hubble/camera/mode" "color")
{:opts {:body "color", :method :put, :url "http://localhost:8500/v1/kv/hubble/camera/mode"}, :body "true", :headers {:content-length "4", :content-type "application/json", :date "Wed, 02 Nov 2016 02:58:36 GMT"}, :status 200}
dev=> (envoy/get-all "http://localhost:8500/v1/kv/hubble")
{:hubble/camera/mode "color",
:hubble/mission "Horsehead Nebula",
:hubble/store "spacecraft://tape"}
dev=> (envoy/get-all "http://localhost:8500/v1/kv/hubble/store")
{:hubble/store "spacecraft"}
in case there is no need to convert keys to keywords, it can be disabled:
dev=> (envoy/get-all "http://localhost:8500/v1/kv/" {:keywordize? false})
{"hubble/camera/mode" "color",
"hubble/mission" "Horsehead Nebula",
"hubble/store" "spacecraft://tape"}
dev=> (envoy/delete "http://localhost:8500/v1/kv/hubble/camera")
{:opts {:method :delete, :url "http://localhost:8500/v1/kv/hubble/camera?recurse"}, :body "true", :headers {:content-length "4", :content-type "application/json", :date "Wed, 02 Nov 2016 02:59:26 GMT"}, :status 200}
dev=> (envoy/get-all "http://localhost:8500/v1/kv/hubble")
{:hubble/mission "Horsehead Nebula", :hubble/store "spacecraft://tape"}
It is often the case when configuration trees need to be copied or moved from one place to another, under a new root or a new nested path. envoy
can do it with copy
and move
commands.
Copying configuration from one place to another is done with a copy
command:
=>
(envoy/copy kv-path from to)
Let's say we need to copy Hubble's mission (i.e. a "sub" config) under a new root "dev", so it lives under "/dev/hubble/mission" instead:
dev=> (envoy/copy "http://localhost:8500/v1/kv" "/hubble/mission" "/dev/hubble/mission")
done. Let's read from this new "dev" root to make sure the mission is there:
dev=> (envoy/consul->map "http://localhost:8500/v1/kv/dev")
{:dev {:hubble {:mission {:target "Horsehead Foo"}}}}
great.
We can of course copy the whole "hubble"'s config under "dev":
dev=> (envoy/copy "http://localhost:8500/v1/kv" "/hubble" "/dev/hubble")
dev=> (envoy/consul->map "http://localhost:8500/v1/kv/dev")
{:dev {:hubble {:camera {:mode "color"}, :mission {:target "Horsehead Nebula"}, :store "spacecraft tape"}}}
copy
is really handy when you need to copy configurations between environments, or just need to copy some nested portion of the config.
A move
command is exactly the same as the copy
, but, as you would expect, it deletes the source after the copy is done.
=>
(envoy/move kv-path from to)
The Hubble's development work is finished, and we are switching to work on the Kepler telescope. Let's say most of the configuration may be reused, so we'll just move Hubble's config to Kepler:
dev=> (envoy/move "http://localhost:8500/v1/kv" "/hubble" "/kepler")
done.
Oh, but we'll need "dev" and "qa" environments for Kepler's development. Let's move it again to live under "dev" root:
dev=> (envoy/move "http://localhost:8500/v1/kv" "/kepler" "/dev/kepler")
and "copy" this config to "qa" before editing it:
dev=> (envoy/copy "http://localhost:8500/v1/kv" "/dev/kepler" "/qa/kepler")
Let's look at Kepler's Consul universe:
dev=> (envoy/consul->map "http://localhost:8500/v1/kv")
{:dev
{:kepler
{:mission {:target "Horsehead Nebula"},
:camera {:mode "color"},
:store "spacecraft tape"}},
:qa
{:kepler
{:mission {:target "Horsehead Nebula"},
:camera {:mode "color"},
:store "spacecraft tape"}}}
Niice, universe awaits...
Often there is an internal configuration some parts of which need to be overridden with values from Consul. Envoy has merge-with-consul
function that does just that:
(envoy/merge-with-consul config "http://localhost:8500/v1/kv/hubble")
will deep merge (with nested kvs) config with a map it'll read from Consul.
In case a Consul space is protected by a token, or any other options need to be passed to Consul to read the overrides, they can be added in an optional map:
(envoy/merge-with-consul config
"http://localhost:8500/v1/kv/hubble"
{:token "7a0f3b39-8871-e16e-2101-c1b30a911883"})
Consul provides a session mechanism which can be used to build distributed locks.
Sessions can be created, deleted, listed, listed for a node, etc. Once the session is created locks can be acquired for these sessions.
Consul session API have multiple parameters, do read the Consul docs in case you need to deviate from defaults.
An example we'll look at here will:
- create two sessions with TTL 45 seconds (
moon
andmars
) - acquire a
start-stage-one-booster
lock for themoon
session - try to acquire the same lock with
mars
(fail) - wait until the first,
moon
session expires - renew the
mars
session to make sure we can still use it after themoon
expires - try to acquire the same lock with the
mars
session again (succeed)
(require '[envoy.session :as es])
=> (def moon (es/create-session "http://localhost:8500" {:name "fly-me-to-the-moon" :ttl "45s"}))
#'user/moon
=> (def mars (es/create-session "http://localhost:8500" {:name "fly-me-to-the-mars" :ttl "45s"}))
#'user/mars
=> moon
{:id "3f230917-90c9-6b5e-f579-43d854ba9cfe"}
=> mars
{:id "371445e6-d74f-9524-de06-8622fa4344cd"}
sessions are created and Consul returned session ids to refer to these sessions by.
let's list these sessions:
=> (es/list-sessions "http://localhost:8500")
[{:service-checks nil,
:modify-index 18157665,
:name "fly-me-to-the-mars",
:behavior "release",
:node "pluto",
:ttl "45s",
:id "371445e6-d74f-9524-de06-8622fa4344cd",
:create-index 18157665,
:lock-delay 15000000000,
:node-checks ["serfHealth"]}
{:service-checks nil,
:modify-index 18157655,
:name "fly-me-to-the-moon",
:behavior "release",
:node "pluto",
:ttl "45s",
:id "3f230917-90c9-6b5e-f579-43d854ba9cfe",
:create-index 18157655,
:lock-delay 15000000000,
:node-checks ["serfHealth"]}]
now let's acquire a start-stage-one-booster
lock with the moon
session:
=> (es/acquire-lock "http://localhost:8500" {:task "start-stage-one-booster"
:session-id (moon :id)})
true
"true" means Consul said the lock was successfully acquired.
now let's try to acquire the same lock with the mars
session:
=> (es/acquire-lock "http://localhost:8500" {:task "start-stage-one-booster"
:session-id (mars :id)})
false
ooops, can't touch this: since this lock is already acquired by another session.
while we are waiting on the moon
session to expire (that "45s" TTL), let's renew the mars
session so it outlives the moon
one:
=> (es/renew-session "http://localhost:8500" {:uuid (mars :id)})
[{:service-checks nil,
:modify-index 18157665,
:name "fly-me-to-the-mars",
:behavior "release",
:node "pluto",
:ttl "45s",
:id "371445e6-d74f-9524-de06-8622fa4344cd",
:create-index 18157665,
:lock-delay 15000000000,
:node-checks ["serfHealth"]}]
after waiting for over 45 seconds, let's try to acquire the same lock with the mars
session again:
=> (es/acquire-lock "http://localhost:8500" {:task "start-stage-one-booster"
:session-id (mars :id)})
true
great success!
by the way we also get a visual:
all the params and options can still be used with session api.
for example to pass a "data center" and an "ACL token":
=> (es/create-session "http://localhost:8500" {:dc "asteroid-belt" :name "fly-me-to-the-moon" :ttl "45s"}
{:token "73e5a965-40af-9c70-a817-b065b6ef82db"})
the reason they are in two maps is because a dc
is part of create session API parameters, whereas a token is a general Consul param.
All commands take an optional map of parameters. These parameters will get converted into Consul KV Store Endpoints params. Thus making all of the KV Store Endpoint params supported.
For example, in case keys are protected by ACL, you can provide a token:
dev=> (envoy/consul->map "http://localhost:8500/v1/kv"
{:token "4c308bb2-16a3-4061-b678-357de559624a"})
{:hubble {:mission "Butterfly Nebula", :store "spacecraft://ssd"}}
or a token and a datacenter:
dev=> (envoy/consul->map "http://localhost:8500/v1/kv"
{:token "63aaa731-b124-40ef-9425-978aba612a1d"
:dc "phloston"})
{:hubble {:mission "Ghost of Jupiter", :store "spacecraft://tape"}}
or any other Consul supported parameters.
By default envoy will serialize and deserialize data in EDN format. Which usually is quite transparent, since EDN map gets written and read from Consul as a nested key value structure which is just that: a map.
There are cases where values are sequences: i.e. {:hosts ["foo1.com", "foo2.com"]}
in which case they will still be serialized and deserialized as EDN by default, however it might be harder to consume these EDN sequences from languages other than Clojure which do not speak EDN natively.
While there are libraries for other languages that support EDN
- Java: https://github.com/danboykis/trava
- Go: https://github.com/go-edn/edn
- Ruby: https://github.com/relevance/edn-ruby
- Python: https://github.com/swaroopch/edn_format
- etc.
envoy would allow to specify other serializers via a :serializer
option:
dev=> (def config {:system {:hosts ["foo1.com", "foo2.com", {:a 42}]}})
#'dev/config
dev=> (envoy/map->consul "http://localhost:8500/v1/kv" config {:serializer :json})
nil
dev=> (envoy/consul->map "http://localhost:8500/v1/kv/system" {:serializer :json})
{:system {:hosts ["foo1.com" "foo2.com" {:a 42}]}}
If {:serializer :json}
option is provided sequence values will be stored in Consul as JSON:
which can be consumed from other languages without the need to know about EDN.
Copyright © 2023 tolitius
Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.