Cuttlefish for Erlang Developers

Note: If you are looking for information in how to "wire" Cuttlefish to your application, consult Cuttlefish for node_package users or Cuttlefish for non-node_package users.

As an Erlang developer, you're probably used to application:set_env and app.config files. The good news for you is that you can keep on coding that way! The... additional news is that people who are using your application may not understand that syntax so easily. So how can you help? I'm glad you asked!

Write Cuttlefish Schemas!

You have a new job. You get to choose which knobs you expose to users! You can choose to name these things anything you want, so where you previously might have been confined to including a dependency's application name, you are now not.

You can define datatypes for these settings, and you can explain to Cuttlefish how a simple name-value pair becomes part of a complex hierarchy of Erlang terms!

Want to see how?

As the Erlang developer, you are the person responsible for being the ambassador of your setting to the world. Here's how it looks in ASCII art:

┌--------------------┐                                                                      ┌--------------------┐
| <app>.conf         |                                                                      | app.config         |
|--------------------|       {mapping,                       {translation,                  | -------------------|
| my.setting = value | --->   "my.setting",            --->   "<dependency>.setting",  ---> | [{<dependency>, [  |
| ...                |        "<dependency>.setting",         fun(Conf) ->                  |    {setting, value}|
|                    |        []}.                              %% erlang fun               |  ]}, ... ].        |
└--------------------┘                                        end}.                         └--------------------┘

The Schema

There are three types of Schema elements in Cuttlefish: mapping, translation and validator. It's easy to tell the difference! They're all tuples, and the first element is an atom: mapping, translation, or validator. You're welcome.

Mappings

@ - Annotations

Mappings are the only schema element type that support annotations and there are two available: @doc and @see.

@doc: If you write a multiline @doc it will be included in your generated .conf file. These docs will be available programatically. We chose to make it an annotation because as Erlangers, you already know and love @doc AND we didn't want you to worry about multiline strings and an array of strings as a member of a proplist. This just seemed cleaner.

@see: If you already wrote an @doc for a setting, but you were describing how multiple settings work together, you can just specify an @see to that mappings name. It's just a pointer to additional documentation.

%% @doc 'a.x' and 'a.y' set the start coordinate
{mapping, "a.x", "app.a.x", []}.

%% @see a.x
{mapping, "a.y", "app.a.y", []}.

Note: Cuttlefish has a bit of fun with these when generating your default .conf file. It will include the documentation inline if you have a @doc. If you also have a @see, it will include references to those other settings. If it only has an @see, it will include the @doc of the mapping you specified in the @see.

Elements of the mapping tuple

Aside from documentation, there is plenty going on with mappings, but the basic form is as follows:

%% {mapping, string(), string(), proplist()}
Mapping = {mapping, ConfKey, ErlangConfMapping, Attributes}.

element(1, Mapping) = mapping
element(2, Mapping) = ConfKey - the string key that you want this setting to have in the .conf file
element(3, Mapping) = ErlangConfMapping - the nested location of the thing in the app.config that this field represents
element(4, Mapping) = Attributes - other helpful things we'll go into right... about... now!

Attributes is a proplist, and let's assume you know how those work. Here are the keys in that proplist that we work with:

default - This is the default value for the setting. When a .conf file is generated, a line will be added to set the setting to this value. Additionally, if no value is specified in the .conf file, the value specified in the default attribute will be used when generating the app.config file.
new_conf_value - If this is defined, then when you generate a .conf file, this value will be used for the default setting. This causes the default attribute to be overridden when generating a .conf file, but has no effect on what default value is used when generating app.config.
commented - If this is defined, then when you generate a .conf file the documentation for this setting appears, along with the setting, but the setting is commented out and that comment includes this value.
datatype - This is the datatype for the field. For a list of those, check Datatypes.
hidden - If this atom is present or set to true, this value will be in the generated app.config, but not the generated .conf file. It still can be overridden in the .conf file, you just have to know about it. It's a way of adding "undocumented knobs".
include_default - If there is a substitutable value in the ConfKey, in the generated .conf file, this value is substituted. (don't worry if that last one didn't make so much sense now, I'll explain more below)
validators - the list of names for validators for this mapping.
merge - is the only thing included just as an atom. It specifies that this is supplemental to an existing mapping, not a replacement mapping. The default behavior is to replace an existing mapping with this newer one

The best way to get it, is to take a look at some examples. Let's start with Riak's ring_size.

%% example of super basic mapping
%% @doc Default ring creation size.  Make sure it is a power of 2,
%% e.g. 16, 32, 64, 128, 256, 512 etc
{mapping, "ring_size", "riak_core.ring_creation_size", [
  {datatype, integer},
  {commented, 64}
]}.

First of all, comments before the @doc annotation are ignored, so feel free to put Schema specific comments in here as you see fit. Everything after the @doc in the comments, is part of the documentation. Cuttlefish will treat this documentation as:

[
  "Default ring creation size.  Make sure it is a power of 2,",
  "e.g. 16, 32, 64, 128, 256, 512 etc"
].

Then, we can also see from element(1) that this is a mapping. element(2) says that it's represented by "ring_size" in the riak.conf file. element(3) says that it's "riak_core.ring_creation_size" in the app.config. We also know from the attibutes that it is an integer, and that it will appear in the generated riak.conf file with a value of 64. It just so happens that the default is also 64, but that's specified in riak_core's app.src.

Let's talk about element(3) here for a minute. What that means is that there's an app.config out there that looks like this:

[
  {riak_core, [
    {ring_creation_size, X}
  ]}
].

and that we're concerned with X.

An important note on Cuttlefish's Philosophy

Here at cuttlefish, one thing we're really disturbed by is this idea of a "magic number". A magic number is a setting that we're not actually sure where the default value comes from. In the above example it comes from riak_core's app.src file, but they can come from anywhere, even the dreaded application:get_env/3.

We encourage every mapping in a cuttlefish schema to set an explicit default. In that case your application has a single location for default values, and it generates a complete app.config.

Now, if life were as simple as 1:1 mappings like this, we'd be done. But it's not, and so we need to introduce translations.

Lost in Translations

Actually, they're pretty easy.

A translation looks like this:

%% {translation, string(), fun((proplist()) -> term())}
Translation = {translation, ErlangConfMapping, TranslationFunction}.

Let's break it down:

element(1, Translation) = translation
element(2, Translation) = ErlangConfMapping this is the same as the corresponding ErlangConfMapping in the mapping above. This is how we tie back to a mapping
element(2, Translation) = TranslationFunction this is a fun() that takes in a proplist representing the .conf file and returns an erlang term. This erlang term will be the value of the ErlangConfMapping.

Ok, that does sound more confusing than it should. Let's take a look at one, you'll like it better in practice.

%% @doc How Riak will repair out-of-sync keys. Some features require
%% this to be set to 'active', including search.
%%
%% * active: out-of-sync keys will be repaired in the background
%% * passive: out-of-sync keys are only repaired on read
%% * active-debug: like active, but outputs verbose debugging
%%   information
{mapping, "anti_entropy", "riak_kv.anti_entropy", [
  {datatype, {enum, [active, passive, 'active-debug']}},
  {default, active}
]}.

{translation,
 "riak_kv.anti_entropy",
 fun(Conf) ->
    Setting = cuttlefish:conf_get("anti_entropy", Conf),
    case Setting of
      active -> {on, []};
      'active-debug' -> {on, [debug]};
      passive -> {off, []};
      _Default -> {on, []}
    end
  end
}.

See what's happening? First of all, you need a mapping. If you don't have one, don't bother writing a translation for it. It will not get run. It will also not get run if you have a mapping for it, but you don't have a value for it. That value can be a default in the mapping or set in your .conf file. The mapping we defined for "anti_entropy" says that it's an enum with values "active", "passive", and "active-debug". The configuration in the app.config is more complicated. Basically, it works like this:

active - {on, []}
passive - {off, []}
active-debug - {on, [debug]}

It's a relatively simple translation, but we want to spare non-Erlangers from this very Erlangy syntax. So, we give them the values "active", "passive", and "active-debug" and the translation "translates" (not just a clever nickname!) them into the erlang value we expect.

Translations and the Conf proplist

You may have noticed that the translation fun takes an argument Conf. Conf is a cool proplist and needs to get its "props". Here are some fun facts about Conf:

By the time Conf gets to this function, a lot has happened since it was a .conf file. First of all, if you omitted a variable in your .conf file, but it had a default, that default value is in Conf.

Conf is a proplist, but its keys are no longer in the form "a.b.c". For "easier" erlang processing (pattern matching, etc), the keys come in the form ["a", "b", "c"]. The values are also different. As you might imagine, in the .conf file, they're all just strings. If you specified a datatype in the mapping, it will be transformed into that erlang type by the time it gets here.

There are convenience functions in the cuttlefish module for accessing these. They allow you to pass variables in with either "a.b.c" or ["a", "b", "c"] notation. For more on that, see the cuttlefish_variable module.

See the Cuttlefish Developer's API page for more.

The Curious Case of Lager Config

There are other cases when multiple values turn into a single app.config complex data structure. Take lager as an example.

%% complex lager example
%% @doc location of the console log
{mapping, "log.console.file", "lager.handlers", [
  {default, "./log/console.log"}
]}.

%% *gasp* notice the same @mapping!
%% @doc location of the error log
{mapping, "log.error.file", "lager.handlers", [
  {default, "./log/error.log"}
]}.

%% *gasp* notice the same @mapping!
%% @doc turn on syslog
{mapping, "log.syslog", "lager.handlers", [
  {default, off},
  {datatype, enum},
  {enum, [on, off]}
]}.

{ translation,
  "lager.handlers",
  fun(Conf) ->
    SyslogHandler = case cuttlefish:conf_get("log.syslog", Conf) of
      on ->  {lager_syslog_backend, ["riak", daemon, info]};
      _ -> undefined
    end,
    ErrorHandler = case cuttlefish:conf_get("log.error.file", Conf) of
      undefined -> undefined;
      ErrorFilename -> {lager_file_backend, [{file, ErrorFilename}, {level, error}]}
    end,
        ConsoleHandler = case cuttlefish:conf_get("log.console.file", Conf) of
          undefined -> undefined;
          ConsoleFilename -> {lager_file_backend, [{file, ConsoleFilename}, {level, info}]}
        end,
        lists:filter(fun(X) -> X =/= undefined end, [SyslogHandler, ErrorHandler, ConsoleHandler])
  end
}.

We define three mappings here, that have different values in the riak.conf file, but represent a complex list of lager handlers in the app.config. The solution is to have them all map to the same ErlangConfMapping, which references lager.handlers. When we create a translation for that, we're basically saying that "The return value of this function will be the value of {lager, [{handers, X}]}". that was a weird way of saying it, but the generated app.config looks like this:

 {lager,
     [
      {handlers,
          [{lager_syslog_backend,["riak",daemon,info]},
           {lager_file_backend,[{file,"/var/log/error.log"},{level,error}]},
           {lager_file_backend,[{file,"/var/log/console.log"},{level,info}]}]},
          ]},

Lists and Proplists and $names, oh my!

Sometimes you'll find yourself needing to map elements of a list or proplist. Consider the way we configure HTTP listeners for Riak.

 {riak_core,
    [
      {http,
        [
          {"127.0.0.1",8098},
          {"10.0.0.1",80}
        ]
      }
    ]
  }

We got really aggressive with the line breaks here, to illustrate that riak_core.http is a list of {IP, Port} tuples. Now, say for some reason, you wanted 10 of these listeners. We're not here to judge you, we're here to help. What we didn't want to do was introduce some kind of list data structure on the right hand side of our .conf file. Instead we took a "list element per line" approach. We wanted to give you a syntax that was something like this:

listener.http.internal = 127.0.0.1:8098
listener.http.external = 10.0.0.1:80

But wait, what's the deal with this "internal"/"external" business? Well, the mapping is defined with a wildcard. Think of it like a match group in a regex.

%% HTTP Listeners
%% @doc listener.http.<name> is an IP address and TCP port that the Riak
%% HTTP interface will bind.
{mapping, "listener.http.$name", "riak_api.http", [
  {default, {"127.0.0.1", 8098}},
  {datatype, ip},
  {include_default, "internal"}
]}.

{translation,
 "riak_api.http",
  fun(Conf) ->
      HTTP = cuttlefish_variable:filter_by_prefix("listener.http", Conf),
      [ IP || {_, IP} <- HTTP]
  end
}.

See the $name? it can be anything! Then the translation is "smart" enough to parse all the listner.http.* config keys and create the list of {IP, Port}s for the "riak_core.http" section. (TODO: in the future, we'll add the ability to refer back to $name as a variable, but for now, we didn't need to because in this case, name was a throwaway).

Also, notice the {datatype, ip}, that is smart enough to turn "IP:Port" into {IP, Port}. Don't worry, it works for IPv6 too. More on Datatypes.

More about include_default

This is the perfect place to talk about 'include_default'. If there's a wildcard in the ConfKey, we don't want to include that wildcard in the default generated .conf file, so we need an example. The value from include_default provides that sample. So, the generated .conf looks like this:

## listener.http.<name> is an IP address and TCP port that the Riak
## HTTP interface will bind.
listener.http.internal = 127.0.0.1:8098

So, $names don't matter?

What's in a $name? That which we call internal by any other name would still bind internally; so HTTP would, were it not HTTP called.

Not so fast, Billy! Sometimes it does matter. Let's look at the userlist in Riak Control:

%% @doc If auth is set to 'userlist' then this is the
%% list of usernames and passwords for access to the
%% admin panel.
{mapping, "riak_control.user.$username.password", "riak_control.userlist", [
  {default, "pass"},
  {include_default, "user"}
]}.

{translation,
"riak_control.userlist",
fun(Conf) ->
  UserList = lists:filter(
    fun({K, _V}) ->
       cuttlefish_variable:is_fuzzy_match(K, string:tokens("riak_control.user.$username.password", "."))
    end,
    Conf),
  Users = [ {Username, Password} || {[_, _, Username, _], Password} <- UserList ],
  case Users of
      [] ->
          throw(unset);
      _ -> Users
  end

end}.

Right now, we're leaving it up to the translation fun to tokenize the string and extract the username, but it shouldn't have to. We should provide helpers for this, and we will.

Validators

You can write validator functions to perform advanced validation for mappings. Let's take the ring size in riak as an example:

{validator, "ring_size", "not a power of 2 greater than 1",
 fun(Size) ->
  Size > 1 andalso (Size band (Size-1) =:= 0)
 end}.

What we're saying here is that ring_size has to be an integer that's a power of 2 and greater than 1.

So how does this get triggered?

%% @doc Number of partitions in the cluster (only valid when first
%% creating the cluster). Must be a power of 2, minimum 8 and maximum
%% 1024.
{mapping, "ring_size", "riak_core.ring_creation_size", [
  {datatype, integer},
  {default, 64},
  {validators, ["ring_size^2", "ring_size_max", "ring_size_min"]},
  {commented, 64}
]}.

%% ring_size validators
{validator, "ring_size_max", "2048 and larger are supported, but considered advanced config",
 fun(Size) ->
  Size =< 1024
 end}.

{validator, "ring_size^2", "not a power of 2",
 fun(Size) ->
  (Size band (Size-1) =:= 0)
 end}.

{validator, "ring_size_min", "must be at least 8",
 fun(Size) ->
  Size >= 8
 end}.

In the end, we went with three different validators. The validators property of a mapping specifies which validators get run for a field.They can be reused on multiple mappings, and you can add as many as you want.

I like Eunit!

Me too. It'd be pretty crazy to just write a bunch of Erlang and expect it to just work in your application. So we decided to give you a module of unit test helper functions: cuttlefish_unit. Please see the Unit Testing A Schema page.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly