Configuration library for Clojure that favors explicitness
Clojure
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
example-boot
example-lein
src/omniconf
test/omniconf
.gitignore
LICENSE
README.org
boot.properties
build.boot
circle.yml

README.org

Omniconf

I Changelog I https://circleci.com/gh/grammarly/omniconf/tree/master.png I 6.1k downloads I

Command-line arguments. Environment variables. Configuration files. Java properties. Almost every program requires some configuration which is usually spread around multiple sources. Keeping track of all of them, mapping ones to others, making sure they are present and correct, passing them around is a laborious and thankless task.

Configuration libraries, which there are plenty of, promise to solve the configuration problem, and they do. However, they usually provide only the mapping of configuration sources to Clojure data, leaving out the verification part. Omniconf’s value proposition, among other features, is the requirement to declare the expected configuration upfront, and the ability to validate the configuration early and display helpful messages if the application is misconfigured.

In terms of configuration sources, Omniconf supports:

Rationale

Omniconf is developed with the following principles in mind:

  1. Explicit over implicit. Most configuration libraries allow to grab a configuration value (e.g. from ENV) at any point of time from any place in the code. This makes it impossible to list all configuration options that the program uses without reading the entire source. Omniconf requires you to declare all possible configuration upfront, at the beginning of the program execution.
  2. All configuration sources must be unified. It shouldn’t matter where the option is set from — environment variables, CLI arguments, or config files. It is uniformly initialized, verified, and accessed as regular Clojure data.
  3. Maximum verification. We don’t want to see FileNotFound stacktraces in the middle of your program run. The whole configuration should be checked early and automatically before the rest of the code is executed. If there are any problems with it, a helpful message should be presented to the user.

Installation

Add this line to your list of dependencies:

https://clojars.org/com.grammarly/omniconf/latest-version.svg

Usage

  1. You start by defining a set of supported options. cfg/define takes a map of options to their different parameters. The following small example shows the syntax:
    (require '[omniconf.core :as cfg])
    (cfg/define
      {:hostname {:description "where service is deployed"
                  :type :string
                  :required true}
       :port     {:description "HTTP port"
                  :type :number
                  :default 8080}})
        

    The full list of supported parameters is described here.

  2. Populate the configuration from available sources:
    (cfg/populate-from-cmd args) ;; args is a command-line arguments list
    (when-let [conf (cfg/get :conf)]
      (cfg/populate-from-file conf))
    (cfg/populate-from-properties)
    (cfg/populate-from-env)
        

    The order in which to tap the sources is up to you. Perhaps you want to make environment variables overwrite command-line args, or give the highest priority to the config file. In the above example we get the path to the configuration file as --conf CMD argument. For more information, see this.

  3. Call verify. It marks the boundary in your system, after which the whole configuration is guaranteed to be complete and correct.
    (cfg/verify)
        

    If there is something wrong with the configuration, verify will throw a proper exception. If called not in the REPL environment, the exception will be stripped of its stacktrace, so that you only see the exact error.

    If everything is alright, verify will pretty-print the whole configuration map into the standard output. It is convenient because it gives you one final chance to look at your config values and make sure they are good. :silent true can be passed to verify to prevent it from printing the map.

  4. Use get to extract arbitrary value from the configuration.
    (cfg/get :hostname)
        

    For nested values you can pass an address of the value, either as a vector, or like varargs:

    (cfg/get :database :ip)
    (cfg/get [:database :ip])
        

    set allows you to change a value. It is definitely not recommended to be used in production code, but may be convenient during development:

    (cfg/set :database :port 3306)
    (cfg/set [:database :port] 3306)
        

Examples

Sample programs that use Omniconf: example-lein and example-boot. There is not much difference in using Omniconf with these build tools, but Boot requires a little hack to achieve parity with Leiningen.

Configuration scheme syntax

Configuration scheme is a map of option names to maps of their parameters. Option name is a keyword that denotes how the option is retrieved inside the program, and how it maps to configuration sources. Naming rules are the following:

For command-line arguments:

:some-option    =>    --some-option

For environment variables:

:some-option    =>    SOME_OPTION

For Java properties:

:some-option    =>    some-option   (java -Dsome-option=... if set from command line)

Each option can have the following parameters:

  • :description — string that describes this option. This description will be used to generate a help message for the program.
  • :type — currently the following types are supported: :string, :keyword, :number, :boolean, :edn, :file, :directory. Setting a type automatically defines how to parse a value for this option from a string, and also verifies that the resulting value has the correct Clojure type.

    Boolean types have some special treatment. When setting them from the command line, one can omit the value completely.

    (cfg/define {:foo {:type :boolean}, :bar {:type :boolean}})
    ...
    $ my-app --foo --bar    # Confmap is {:foo true, :baz true}
        

    A string parser for booleans treats strings “0” and “false” as false, anything else as true.

  • :parser — a single-arg function that converts a string value (given in command-line option or environment variable) into a Clojure value. This option can be used instead of :type if you need a custom option type.
  • :default — the option will be initialized with this value. The default value must be specified as a Clojure datatype, not as a string yet to be parsed.
  • :required — if true, the value for this option must be provided, otherwise verify will fail. The value of this parameter can also be a nullary function: if the function returns true then the option value must be provided. It is convenient if the necessity of an option depends on the values of some other options. Example:
    (cfg/define {:storage   {:one-of [:file :s3]}
                 :s3-bucket {:required #(= (cfg/get :storage) :s3)}})
        
  • :one-of — a sequence of values that an option is allowed to take. If the value isn’t present in the :one-of list, verify will fail. :one-of automatically implies :required true unless you add nil as a permitted value.
  • :verifier — a function of [option-name value] that should throw an exception if the value is not correct. Verifier is only executed if the value is not nil, so it doesn’t imply :required true. Predefined verifiers:
    • cfg/verify-file-exists
    • cfg/verify-directory-non-empty — checks if the value is a directory, and if it is non-empty.
  • :delayed-transform — a function of option value that will be called not immediately, but the first time when the option is accessed in the code. Transform will be applied only once, and after that the option will store the transformed value. Usefulness of this feature is yet in question. You can mimic it by using a custom parser that wraps the value in a delay, the only difference that you will also have to dereference it manually every time.
  • :nested — a map that has the same structure as the top-level configuration scheme. Nested options have the same rights as top-level ones: they can have parsers, verifiers, defaults, etc. Example:
    (cfg/define
      {:statsd {:nested {:host {:type :string
                                :required true
                                :description "IP address of the StatsD server"}
                         :port {:type :number
                                :default 8125}}}})
        

    CLI and ENV arguments have special transformation rules for nested options — dot as a separator for CLI arguments and Java properties, and double underscore for ENV.

    [:statsd :host]    =>    --statsd.host   (cmdline args)
    [:statsd :host]    =>    -Dstatsd.host   (properties)
    [:statsd :host]    =>    STATSD__HOST    (env variables)
        

    In the program you can use cfg/get to fetch a concrete value, or a whole map at any level:

    (cfg/get :statsd :port) ;=> 8125
    (cfg/get :statsd) ;=> {:host "127.0.0.1", :port 8125}
        
  • :secret — if true, the value of this option won’t be printed out by cfg/verify. You will see <SECRET> instead. Useful for passwords, API keys and such.

Providing configuration as files

Omniconf can use EDN files as a configuration source. A file must contain a map of options to their values, which will be merged into the config when populate-from-file is called. The values should already have the format the option requires (number, keyword); but you can also use strings so that parser will be called on them.

You can hardcode the name of the file where to look for configuration (e.g. config.edn in the current directory). It is somewhat trickier to tell the name of the file dynamically. One of the solutions is to expect the configuration file to be provided in one of the command-line arguments. So you have to populate-from-cmd first, and then to populate from config file if it has been provided. However, this way the configuration file will have the priority over CLI arguments which is not always desirable. As a workaround, you can call populate-from-cmd again, but only if your CLI args are idempotent (i.e. they don’t contain ^:concat, see below).

Fetching configuration from AWS Systems Manager (SSM)

Since version 0.3, Omniconf supports Amazon SSM, particularly its Parameter Store, as a configuration source. SSM works well as a storage for secrets — passwords, tokens, and other sensitive things that you don’t want to check into the source control.

To use SSM backend, you’ll need to add an extra dependency:

https://clojars.org/com.grammarly/omniconf.ssm/latest-version.svg

The function omniconf.core/populate-from-ssm will be available now. It takes path as an argument which will be treated as root path to nested SSM parameters. For example:

(cfg/define
  {:db {:nested {:password {:type :string
                            :secret true}}}})

(cfg/populate-from-ssm "/prod/myapp/")

This will fetch /prod/myapp/db/password parameter from SSM and save it as [:db :password] in Omniconf.

You can also specify explicit mapping between SSM and Omniconf like this:

(cfg/define
  {:db {:nested {:password {:type :string
                            :secret true}}}
   :github-token {:type :string
                  :secret true
                  :ssm-name "/myteam/github/oauth-token"}})

(cfg/populate-from-ssm "/prod/myapp/")

Parameters with an absolute :ssm-name parameter will ignore the path argument and will fetch the value directly by name. In case you still want to use path for some keys but the layout in SSM differs from one in Omniconf, you can use ./ as a prefix to signify that it is relative to the path:

(cfg/define
  {:db {:nested {:password {:type :string
                            :secret true
                            :ssm-name "./db-pass"}}}})

(cfg/populate-from-ssm "/prod/myapp/")

This will set [:db :password] parameter from /prod/myapp/db-pass.

Tips, tricks, and FAQ

Are there any drawbacks? What’s the catch?

There are a few. First of all, Omniconf is much more complex and intertwined than, say, Environ. This might put off some developers, although we suspect they are re-implementing half of Omniconf functionality on top of Environ anyway (like we did before).

Omniconf is not suited for dynamic configuration. If you need options to be changed during runtime, values coming from some external dynamic sources, you are better off using a proper solution for that, e.g. Zookeeper together with some wrapper library.

Omniconf configuration map is a global mutable singleton. It is OK if you use Omniconf like we suggest to — populate the values before any application code is executed, and then never change them again — but there might be usecases where this approach does not fit.

Omniconf is an application-level tool. You most likely don’t want to make your library depend on it, forcing the library users to configure through Omniconf too.

Why are there no convenient Leiningen plugins/Boot tasks for Omniconf?

In the end we distribute and deploy our applications as uberjars. As a standalone JAR our program doesn’t have access to Leiningen or Boot. Hence, it is better not to offload anything to plugins to avoid spawning differences between development and production time.

CLI help command

:help option gets a special treatment in Omniconf. It can have :help-name and :help-description parameters that will be used when printing the help message. If populate-from-cmd encounters --help on the arguments list, it prints the help message and quits.

Useful functions and macros

with-options works as let for configuration values, i.e. it takes a binding list of symbols that should have the same names as options’ keyword names. Only top-level options are supported, destructuring of nested values is not possible right now.

(cfg/with-options [username password]
  ;; Binds (cfg/get :username) to username, and (cfg/get :password) to password.
  ...)

Verify configuration during builds

It may be useful to run cfg/verify as a part of the build step. If you provide all the options during that step as you do when running the program, then you will be able catch the misconfiguration errors before the app is deployed.

To do this properly you have to create another entry point into your program that only runs the config definition, population and verification. Look into example projects for inspiration.

Special operations for EDN options

Sometimes you don’t want to completely overwrite an EDN value, but append to it. For this case two special operations, — ^:concat and ^:merge — can be attached to a map or a list when setting them from any source. Example:

(cfg/define {:emails {:type :edn
                      :default ["admin1@corp.org" "admin2@corp.org"]}
             :roles  {:type :edn
                      :default {"admin1@corp.org" :admin
                                "admin2@corp.org" :admin}}})
...
$ my-app --emails '^:concat ["user1@corp.org"]' --roles '^:merge {"user1@corp.org" :user}'

Custom logging for Omniconf

By default, Omniconf prints errors and final configuration map to standard output. But if you have many servers, it may not be very convenient to connect to each to see if all of them are correctly configured. Perhaps you have a Logstash forwarder running on the instance, or some other centralized logging solution. So, you can call cfg/set-logging-fn to make Omniconf use it instead of println. For Timbre 4.3.1 it will be something like this:

(require '[taoensso.timbre :as log])
(cfg/set-logging-fn (fn [& args]
                      (log/-log! log/*config* :info "omniconf.core"
                                 nil nil :p nil (delay (vec args)) nil)))

Note that this will only work if you are able to initialize logging without any data from Omniconf. This is a chicken-and-egg problem that doesn’t have a proper solution, as it is very case-specific.

License

© Copyright 2016-2018 Grammarly, Inc.

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.