Config - Smart and flexible system configuration

Config is a very small library used to handle configuration of a server; it works quite well with a system defined in terms of Stuart Sierra’s component library.

This posting provides a lot of detail on the requirements and capabilities of config.

API Documentation

Overview

Config reads a series of files, primarily from the classpath. The files contain contain configuration data in EDN format.

The files are read in a specific order, based on a set of profiles. The name of the file to read is based on the profile and the variant (described shortly).

The contents of all the configuration files are converted to Clojure maps and are deep-merged together.

The intent of profiles is that there is an approximate mapping between components and profiles: generally, each component will have exactly one profile.

Each component may, optionally, define a spec for its configuration data.

But what about the 12 Factor App's guideline to store configuration only as environment variables? This is embraced by config, because the files may contain environment variable references that are expanded at runtime.

At Aviso, we use these features in a number of ways. For example, for quick testing we combine a number of microservices (each of which has its own configuration profile and schema) together into a single system. Meanwhile, in production (on AWS) we can build a smaller system with a single microservice. We can also provide an additional configuration file that enables configuration overrides based on environment variables set by CloudFormation.

Implementing Components

You might define a web service as:

(require '[clojure.spec :as s]
         '[io.aviso.config :as config]
         '[com.example.jetty :as jetty]
         '[com.stuartsierra.component :as component])

(defrecord WebService [port request-handler jetty-instance]

  config/Configurable

  (configure [this configuration]
    (merge this configuration))

  component/Lifecycle

  (start [this]
    (assoc :jetty-instance (jetty/run-jetty
                              request-handler
                              {:port port
                               :join? false})))

  (stop [this]
    (.stop jetty-instance)
    (assoc this :jetty-instance nil))

(s/def ::port (s/and int? pos?)
(s/def ::config (s/keys :req-un [::port])

(defn web-service
  []
  (-> (map->WebService {})
      (component/using [:request-handler])
      (config/with-config-spec :web-service ::config)))

This is a standard component, with the start and stop lifecycle methods, a dependency on another component (:request-handler), and a local field for the instance of Jetty managed by the component.

In addition, the component is configurable: it implements the Configurable protocol, and receives its specific configuration as a map. The configuration passed to the component conforms to the ::config spec; it will have a :port key.

The configure method is invoked before the start method.

Providing Configuration Files

Without configuration files, your application will not start up; you will see errors about invalid specs, because there is (in this example) no :web-service top-level key, and no :port key below that.

Configuration files are located on the class path, within a conf package; this means inside the resources/conf folder in a typical project.

For the :web-service component, you would provide a default configuration file, web-service.edn:

{:web-service {:port 8080}}

Starting the System

And finally, build and start a system from all this:

(let [system (component/system-map
               :web-service (web-service)
               :request-handler (request-handler))]
    (-> system
        (config/configure-using nil)
        component/start-system))

The configure-using function reads the configuration files and assembles the configuration map, then applies the the configuration to each component.

The second parameter to configure-using is map of options.

configure-using generates default profiles from components in the system. Any component that declared a configuration key using with-config-spec will be included.

Here, the default list of profiles is just :web-service.

The :web-service keyword is being used in three ways here:

• As the component key in the system map

• As the name of the profile for the component, identifying the configuration file(s) for the component

• As the key within the system configuration containing the component’s specific configuration

Unless you have a compelling reason otherwise, you should always follow this pattern; the profile name should match the configuration key, which should match the component key in the system map.

The default profiles are in dependency order. If :request-handler has a configuration key, then it will be ordered ahead of :web-service, because the :web-service component depends on the :request-handler component.

For each component that defines a configuration spec, configure-using will:

• Extract the component’s configuration

• Conform the configuration

• Throw an exception if the configuration contains invalid data

• Either invoke the configure method, or associate a :configuration key, providing the conformed configuration

Configuration Overrides

But what if you want to override part of the :web-service configuration …​ for example, to specify a different port? This is very common …​ your local development configuration is going to vary considerably from your deployed production configuration.

This can be accomplished in a number of ways.

Explicit Overrides

First off all, it is possible to provide an explicit map of overrides when constructing the configuration map:

   (config/configure-using {:overrides {:web-service {:port 9999}}})

However, that option is generally intended for special cases, such as overrides during testing.

Most other approaches involve controlling which files are loaded to form the system configuration.

Explicit Profiles

So if you wish to have some overrides, you could provide a configuration file named overrides.edn and ensure that is loaded after the :web-service profile:

   (config/configure-using {:profiles [:overrides]})

Implicit profiles, via with-config-spec are loaded first, then explicit profiles in the options. Order can be important here, and later-loaded profiles will override earlier profiles if there are conflicts.

Variants

Another option is to support an additional variant to customize the configuration.

For each profile, config searches for any variant.

In this case, the file name would be web-service-production.edn. web-service comes from the profile and production from the variant.

   (config/configure-using {:variants [:production]})

The nil variant (web-service.edn) is always loaded first to provide the defaults, the provided variants (when they exist) overlay the nil variant.

In this example, the normal configuration is safe; it’s for local testing. Only when deploying to production does the :production variant get added in.

Additional Files

You could also explicitly load one or more configuration files stored on the file system (rather than as classpath resources):

   (config/configure-using {:additional-files ["overrides/production.edn"]})

This is another possible way to provide overrides that only apply in production; the difference being that this file is on the file system, not packaged inside the application JARs.

Runtime Properties

Often, especially in production, you don’t know all of the configuration until your application is actually started. For example, in a cloud provider, important IP addresses and port numbers are often assigned dynamically. This information is provided to the processes via environment variables.

Although this information could be extracted by startup code, and provided to the configure-using function using the :overrides configuration, that is both rigid and clumsy.

Instead, it is possible to reference these dynamic properties inside the configuration files using the special reader macros supplied by config.

Properties are:

• Shell environment variables.

• JVM System properties.

• The :properties option, passed to configure-using.

The following reader macros are available:

#config/prop

Accesses dynamic properties. The value is either a single string key, or a vector of string key followed by a default value.

#config/join

Joins a number of values together to form a single string; this is used when an building a single string from a mix of properties and static text.

#config/long

Converts a string to a long value. Typically used with #config/prop.

#config/keyword

Converts a string to a keyword value. Typically used with #config/prop.

Here’s an example showing all the variants:

{:connection-pool
  {:user-name #config/prop ["DB_USER" "accountsuser"]
   :user-pw #config/prop "DB_PW"
   :url  #config/join ["jdbc:postgresql://"
                       #config/prop "DB_HOST"
                       ":"
                       #config/prop "DB_PORT"
                       "/accounts"]}
 :web-server
 {:port #config/long #config/prop "WEB_PORT"}}

In this example, the DB_USER, DB_PW, DB_HOST, and DB_PORT, and WEB_PORT environment variables all play a role (though DB_USER is optional, since it has a default value).

In the final configuration, the key [:connection-pool :url] is a single string, and the key [:web-server :port] is a long (not a string).

License

Config is available under the terms of the Apache Software License 2.0.