Skip to content
Hiera in golang - Elegant, powerful hierarchical key-value lookups
Go Dockerfile
Branch: master
Clone or download
ahpook Merge pull request #44 from thallgren/sdk-020
Rename directory examples_test to example and update to hierasdk 0.2.0
Latest commit db4dba0 Oct 23, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github/workflows
.idea Add ability to dynamically load and call external RESTful lookup plugin. Oct 9, 2019
cli Changes in response to code review Oct 16, 2019
examples Rename directory examples_test to example and update to hierasdk 0.2.0 Oct 23, 2019
explain Enable rendering of explanation output Jul 16, 2019
hiera Switch from .travis to github actions and add more linting Oct 15, 2019
hieraapi Changes in response to code review Oct 16, 2019
hieraserver/rest Changes in response to code review Oct 16, 2019
internal Changes in response to code review Oct 16, 2019
lookup
provider Changes in response to code review Oct 16, 2019
schema Add ability to dynamically load and call external RESTful lookup plugin. Oct 9, 2019
yaml2json Add json-schema for the hiera.yaml configuration Jul 18, 2019
.dockerignore Add ability to dynamically load and call external RESTful lookup plugin. Oct 9, 2019
.gitignore Add ability to dynamically load and call external RESTful lookup plugin. Oct 9, 2019
.golangci.yml Switch from .travis to github actions and add more linting Oct 15, 2019
Dockerfile No plugins currently exist. Oct 17, 2019
LICENSE Create LICENSE Dec 8, 2018
README.md Rename 'build' workflow to 'tests' Oct 17, 2019
go.mod Rename directory examples_test to example and update to hierasdk 0.2.0 Oct 23, 2019
go.sum Rename directory examples_test to example and update to hierasdk 0.2.0 Oct 23, 2019

README.md

Hiera lookup framework

Introduction

Hiera is a flexible, powerful tool for resolving values for variable lookups, which was first popularised by its use in Puppet.

This module is a "clean-room" Go implementation of the Hiera framework, suitable for use as a library from other tools.

Details

Hiera uses the concept of "managing by exception": you design a hierarchy of data sources, with the most specific source at the top and least-specific defaults at the bottom. Hiera searches for keys starting at the top, allowing more-specific sources to override defaults. Sources are usually YAML files stored on the filesystem, and layers usually use variable interpolation to find the right file, allowing the context of the lookup to pick the right file.

How to run it

Standalone execution

Install the module

To install the module under $GOPATH/src:

go get github.com/lyraproj/hiera

Install the lookup binary

Install the lookup binary under $GOPATH/bin:

go install github.com/lyraproj/hiera/lookup

Run the binary

lookup --help

Containerized execution

Download the container

You can pull the latest containerized version from docker hub:

docker pull lyraproj/hiera:latest

The docker repository with previous tags is viewable at https://hub.docker.com/r/lyraproj/hiera .

Run the container

The docker image accepts environment variables to override default behaviour:

  • port - which port to listen on inside the container (default: 8080)
  • loglevel - how much logging to do (default: error, possible values: error, warn, info, debug)
  • config - path to a hiera.yaml configuration (default: /hiera/hiera.yaml)

Make sure to pass the port on your host through to the container. A directory with a hiera configuration and data files (see below) should be mounted under /hiera in the image using a bind mount:

docker run -p 8080:8080 --mount type=bind,src=$HOME/hiera,dst=/hiera lyraproj/hiera:latest

Query the container

The web service in the container responds to the /lookup endpoint with an additional path element of which key to look up. Nested keys can be looked up using dot-separation notation. Given a yaml map without any overrides like:

aws:
  tags:
    Name:       lyra-sample
    created_by: lyra
    department: engineering
    project:    incubator
    lifetime:   1h

You can get back the entire map or specific parts of it:

$ curl http://localhost:8080/lookup/aws
{"tags":{"Name":"lyra-sample","created_by":"lyra","department":"engineering","lifetime":"1h","project":"incubator"}}
$ curl http://localhost:8080/lookup/aws.tags
{"Name":"lyra-sample","created_by":"lyra","department":"engineering","lifetime":"1h","project":"incubator"}
$ curl http://localhost:8080/lookup/aws.tags.department
"engineering"

Pass values for interpolation

If your hierarchy config contains variable interpolation, you can provide context for the lookup using the var query parameter. Repeated var parameters will create an array of available parameters. The values should be colon-separated variable-value pairs:

curl 'http://localhost:8080/lookup/aws.tags?var=environment:production&var=hostname:specialhost'

TODO: Nested variable lookups such like os.family are not yet working.

Hiera configuration and directory structure

Much of hiera's power lies in its ability to interpolate variables in the hierarchy's configuration. A lookup provides values, and hiera maps the interpolated values onto the filesystem (or other back-end data structure). A common example uses two levels of override: one for specific hosts, a higher layer for environment-wide settings, and finally a fall-through default. A functional hiera.yaml which implements this policy looks like:

---
version: 5
defaults:
datadir: hiera
data_hash: yaml_data

hierarchy:
- name: "Host-specific overrides"
    path: "hosts/%{hostname}.yaml"
- name: "Environmental overrides"
    path: "environments/%{environment}.yaml"
- name: "Fall through defaults"
    path: "defaults.yaml"

This maps to a directory structure based in the hiera subdirectory (due to the datadir top level key) containing yaml files like:

hiera
├── defaults.yaml
├── environments
│   └── production.yaml
└── hosts
    └── specialhost.yaml

Extending Hiera

When Hiera performs a lookup it uses a lookup function. Unless the function embedded in the hiera binary, it will make an attempt to load a RESTful Plugin that provides the needed function. Such plugins are enabled by using the API provided by the hierasdk library.

How Hiera finds its plugins

The resolution of a plugin can be controlled using a "plugindir" key in the Hiera configuration file. As with "datadir", the "plugindir" can be specified both in the defaults hierarchy or explicitly in a specific hierarchy. Unless specified, the "plugindir" is assumed to be the directory "plugin" present in the same directory as the configuration file.

In addition to "plugindir", a hierarchy may also specify a "pluginfile". Unless specified, the "pluginfile" is assumed to be equal to the name of the lookup function (with the extension ".exe" in case of Windows).

Containerized extension

In order to include an extension in a Hiera Docker image you need to:

  1. Copy the source (or clone the git repository) of the desired extensions into the hiera plugin directory (don't worry, this directory will be ignored by git when doing commits to hiera).
  2. For each extension, add a line like the following line to the Hiera Dockerfile below the comment # Add plugin builds here:
    RUN (cd plugin/hiera_terraform && go build -o ../terraform_backend)
    
  3. Run the docker build.

Useful extensions

Implementation status

  • lookup CLI
  • lookup function
  • lookup context
  • dotted keys (dig functionality)
  • interpolation using scope, lookup/hiera, alias, or literal function
  • Hiera version 5 configuration in hiera.yaml
  • merge strategies (first, unique, hash, deep)
  • YAML data
  • JSON data
  • lookup options stored adjacent to data
  • convert_to type coercions
  • Sensitive data
  • configurable deep merge
  • pluggable back ends
  • explain functionality to show traversal
  • containerized REST-based microservice
  • JSON and YAML schema for the hiera.yaml config file (see schema/hiera_v5.yaml)
You can’t perform that action at this time.