Salt for configuration management proof of concept

• SaltStack overview
• Ansible User's Guide to Saltstack
• Salt In 10 Minutes Walkthrough
• Salt User Guide
• Salt Documentation

Directory structure

config/                         # Configuration files
    master.conf                 # Master configuration
pillar/                         # Pillar data                      
    top.sls                     # Pillar top file
    salt/                       # Salt variables
        common.sls              # Common variables
        master.sls              # Master variables          
        minion.sls              # Minion variables
states/                         # Salt state formulas
    top.sls                     # Formula top file
    salt/                       # Formulas for configuring Salt
        init.sls                # Init formula          
        master.sls              # Master formula
        minion.sls              # Minion formula
        files/                  # Files for formula
            roots.conf.jinja    # Jinja template for roots.conf

• config/ contains configuration files for the Salt master and minion that need to be copied manually.
• config/master.conf contains the Salt master configuration.
• pillar/ contains pillar data AKA the variables.
• pillar/top.sls contains the pillar top file that defines which pillar data is applied to which minions. The top file is not used as a location to declare variables and their values, it is used as a way to include other pillar files and organize the way they are matched based on environments or grains.
• states/ contains state formulas AKA the "playbooks".
• states/top.sls contains the state top file that defines which state formulas are applied to which minions. The top file is not used as a location to declare formulas, it is used to include other state files and organize the way they are matched based on environments or grains.
• Each state formula is a directory under states/ with an init.sls file that contains the state formula. The init.sls file is the entry point for the formula. The init.sls file can contain other state formulas as dependencies. files/ contains files and Jinja templates that are used by the formula.

Setup

Master setup

By default, the minions assume that the Salt master can be resolved in DNS using the hostname salt. Change the DNS records for the Salt master to point salt.example.com (replacing example.com with your domain) and hostname of the salt master to salt:

sudo hostnamectl set-hostname salt

To install Salt on CentOS 7:

Run the following commands to install the Salt Project repository and key:

sudo rpm --import https://repo.saltproject.io/salt/py3/redhat/7/x86_64/latest/SALTSTACK-GPG-KEY.pub
curl -fsSL https://repo.saltproject.io/salt/py3/redhat/7/x86_64/latest.repo | sudo tee /etc/yum.repos.d/salt.repo

Then do sudo yum clean expire-cache to clear the repository metadata.

Install master and minion (because master will also be managed by itself):

sudo yum install salt-master
sudo yum install salt-minion

Set up the configuration management serivce account SSH keys that can clone Salt repositories under /root/.ssh/id_rsa and /root/.ssh/id_rsa.pub.

From this repositories config directory copy master.conf to /etc/salt/master.d/.

Clone this repository to /srv/salt/master (the base environment root directory):

mkdir /srv/salt; git clone git@gitlab.hpc.taltech.ee:hpc/salt/salt-poc.git /srv/salt/master

Then enable and start both master and minion:

sudo systemctl enable salt-master && sudo systemctl start salt-master
sudo systemctl enable salt-minion && sudo systemctl start salt-minion

Provided that the hostname is salt as described, the minion should automatically try to authenticate with master. Check if there are any unaccepted keys with salt-key -L.

Accept the minion key with salt-key -A (command to accept all keys).

And run salt 'salt' state.apply salt.refresh_repo (rerun on auth error). This will bootstrap the Salt environments per branch.

Minion setup

To install Salt on CentOS 7:

Run the following commands to install the Salt Project repository and key:

sudo rpm --import https://repo.saltproject.io/salt/py3/redhat/7/x86_64/latest/SALTSTACK-GPG-KEY.pub
curl -fsSL https://repo.saltproject.io/salt/py3/redhat/7/x86_64/latest.repo | sudo tee /etc/yum.repos.d/salt.repo

Then do sudo yum clean expire-cache to clear the repository metadata.

Install minion:

sudo yum install salt-minion

Then enable and start minion:

sudo systemctl enable salt-minion && sudo systemctl start salt-minion

By default, the minions assume that the Salt master can be resolved in DNS using the hostname salt. If this is so, the only thing needed is to accept the minion authentication key on master with salt-key -A (command to accept all keys).

Version controlling Salt

It is possible to use gitfs alongside the default rootfs to automatically fetch Salt formulas, pillars etc from remote git repos, however gitfs is unfortunately buggy and full of python version mismatch errors (Check the warnings in the gitfs documentation).

Thus it is better to use the salt.refresh_repo formula (in states/salt/refresh_repo.sls). This state ensures that the salt-master service is running. It pulls the list of branches from git remote and makes sure that that each branch is cloned into a directory under /srv/salt/. It also manages a file in /etc/salt/master.d/roots.conf which defines an environment for each branch and restarts the salt-master process when the file changes. This uses one repository for both states and pillars, so states are in the states/ directory and pillars are in the pillar/ directory.

The salt.refresh_repo formula is run on schedule defined in the master.conf:

schedule:
  pull_latest_git:
    function: salt.refresh_repo
    cron: '* * * * *'

It is also possible to pull the latest on commit using the steps defined in this blog post.

Best practices, procedures and style guide

1. Modularity and clarity should be emphasized whenever possible.
2. Create clear relations between pillars and states.
3. Use variables when it makes sense but don't overuse them.
4. Store sensitive data in pillar.
5. Don't use grains for matching in your pillar top file for any sensitive pillars.

Keep directory structure shallow and flat. Total depth should 3 levels. The files/ directory (example: states/<state name>/files/) should be the deepest.

When referencing state formulas, keep in mind the SLS File Namespace.

Separate state formulas into separate files and include them in the main init.sls file.

Use snake_case for file names.

Non-SLS files should be in files/ directory. Including jinja templates which should be suffixed with .jinja.

"Unhide" dot files with the prefix dot_. For example, .bashrc should be named dot_bashrc.

Comment Salt-managed files with # Managed by Salt, do not edit manually! at the top.

Pillars should contain as little as possible to reduce rendering load on the master, but what should definitely be kept in pillars is very specific variables like sensitive/secret data.

Define variables in pillars by grains using jinja to set differing variables according to OS, environment, custom grains etc.

Use grains and custom grains on minion configuration to match minions to states and handle variables in defaults.yaml.

Use SLS template variables.

Don't Repeat Yourself (keep it DRY). Write formulas to be generic and reusable. Use includes and extends to override and avoid repetition.

Avoid cascading style, separate each task into its own section with its own declaration ID.

Use name declarations to avoid clashing ID declarations.

Separate logic from states to maintain readability. Avoid inline conditionals, instead use conditionals outside state definitions and use variables to pass the result to the state definition.

Jinja is not a programming language: move complex queries into custom pillars/grains and complex logic/control into Python execution modules.

Default render pipe is #!jinja|yaml. To use Python you can add python to the pipe, for example #!py|jinja|yaml.

Environments are defined in the state top file.

Schedule jobs with schedules from pillars.