yaml set Examples

1. Introduction
2. Setting Values from Shell Scripts
   1. Data Files As Templates
3. Password Rotation
   1. Unverified Changes
   2. Verified Changes

Introduction

This page explores various real-world use-cases for the yaml-set command-line tool. This is not an exploration of the Segments-of-a-YAML-Path, which are already well-covered.

It is important to know that yaml-set exists to set exactly one value to one or more nodes simultaneously in an existing YAML/JSON/EYAML/Compatible file (even if the file is completely empty). The structure to store the value does not have to pre-exist unless you set --mustexist (-m); this command can automatically build most missing structure like missing Hash keys and Array elements at any depth.

Should you need to set more than one discrete value at the same time, set a complex value like an entire Array or Hash, or write to a file which does not already exist, use yaml-merge, instead.

Setting Values from Shell Scripts

It is trivial to set precise values in YAML/JSON/EYAML/Compatible files from the command-line or shell-script (or anything which can run commands or import Python libraries). In the most rudimentary case, consider this shell script:

#!/bin/bash
################################################################################
# Aggregate and tally data, producing a simple report.
################################################################################
reportFile=~/tally_report.yaml
dateNow=$(date +%Y%m%d)

# Merge all data files together (one per event source) and tally the number of
# generated events for today, only.  Note that this is performed as a stream
# operation; no files are changed by any of this.
eventTally=$(yaml-merge \
                 --config=~/tally_report.ini \
                 /var/event_data/event-source-*.{yaml,json} \
             | yaml-get --query="/events/sources/*/date/${dateNow}/observations/*" \
             | wc -l)

# yaml-set requires the data file already exist, so make one when missing and
# write the new tally to it.
[ -f "$reportFile" ] || echo "---" >"$reportFile"
yaml-set --change="/observations/${dateNow}" --value="$eventTally" "$reportFile"

In this use-case, several tools are combined to parse a great many data files in order to tally the number of observed events per day. A variable number of YAML and JSON data files are first merged together; none are changed by this process. From the merged result, a listing of every record -- across all sources and types -- is made for the present day. That list is tallied and the resulting count is stored in a shell script variable. That variable is then written to a different YAML data file. If the output file does not already exist, a minimally viable YAML file is created to receive the report data. If the document structure and key for the new tally data doesn't already exist, it is created on-the-fly.

To illustrate, suppose we have these two files (among many others):

File: event-source-anemometer.yaml

---
events:
  sources:
    anemometer:
      date:
        20201008:
          observations:
            - time: 0041:33.548
              data: 7
              aux: NW
            - time: 0041:33.983
              data: 4
              aux: N

File event-source-barometer.json

{
  "events":{
    "sources":{
      "barometer":{
        "date":{
          "20201008":{
            "observations":[
              {
                "time":"0041:33.548",
                "data":1048.72,
                "aux":"Rising"
              },
              {
                "time":"0041:33.983",
                "data":1048.77,
                "aux":"Rising"
              }
            ]
          }
        }
      }
    }
  }
}

Upon running the above shell script on the same day as the observations, we'd end up with a report YAML file like:

---
observations:
  20201008: 4

Now, imagine there were far more event data sources with much more data to chew. While we could certainly perform more interesting tasks with the data at hand, this demonstration merely shows that these tools can work together to make otherwise very complex tasks, trivially simple.

Data Files As Templates

You can also combine yaml-set's ability to stream document with YAML Path's wildcard segment to treat YAML/JSON/Compatible data files as templates.

#!/bin/bash
################################################################################
# Implement a TEMPLATE data file for each new domain.  There are several places
# within the data file where the TEMPLATE_DOMAIN would ordinarily need to be
# updated by hand.  This script automates all of those updates.
################################################################################
$newDomain=${1:?"ERROR:  You must set the new domain!"}
domainFile="/var/www/domains/${newDomain}.json"
cat /var/lib/vendor/templates/domain_admin_template.json | \
    yaml-set --change='**[.=TEMPLATE_DOMAIN]' --value="$newDomain" >"$domainFile"

This simple script takes the name of a new domain, then pipes a template file through yaml-set which changes every use of a predetermined template variable to the new domain, writing the updated file into place. Note that only exact matches are made; any fields which combine the template variable with anything else are ignored.

If you would rather replace anything and everything which contained the TEMPLATE_VARIABLE anywhere within it, you could use /**/*TEMPLATE_VARIABLE*, instead. This however, would be less precise, replacing text like "something TEMPLATE_VARIABLE", "TEMPLATE_VARIABLE something else", and such.

Password Rotation

When dealing with secrets, use some form of encryption. The yaml-set tool has native support for EYAML, which will be demonstrated here. You can pipe in new values from any other encryption solution you have, but yaml-set won't be able to compare decrypted values for the --check (-c) option unless you use EYAML (should you care for a simple verification before rotation).

Note that rotating a password is slightly different from changing a password. When you change a password, you just write the new value to the correct field and you're done. To rotate a password, you need to save off the old value so it can be used to affect the same change in some other system. Such will be the pattern for this use-case.

For this presentation, the original YAML file will be:

File: CREDENTIALS.YAML

---
credentials:
  username: >
    ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEw
    DQYJKoZIhvcNAQEBBQAEggEAiT2ZIXN5RGzIFPdBuXq6gL46q+8qsvQN7riR
    zApiSfC8D5jbNXONXUdC67ClYSf0wyAfMvGCfqd6pv3KQ2wi8tDe58SJ16Of
    BJrpIb6AtKZAsHYY6wTa4D8DST9sCp1REibVXbIF4kqphvgaR9LilmhCJ/Y0
    Xj74sT0QBBL5SGgA9TAUVio2Eo2sEZGV5fTntFplT17qi9AyigYlwANUmMz6
    quVv64GwBe+E7hCUQw7NjVC8UhydZM+DXxJBmTvp7kKTkJwfOxKHdgLPJa9u
    UvKzKTO0GnX7uShHKGuViKsDxXxrg1/KII48zF35jCjdeE4g+MQ5C4tfBWUv
    kh3aKzA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBC31J5O4xavJNOU2JEr
    CVGVgBBi/wYVAANhLHN2Ah4ThNxC]
  password: >
    ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAACAQEw
    DQYJKoZIhvcNAQEBBQAEggEAQcvTDq8m2G+9bBC9xrZgHhyg8tjjvZeufFxU
    p12TAUZ5Kf1FD4Rp45NUJ7/9cWV09oTBCN4CW6SFh7X/aFoIvv8mKqzk9Up2
    KLBQIUVbhlh+5dVPPsZ8xX+I7zipL743XGzRHWm3KqzIR1cvpkq+eNsgPRMG
    qvOh2JaMkampbK3StMgVPrt8R4JcMABlyV2ICRN9yyXPlF7N3agYiREn+skn
    0sKwFdjn4n/V3JwDsm1ELzbVmsCzvr9M5dIO95pUhs+c3T/MmoOIjmR+X0/T
    cKed2qcycHLE0PPOJktT0Hiiv9Bmz0pgx5yF5SY8g44+yyXyFaT8NSADc/+4
    G/9C5zBMBgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBAvEEg8whyHuW228o5S
    EI38gCDw0Jvdzzv7oQ439Js8Hc8rANdbgu6IFYk5HM7ddI2lew==]

Any values which take the form of ENC[algorythm,value] are EYAML values which were encrypted using a key-pair. For this demonstration, they use block (i.e.: folded) values to make them presentable; they are valid in this form.

The data file is one small part of a holistic credential management system. Another part would be some code which actually performs the remote credential change. For simplicity sake, this hypothetical shell script serves such a purpose:

File: ROTATE-PASSWORD.SH

#!/bin/bash
###############################################################################
# Conduct a (fictional) password rotation.
#
# This works by attempting a remote system connection using the "present"
# credentials.  Should it fail due to bad credentials, the change is attempted
# using the "former" credentials.
###############################################################################
dataFile=${1:?"ERROR:  A credentials file (YAML or JSON) must be provided!"}
userName=$(yaml-get --query=/credentials/username "$dataFile")
passWord=$(yaml-get --query=/credentials/password "$dataFile")

# Verify data was retrieved
if [ -z "$userName" -o -z "$passWord" ]; then
	echo "ERROR:  Verify both username and password are at /credentials in ${dataFile}." >&2
	exit 1
fi

# Attempt to connect using the present credentials
some-client --username="$userName" --password="$passWord"
exitState=$?

# Assume an exit-state of 86 means "wrong credentials"
if [ 86 -eq $exitState ]; then
	# Get the former password and attempt a rotation
	oldPass=$(yaml-get --query=/credentials/password_old "$dataFile")
	if [ -z "$oldPass" ]; then
		echo "ERROR:  Bad credentials and no former password to rotate!" >&2
		exit 1
	fi

	some-client --username="$userName" --password="$oldPass" --newpassword="$passWord"
	exitState=$?
fi

# Report success/fail to the calling process
exit $exitState

Note that this script requires the former password to be found at /credentials/password_old with the present password at /credentials/password.

Unverified Changes

When you do not have the old value, cannot verify the old value, simply don't care, or cannot risk exposing the old value in your machine's process list or command history, you can make an unverified change. This would look like: some-password-generator | yaml-set --change=/credentials/password --saveto=/credentials/password_old --stdin --eyamlcrypt CREDENTIALS.YAML

In this example, another fictitious command, some-password-generator, securely produces a new password which is piped into yaml-set across STDIN. This value cannot be snooped. Further, the yaml-set command encrypts the value using EYAML and the system- or user-default encryption keys available to the system running the command. The new password is not disclosed during this change and it is encrypted at storage.

The resulting data file would look like:

---
credentials:
  username: >
    ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEw
    DQYJKoZIhvcNAQEBBQAEggEAiT2ZIXN5RGzIFPdBuXq6gL46q+8qsvQN7riR
    zApiSfC8D5jbNXONXUdC67ClYSf0wyAfMvGCfqd6pv3KQ2wi8tDe58SJ16Of
    BJrpIb6AtKZAsHYY6wTa4D8DST9sCp1REibVXbIF4kqphvgaR9LilmhCJ/Y0
    Xj74sT0QBBL5SGgA9TAUVio2Eo2sEZGV5fTntFplT17qi9AyigYlwANUmMz6
    quVv64GwBe+E7hCUQw7NjVC8UhydZM+DXxJBmTvp7kKTkJwfOxKHdgLPJa9u
    UvKzKTO0GnX7uShHKGuViKsDxXxrg1/KII48zF35jCjdeE4g+MQ5C4tfBWUv
    kh3aKzA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBC31J5O4xavJNOU2JEr
    CVGVgBBi/wYVAANhLHN2Ah4ThNxC]
  password: >
    ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEw
    DQYJKoZIhvcNAQEBBQAEggEAY4g5Rf3NWmNDvHDttCZlZ+Qk0XGxRFEBKOZZ
    yG0KWBQBpWQxVa/qjKo8FMTPU1wsbt26YwySl0yB4JcKWR4Oea8qSRTbHufp
    rpA5gCucsF3pADb5zkHbJ2rhArRf2X9ws82JfYXOkcPAJIm3yznlbLUQZJ6V
    bHwXb5BwkrLc3CdRJSL6T5NSODwWCCAAz0hSHbCHyZtjbB77TNIXjXXLV2HT
    97y3qa3g2PYg5ragKkxktV0bcsVL0aLX6WsPKXInebkeRT9XyfpMrGP6Zc25
    px6096bXQQXNsP2N7+Zv43PDFvwHatqEgLsFwT1WSpGZHD0EmkP1Q7kg0qNL
    SsTVbzA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBD9ILv+9pxfHRqaNu2/
    x5EMgBBC+K3C+zoRiErCKfeKuFXt]
  password_old: >
    ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAACAQEw
    DQYJKoZIhvcNAQEBBQAEggEAQcvTDq8m2G+9bBC9xrZgHhyg8tjjvZeufFxU
    p12TAUZ5Kf1FD4Rp45NUJ7/9cWV09oTBCN4CW6SFh7X/aFoIvv8mKqzk9Up2
    KLBQIUVbhlh+5dVPPsZ8xX+I7zipL743XGzRHWm3KqzIR1cvpkq+eNsgPRMG
    qvOh2JaMkampbK3StMgVPrt8R4JcMABlyV2ICRN9yyXPlF7N3agYiREn+skn
    0sKwFdjn4n/V3JwDsm1ELzbVmsCzvr9M5dIO95pUhs+c3T/MmoOIjmR+X0/T
    cKed2qcycHLE0PPOJktT0Hiiv9Bmz0pgx5yF5SY8g44+yyXyFaT8NSADc/+4
    G/9C5zBMBgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBAvEEg8whyHuW228o5S
    EI38gCDw0Jvdzzv7oQ439Js8Hc8rANdbgu6IFYk5HM7ddI2lew==]

You can see the former password was moved to a newly created key where specified, /credentials/password_old, and the new password was put in its place at /credentials/password.

The password will be changed on the remote system when next the helper script, ROTATE-PASSWORD.SH, runs.

WARNING: The ROTATE-PASSWORD.SH script -- or whatever its real-world equivalent -- must run before the above yaml-set command runs again lest the original password be forever lost. This is one consequence of not verifying the old value before rotating it. You can create one more level of defense against such an accident by adding the --backup flag to yaml-set, but that gives only one more change roll-back. Two runs of the yaml-set command above would still cause the original value to be lost.

Verified Changes

When possible, you should verify the old credentials before rotating them. There are multiple means of doing so. You could (should) first attempt to establish an ephemeral logon session using the old credentials. This case is not presented in this discussion. Rather, if you have access to the old credentials and don't fear exposing them to the process list on whatever machine is conducting the credential rotation, you can use yaml-set to enforce that the old value is presently set before attempting to change it. Such a command would look like: some-password-generator | yaml-set --change=/credentials/password --saveto=/credentials/password_old --stdin --eyamlcrypt --check="THE OLD VALUE" CREDENTIALS.YAML (where "THE OLD VALUE" is the actual, unencrypted original value).

This produces output like that of the previous case with one exception: should the check value be wrong, no change will be made and an informative error message will be printed to STDERR. Because the original value is displayed on the command-line, this should be considered a last-resort assurance of valid credential rotation.