[WIP] State management, variables, and config encryption

[Jc2k]

At present Touchdown is a great way of freezing a set of cloud resources into a reproducible idempotent bundle. But it isn't really a configuration management system, as there is no configuration of that bundle. Out of the box you can't deploy the same environment twice, once with m1.small and then again with m3.medium instances. You'd have to edit the Touchdownfile. This PR considers what a Touchdownfile would look like if we embraced the resource graph system to describe a configuration system.

Storage

First of all we need an abstraction for the storage of blobs of data.

folder = workspace.get_local_folder(name="deployment")
config = folder.add_file(name="test.cfg")

It needs to support S3:

folder = bucket.get_folder(name="deployment")
config = folder.add_file(name="test.cfg")

It needs to support encryption:

kms = aws.add_kms(name='deployment')
config = kms.add_encryption(file=config)

GPG too:

gpg = aws.add_gpg(name='deployment', passphrase='...', identity='...')
config = gpg.add_encryption(file=config)

(It should just work if you wrapped GPG in KMS and vice versa too).

The data format itself is just another layer. For ConfigParser it is just:

config = workspace.add_conf(file=config)

FIXME: Naming things is hard. file or streamable or obj or object. (stream isn't right as its not a stream, its more of a stream factory)

Settings

Each setting would be a resource as well:

setting = config.add_variable(
    name='scale:web.min',
    default=1,
)

There would by type information and validation metadata:

settings = config.add_integer(
  name='scale:web.min',
  min=0,
  max=10,
)

A Django secret key varies between environments. But it is generated according to a recipe and theres no reason to set it to a particular value.

setting = config.add_string(
    name='secret_key',
    default=lambda x: os.random(.......),
)

When this variable is first consumed it is generated using the value callable and stored in the settings object.

The cleanest mechanism for mapping a variable to a resource would be like this:

foo = aws.add_autoscaling_group(
    name='web',
    min=setting,
)

There is no mechanism for doing this presently - the min attribute of an autoscaling_group must be an integer or a serializer. The type validation of serializers is also weaker than it could be. The setter needs updating to be able to acquire a suitable serializer for one of these variables.

This approach decouples the Touchdownfile definition from the backing store. The same Touchdownfile could persist state to local disk, github API, S3. It also allows layering - the resources can be GPG or KMS encrypted.

UI enhancements

A new show subcommand to see the current state of a settings file (complete with default values set):

$ touchdown show settings.conf
[scale:web]
min=1

The new show command would eventually work on any named resource - you could use it on an autoscaling group to see its current state for example.

A new get command for seeing the current value of a setting:

$ touchdown get settings.conf scale:web.min
1

And a new set command:

$ touchdown set settings.conf scale:web.min 5

A refresh command would cause computed variables to be regenerated and the state store to be updated.