#WIP #FF fea: Log config changes.

[artsyjian]

https://artsyproduct.atlassian.net/browse/PLATFORM-2958
Supported by: https://github.com/artsy/infrastructure/pull/257

To demonstrate the idea of logging config changes.

Every run of hokusai...env set/unset... generates a log like these (values may be sensitive so are omitted):

time: 2020-11-03 00:15:32.537098, namespace: default, action: set, var: FOO
time: 2020-11-03 00:16:10.925644, namespace: default, action: unset, var: FOO

and ships it to S3:

Might these logs be useful?

The implementation is not ideal. S3 files are clunky to work with, and updating them adds ~1 second delay to Hokusai's response, which is noticeable. If we consider config changes also as releases, perhaps better to let Horizon manage storage (in a DB), so that Hokusai can just POST these events to Horizon? And later on we make them retrievable via a hokusai...env log command?

If we do go with this implementation, I shall add tests.

[joeyAghion]

Sorry for delay! This was compounded by the fact that I don't see perfect solutions to some of these challenges.

Generally, yes I think being able to trace through config changes is useful. I'm not in love with S3 as the destination because that's somewhat arbitrary and a significant new scope/dependency-increase for Hokusai. Logging the changes with our regular logging solution isn't desirable due to the sensitivity. Putting them in Horizon's database also seems like an arbitrary scope-increase for that system. I keep coming back to whether we can just use kubernetes's native concepts for this, like storing the config-map archives as config-maps (or maybe even within some meta-config-map). Or is there another handy key/value store?

[izakp]

@artsyjian as we discussed on Slack I think, would advise that we try and use a Kubernetes ConfigMap to log changes rather than an S3 bucket

[dblandin]

I keep coming back to whether we can just use kubernetes's native concepts for this, like storing the config-map archives as config-maps (or maybe even within some meta-config-map). Or is there another handy key/value store?

One thought is to use AWS Parameter Store. I've used it the past along with segmentio/chamber to manage configuration for services running in Elastic Container Service.

Some benefits

• Fast
• CLI support
• AWS SDK support
• Versioning
• Can encrypt input

Would be great to stay as close to Kubernetes as possible. If there's a way to react whenever a ConfigMap is created/updated (via hokusai / UI dashboard / etc) and automatically persist the value into Parameter Store, that might be pretty cool. We could then use the Parameter Store APIs within hokusai to list config map versions and rollback to previous states.

Resources

• https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html
• https://docs.aws.amazon.com/cli/latest/reference/ssm/index.html#cli-aws-ssm

Examples

Setting a secure parameter with a JSON payload

$ aws ssm put-parameter --name 'hokusai.force.staging' --type SecureString --value='{ "key1: "0" }'

Retrieving a secure parameter with decryption

$ aws ssm get-parameter --name 'hokusai.force.staging' --with-decryption
{
    "Parameter": {
        "Name": "hokusai.force.staging",
        "Type": "SecureString",
        "Value": "AQICAHhAgvzOnt3NYiFUmNvonq2rTa69MkqjugF47bK/kLyYYQFUCejUHRDawH4uUlZAfCQtAAAAbTBrBgkqhkiG9w0BBwagXjBcAgEAMFcGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMa5gT13X4nHXK0JwXAgEQgCoTZkT/X91vBrPSj13rdSfZwOQBfLAanI5oUlBgiB0k87N2+2NClXbSnfc=",
        "Version": 4,
        "LastModifiedDate": "2021-02-19T11:03:37.820000-05:00",
        "ARN": "arn:aws:ssm:us-east-1:585031190124:parameter/hokusai.force.staging",
        "DataType": "text"
    }
}

Parsing JSON from decrypted parameter

$ aws ssm get-parameters --name 'hokusai.force.staging' --query "Parameters[0].Value" --with-decryption | jq '.|fromjson'
{
  "key1": "0"
}

Retrieving parameter history

$ aws ssm get-parameter-history --name 'hokusai.force.staging' --with-decryption | jq '.Parameters[] | { version: .Version, value: .Value }'
{
  "version": 1,
  "value": "{ \"key1\": \"0\" }"
}
{
  "version": 2,
  "value": "{ \"key1\": \"1\" }"
}
{
  "version": 3,
  "value": "{ \"key1\": \"3\" }"
}
{
  "version": 4,
  "value": "{ \"key1\": \"3\", key2: \"0\" }"
}

[artsyjian]

Thanks @dblandin for the idea. The concept is really cool. If we want configs archived, a non-k8s key/value store (AWS Parameter Store, HashiCorp Vault, ...) makes a lot of sense.

@izakp , @joeyAghion - About using k8s configmap concept, it seems to depend on whether we want changelogs or archives.

For logging, configmap should work. I imagine each app-environment configmap can have a buddy app-environment-changes configmap. When FOO=bar is added (or deleted/modified) to app-environment, we can append to app-environment-changes with a line like this:

time: 2020-11-03 00:20:27.384867, user: banksy, action: set, var: FOO, value: bar

A configmap's size is limited at 1MB. If a line averages 200 bytes, app-environment-changes can store 5k deltas which should be years worth.

However, if we want config archives, we probably shouldn't use configmaps. Say we archive each old version of app-environment in a dedicated configmap, we will end up with hundreds or even 1k+ configmaps. At that number, we might encounter problems as someone did. Or, if we pool all the old versions into one giant configmap, it won't hold enough revisions, due to the 1MB limit (force's configmap is ~20KB for example). The total storage size of old config versions across the entire cluster might also be a concern as etcd seems designed to not store too much data:

Request size limit

etcd is designed to handle small key value pairs typical for metadata. Larger requests will work, but may increase the latency of other requests. By default, the maximum size of any request is 1.5 MiB. This limit is configurable through --max-request-bytes flag for etcd server.
Storage size limit

The default storage size limit is 2GB, configurable with --quota-backend-bytes flag. 8GB is a suggested maximum size for normal environments and etcd warns at startup if the configured value exceeds it.