Support lists and maps in config #3342
Conversation
|
I'm working on writing up docs to add to https://www.pulumi.com/docs/intro/concepts/config/ |
|
One change I had suggested to Luke but have not implemented is adding a |
There was a problem hiding this comment.
A few nit picks here and there. My main concern is that we don't have any integration tests for setting, getting, and removing configuration values using --path by just invoking pulumi directly.
I worry that we are missing some bugs around the corner cases with the way we interpret the --path syntax.
|
@pgavlin, I'd love to get your thoughts on one aspect of this... These are available to Pulumi programs as JSON strings (with all secret leaf values in the object decrypted), so the idea is you'd use the existing If you had any secrets in the object, you'd probably want to use I am wondering if we should do anything special in If we did want to do something like that, we would need to pass more information to the language plugin because right now we pass the config essentially as plaintext key/value pairs (in an environment variable), so we don't know which values were secrets (and for objects, don't know which leaf values in the object were secrets). Aside: it would be nice to have such additional information about config secrets in the language plugin as then we could make the regular (non-secret) config.get/require functions emit a warning that the config.getSecret/requireSecret function variants should be used instead. |
|
@pgavlin, @lukehoban, mind taking a look? I started adding some integration tests in a separate branch (https://github.com/pulumi/pulumi/commits/justin/config-tests). Just need to clean that up and write a few more and I'll add them as a commit to this PR. |
This change adds support for lists and maps in config. We now allow
lists/maps (and nested structures) in `Pulumi.<stack>.yaml` (or
`Pulumi.<stack>.json`; yes, we currently support that).
For example:
```yaml
config:
proj:blah:
- a
- b
- c
proj:hello: world
proj:outer:
inner: value
proj:servers:
- port: 80
```
While such structures could be specified in the `.yaml` file manually,
we support setting values in maps/lists from the command line.
As always, you can specify single values with:
```shell
$ pulumi config set hello world
```
Which results in the following YAML:
```yaml
proj:hello world
```
And single value secrets via:
```shell
$ pulumi config set --secret token shhh
```
Which results in the following YAML:
```yaml
proj:token:
secure: v1:VZAhuroR69FkEPTk:isKafsoZVMWA9pQayGzbWNynww==
```
Values in a list can be set from the command line using the new
`--path` flag, which indicates the config key contains a path to a
property in a map or list:
```shell
$ pulumi config set --path names[0] a
$ pulumi config set --path names[1] b
$ pulumi config set --path names[2] c
```
Which results in:
```yaml
proj:names
- a
- b
- c
```
Values can be obtained similarly:
```shell
$ pulumi config get --path names[1]
b
```
Or setting values in a map:
```shell
$ pulumi config set --path outer.inner value
```
Which results in:
```yaml
proj:outer:
inner: value
```
Of course, setting values in nested structures is supported:
```shell
$ pulumi config set --path servers[0].port 80
```
Which results in:
```yaml
proj:servers:
- port: 80
```
If you want to include a period in the name of a property, it can be
specified as:
```
$ pulumi config set --path 'nested["foo.bar"]' baz
```
Which results in:
```yaml
proj:nested:
foo.bar: baz
```
Examples of valid paths:
- root
- root.nested
- 'root["nested"]'
- root.double.nest
- 'root["double"].nest'
- 'root["double"]["nest"]'
- root.array[0]
- root.array[100]
- root.array[0].nested
- root.array[0][1].nested
- root.nested.array[0].double[1]
- 'root["key with \"escaped\" quotes"]'
- 'root["key with a ."]'
- '["root key with \"escaped\" quotes"].nested'
- '["root key with a ."][100]'
Note: paths that contain quotes can be surrounded by single quotes.
When setting values with `--path`, if the value is `"false"` or
`"true"`, it will be saved as the boolean value, and if it is
convertible to an integer, it will be saved as an integer.
Secure values are supported in lists/maps as well:
```shell
$ pulumi config set --path --secret tokens[0] shh
```
Will result in:
```yaml
proj:tokens:
- secure: v1:wpZRCe36sFg1RxwG:WzPeQrCn4n+m4Ks8ps15MxvFXg==
```
Note: maps of length 1 with a key of “secure” and string value are
reserved for storing secret values. Attempting to create such a value
manually will result in an error:
```shell
$ pulumi config set --path parent.secure foo
error: "secure" key in maps of length 1 are reserved
```
```shell
$ pulumi config --json
```
Will output:
```json
{
"proj:hello": {
"value": "world",
"secret": false,
"object": false
},
"proj:names": {
"value": "[\"a\",\"b\",\"c\"]",
"secret": false,
"object": true,
"objectValue": [
"a",
"b",
"c"
]
},
"proj:nested": {
"value": "{\"foo.bar\":\"baz\"}",
"secret": false,
"object": true,
"objectValue": {
"foo.bar": "baz"
}
},
"proj:outer": {
"value": "{\"inner\":\"value\"}",
"secret": false,
"object": true,
"objectValue": {
"inner": "value"
}
},
"proj:servers": {
"value": "[{\"port\":80}]",
"secret": false,
"object": true,
"objectValue": [
{
"port": 80
}
]
},
"proj:token": {
"secret": true,
"object": false
},
"proj:tokens": {
"secret": true,
"object": true
}
}
```
If the value is a map or list, `"object"` will be `true`. `"value"` will
contain the object as serialized JSON and a new `"objectValue"` property
will be available containing the value of the object.
If the object contains any secret values, `"secret"` will be `true`, and
just like with scalar values, the value will not be outputted unless
`--show-secrets` is specified.
Map/list values are available to Pulumi programs as serialized JSON, so
the existing
`getObject`/`requireObject`/`getSecretObject`/`requireSecretObject`
functions can be used to retrieve such values, e.g.:
```typescript
import * as pulumi from "@pulumi/pulumi";
interface Server {
port: number;
}
const config = new pulumi.Config();
const names = config.requireObject<string[]>("names");
for (const n of names) {
console.log(n);
}
const servers = config.requireObject<Server[]>("servers");
for (const s of servers) {
console.log(s.port);
}
```
Instead of just silently overwriting, if the type of the value does not match the type of the key (e.g. ints for arrays; strings for maps), return an error.
justinvp
force-pushed
the
justin/config
branch
from
8849127
to
98eeaeb
Compare
We can instead just use non-empty `objectValue` to indicate the value is an object.
Just to make it more clear that this flag is related to the `--config` values.
This change adds support for lists and maps in config. We now allow lists/maps (and nested structures) in
Pulumi.<stack>.yaml(orPulumi.<stack>.json; yes, we currently support that).For example:
While such structures could be specified in the
.yamlfile manually, we support setting values in maps/lists from the command line.As always, you can specify single values with:
$ pulumi config set hello worldWhich results in the following YAML:
proj:hello worldAnd single value secrets via:
$ pulumi config set --secret token shhhWhich results in the following YAML:
Values in a list can be set from the command line using the new
--pathflag, which indicates the config key contains a path to a property in a map or list:Which results in:
Values can be obtained similarly:
Or setting values in a map:
$ pulumi config set --path outer.inner valueWhich results in:
Of course, setting values in nested structures is supported:
$ pulumi config set --path servers[0].port 80Which results in:
If you want to include a period in the name of a property, it can be specified as:
Which results in:
Examples of valid paths:
Note: paths that contain quotes can be surrounded by single quotes when set from the command line.
When setting values with
--path, if the value is"false"or"true", it will be saved as the boolean value, and if it is convertible to an integer, it will be saved as an integer. (Unless it is a secret, in which case it will always be stored as an encrypted string).Secure values are supported in lists/maps as well:
$ pulumi config set --path --secret tokens[0] shhWill result in:
Note: maps of length 1 with a key of “secure” and string value are reserved for storing secret values. Attempting to create such a value manually will result in an error:
Accessing config values from the command line with JSON
Will output:
{ "proj:hello": { "value": "world", "secret": false, }, "proj:names": { "value": "[\"a\",\"b\",\"c\"]", "secret": false, "objectValue": [ "a", "b", "c" ] }, "proj:nested": { "value": "{\"foo.bar\":\"baz\"}", "secret": false, "objectValue": { "foo.bar": "baz" } }, "proj:outer": { "value": "{\"inner\":\"value\"}", "secret": false, "objectValue": { "inner": "value" } }, "proj:servers": { "value": "[{\"port\":80}]", "secret": false, "objectValue": [ { "port": 80 } ] }, "proj:token": { "secret": true, }, "proj:tokens": { "secret": true, } }If the value is a map or list,
"value"will contain the object as serialized JSON and a new"objectValue"property will be available containing the value of the object.If the object contains any secret values,
"secret"will betrue, and just like with scalar values, the value will not be outputted unless--show-secretsis specified.Accessing config values from Pulumi programs
Map/list values are available to Pulumi programs as serialized JSON, so the existing
getObject/requireObject/getSecretObject/requireSecretObjectfunctions can be used to retrieve such values, e.g.:Fixes #2306