Utilizing Structured Configuration Mixed with Secrets

[dustinbrooks]

When using a structured config file, some of the entries are secrets. When pulling those specific JsonElements out of the collection, it would be nice to know if it IsSecret like you can do with Pulumi.Output.IsSecret

My specific scenario, I'm reading in a structured config like below, flattening it out, looping through each entry, and storing the key/value pairs in AWS Parameter Store. What's missing is for me to be able to check which of the values is a secret so, I can store it in Parameter Store as SecureString, right now they are going in plain text.

Pulumi.test.yaml

config: 
  Application:Settings:
    AppSettings:
      AllowedLoginAttempts: 4
      CaptchaInfo:
        CaptchaKey: FAKEKEYasdfasdfasdf
        CaptchaSecret:
          secure: REMOVEDojiijfjio32ji23jfj0f3093j923j09

Here is the code where I pull the data out of the config, flatten it, and I'm writing it out to console for testing. NOTE: Both config.RequireObject and config.RequireSecretObject work here. The difference being the latter returns a Pulumi.Output. Both end up working the same in my scenario, properly decrypting the secrets into Parameter Store, but showing them as `[secret]' when output to the console.

public AppConfig()
{
    var config = new Pulumi.Config("Application");
    JsonElement data = config.RequireObject<JsonElement>("Settings");  
    SettingsCollection = GetFlat(data.ToString());     
       
    Console.WriteLine("*************************************");
    Console.WriteLine(JsonSerializer.Serialize(SettingsCollection, new JsonSerializerOptions { WriteIndented = true }));
    Console.WriteLine("*************************************");          
}

GetFlat function returns a collection of flatten keys and their value ie: AppSettings/CaptchaInfo/CaptchaSecret

public static Dictionary<string, JsonElement> GetFlat(string? json)
{
    if (json == null) return new Dictionary<string, JsonElement>();
    IEnumerable<(string Path, JsonProperty P)> GetLeaves(string? path, JsonProperty p)
        => p.Value.ValueKind != JsonValueKind.Object
            ? new[] { (Path: path == null ? p.Name : path + "/" + p.Name, p) }
            : p.Value.EnumerateObject().SelectMany(child => GetLeaves(path == null ? p.Name : path + "/" + p.Name, child));
    using (JsonDocument document = JsonDocument.Parse(json)) 
        return document.RootElement.EnumerateObject()
            .SelectMany(p => GetLeaves(null, p))
            .ToDictionary(k => k.Path, v => v.P.Value.Clone()); //Clone so that we can use the values outside of using
}

Code for the looping and Parameter Store creation.

foreach (var item in appConfig.SettingsCollection)
{
    string name = $"/{prefix}/{item.Key}";
    new Ssm.Parameter(name, new Ssm.ParameterArgs
    {
        Name = name,
        Value = item.Value.ToString(),
        Type = "String", // is a secret? store as SecureString
        DataType = "text"
    });
}

If I could pull IsSecret for each JsonElement in the object, I would be able to then utilize it.

Link to slack conversation: https://pulumi-community.slack.com/archives/C84L4E3N1/p1624971038131700

[lukehoban]

Both config.RequireObject and config.RequireSecretObject work here.

You should generally use RequireSecretObject in cases where there are secrets in the config, or else the secretness will not be tracked and secrets could get leaked by your program. We are soon going to make this a warning/error to not use RequireSecretObject in these cases.

Both end up working the same in my scenario, properly decrypting the secrets into Parameter Store, but showing them as `[secret]' when output to the console.

I suspect this is because there's an additional filter on all CLI output which replaces anything that is a secret config value with [secret], even if it's not actually being treated as a Secret in the program. This is somewhat confusing, and we are considering changing the way this works to make it more clear.

Ultimately, if you use RequireSecretObject, you should then be able to use IsSecret to test whether the value really was secret.