Utilizing Structured Configuration Mixed with Secrets

[dustinbrooks]

When using a structured config file, some of the entries are secrets. When pulling those specific JsonElements out of the collection, it would be nice to know if it IsSecret like you can do with Pulumi.Output.IsSecret

My specific scenario, I'm reading in a structured config like below, flattening it out, looping through each entry, and storing the key/value pairs in AWS Parameter Store. What's missing is for me to be able to check which of the values is a secret so, I can store it in Parameter Store as SecureString, right now they are going in plain text.

Pulumi.test.yaml

config: 
  Application:Settings:
    AppSettings:
      AllowedLoginAttempts: 4
      CaptchaInfo:
        CaptchaKey: FAKEKEYasdfasdfasdf
        CaptchaSecret:
          secure: REMOVEDojiijfjio32ji23jfj0f3093j923j09

Here is the code where I pull the data out of the config, flatten it, and I'm writing it out to console for testing. NOTE: Both config.RequireObject and config.RequireSecretObject work here. The difference being the latter returns a Pulumi.Output. Both end up working the same in my scenario, properly decrypting the secrets into Parameter Store, but showing them as `[secret]' when output to the console.

public AppConfig()
{
    var config = new Pulumi.Config("Application");
    JsonElement data = config.RequireObject<JsonElement>("Settings");  
    SettingsCollection = GetFlat(data.ToString());     
       
    Console.WriteLine("*************************************");
    Console.WriteLine(JsonSerializer.Serialize(SettingsCollection, new JsonSerializerOptions { WriteIndented = true }));
    Console.WriteLine("*************************************");          
}

GetFlat function returns a collection of flatten keys and their value ie: AppSettings/CaptchaInfo/CaptchaSecret

public static Dictionary<string, JsonElement> GetFlat(string? json)
{
    if (json == null) return new Dictionary<string, JsonElement>();
    IEnumerable<(string Path, JsonProperty P)> GetLeaves(string? path, JsonProperty p)
        => p.Value.ValueKind != JsonValueKind.Object
            ? new[] { (Path: path == null ? p.Name : path + "/" + p.Name, p) }
            : p.Value.EnumerateObject().SelectMany(child => GetLeaves(path == null ? p.Name : path + "/" + p.Name, child));
    using (JsonDocument document = JsonDocument.Parse(json)) 
        return document.RootElement.EnumerateObject()
            .SelectMany(p => GetLeaves(null, p))
            .ToDictionary(k => k.Path, v => v.P.Value.Clone()); //Clone so that we can use the values outside of using
}

Code for the looping and Parameter Store creation.

foreach (var item in appConfig.SettingsCollection)
{
    string name = $"/{prefix}/{item.Key}";
    new Ssm.Parameter(name, new Ssm.ParameterArgs
    {
        Name = name,
        Value = item.Value.ToString(),
        Type = "String", // is a secret? store as SecureString
        DataType = "text"
    });
}

If I could pull IsSecret for each JsonElement in the object, I would be able to then utilize it.

Link to slack conversation: https://pulumi-community.slack.com/archives/C84L4E3N1/p1624971038131700

[lukehoban]

Both config.RequireObject and config.RequireSecretObject work here.

You should generally use RequireSecretObject in cases where there are secrets in the config, or else the secretness will not be tracked and secrets could get leaked by your program. We are soon going to make this a warning/error to not use RequireSecretObject in these cases.

Both end up working the same in my scenario, properly decrypting the secrets into Parameter Store, but showing them as `[secret]' when output to the console.

I suspect this is because there's an additional filter on all CLI output which replaces anything that is a secret config value with [secret], even if it's not actually being treated as a Secret in the program. This is somewhat confusing, and we are considering changing the way this works to make it more clear.

Ultimately, if you use RequireSecretObject, you should then be able to use IsSecret to test whether the value really was secret.

[dustinbrooks]

@lukehoban Its possible I'm missing something, but I tried with RequireSecretObject. Once I have the full object and then Apply so I can separate it out into its individual JsonElements, I no longer have a Pulumi.Output to run the IsSecret against. Its getting the whole object in one shot, but the whole object isn't a secret, only individual JsonElements. How do I test if an individual JsonElement is a secret.

[lukehoban]

Once I have the full object and then Apply so I can separate it out into its individual JsonElements, I no longer have a Pulumi.Output to run the IsSecret against.

Doing an Apply on an Output should return another Output, and propagate the secretness bit to it (in fact, doing precisely that is effectively what secretness means in Pulumi). The whole Output returned by RequireSecretObject should indicate whether there are any secrets in the value.

[dustinbrooks]

Okay, I think that should work. I just have to iterate down using the Outputs rather than doing the Apply only at the higher level. That makes sense, Thanks Luke.

[dustinbrooks]

@lukehoban Well, I keep getting back true for every IsSecret call regardless of the JsonElement or JsonProperty I run it against.

So using the sample config at the top, I call this function to get a single Pulumi.Output, which is the JsonProperty value for "AllowedLoginAttempts".

static Func<Pulumi.Output<int>> getOutput =>
  {
      var config = new Pulumi.Config("Application");
      Pulumi.Output<JsonElement> data = config.RequireSecretObject<JsonElement>("Settings");
      
      return data.Apply(j => {
          return JsonDocument.Parse(j.ToString()!)
              .RootElement
              .GetProperty("AppSettings")
              .GetProperty("AllowedLoginAttempts")
              .GetInt32();                 
      });
  };

var item = getOutput();

var issecret = Pulumi.Output.IsSecretAsync(item);
issecret.Wait();
Console.WriteLine($"{issecret.Result}");

When I check it for IsSecret it returns true, which is incorrect. The only secret is at the bottom "CaptchaSecret". I feel like I have to be doing something wrong if this should be working.