[dotnet] pulumi preview fails when giving a Provider with unknowns to a ConfigGroup

[desteves]

What happened?

I'm testing a C# Pulumi program with pulumi preview and ran into an issue using a ConfigGroup with a Provider option.

I tried updating the Pulumi.Kubernetes package to use the latest alpha as it contains #2720 however the error persists.

The full error message:

Previewing update (dev):
     Type                                   Name                                       Plan       Info
 +   pulumi:pulumi:Stack                    awsx-eks-cluster-guestbook-csharp-dev      create     1 error
 +   ├─ kubernetes:yaml:ConfigGroup         guestbook                                  create     
 +   │  └─ kubernetes:yaml:ConfigFile       app/guestbook.yaml                         create     
 +   ├─ eks:index:Cluster                   cluster                                    create     
 +   │  ├─ eks:index:ServiceRole            cluster-eksRole                            create     
 +   │  │  ├─ aws:iam:Role                  cluster-eksRole-role                       create     
 +   │  │  └─ aws:iam:RolePolicyAttachment  cluster-eksRole-4b490823                   create     
 +   │  ├─ eks:index:ServiceRole            cluster-instanceRole                       create     
 +   │  │  ├─ aws:iam:Role                  cluster-instanceRole-role                  create     
 +   │  │  ├─ aws:iam:RolePolicyAttachment  cluster-instanceRole-3eb088f2              create     
 +   │  │  ├─ aws:iam:RolePolicyAttachment  cluster-instanceRole-03516f97              create     
 +   │  │  └─ aws:iam:RolePolicyAttachment  cluster-instanceRole-e1b295bd              create     
 +   │  ├─ eks:index:RandomSuffix           cluster-cfnStackName                       create     
 +   │  ├─ aws:iam:InstanceProfile          cluster-instanceProfile                    create     
 +   │  ├─ aws:ec2:SecurityGroup            cluster-eksClusterSecurityGroup            create     
 +   │  ├─ aws:ec2:SecurityGroupRule        cluster-eksClusterInternetEgressRule       create     
 +   │  ├─ aws:eks:Cluster                  cluster-eksCluster                         create     
 +   │  ├─ eks:index:VpcCni                 cluster-vpc-cni                            create     
 +   │  ├─ pulumi:providers:kubernetes      cluster-eks-k8s                            create     
 +   │  ├─ aws:ec2:SecurityGroup            cluster-nodeSecurityGroup                  create     
 +   │  ├─ kubernetes:core/v1:ConfigMap     cluster-nodeAccess                         create     
 +   │  ├─ aws:ec2:SecurityGroupRule        cluster-eksExtApiServerClusterIngressRule  create     
 +   │  ├─ aws:ec2:SecurityGroupRule        cluster-eksNodeIngressRule                 create     
 +   │  ├─ aws:ec2:SecurityGroupRule        cluster-eksClusterIngressRule              create     
 +   │  ├─ aws:ec2:SecurityGroupRule        cluster-eksNodeInternetEgressRule          create     
 +   │  ├─ aws:ec2:SecurityGroupRule        cluster-eksNodeClusterIngressRule          create     
 +   │  ├─ aws:ec2:LaunchConfiguration      cluster-nodeLaunchConfiguration            create     
 +   │  ├─ aws:cloudformation:Stack         cluster-nodes                              create     
 +   │  └─ pulumi:providers:kubernetes      cluster-provider                           create     
 +   └─ pulumi:providers:kubernetes         eks-provider                               create     

Diagnostics:
  pulumi:pulumi:Stack (awsx-eks-cluster-guestbook-csharp-dev):
    error: Running program '/Users/d/github/desteves/pulumi-hugo-fork/themes/default/static/programs/awsx-eks-cluster-guestbook-csharp/bin/Debug/net6.0/awsx-eks-cluster-guestbook-csharp.dll' failed with an unhandled exception:
    System.InvalidOperationException: This operation cannot be performed on a default instance of ImmutableArray<T>.  Consider initializing the array, or checking the ImmutableArray<T>.IsDefault property.
       at IEnumerator<T> System.Collections.Immutable.ImmutableArray<T>.System.Collections.Generic.IEnumerable<T>.GetEnumerator(?)
       at bool System.Linq.Enumerable+SelectManySingleSelectorIterator<TSource, TResult>.MoveNext()
       at TResult[] System.Linq.Enumerable+SelectEnumerableIterator<TSource, TResult>.ToArray()
       at TSource[] System.Linq.Enumerable.ToArray<TSource>(IEnumerable<TSource> source)
       at ImmutableArray<T> System.Collections.Immutable.ImmutableArray.CreateRange<T>(IEnumerable<T> items)
       at ImmutableArray<TSource> System.Collections.Immutable.ImmutableArray.ToImmutableArray<TSource>(IEnumerable<TSource> items)
       at Output<ImmutableDictionary<string, KubernetesResource>> Pulumi.Kubernetes.Yaml.Parser.ParseYamlDocument(ParseArgs config, CustomResourceOptions options)+(ImmutableArray<ImmutableDictionary<string, object>> objs) => { } [0]
       at async Task<OutputData<U>> Pulumi.Output<T>.ApplyHelperAsync<U>(Task<OutputData<T>> dataTask, Func<T, Output<U>> func) x 2
       at async Task<SerializationResult> Pulumi.Deployment.SerializeFilteredPropertiesAsync(string label, IDictionary<string, object> args, Predicate<string> acceptKey, bool keepResources, bool keepOutputValues)
       at async Task<OutputData<U>> Pulumi.Output<T>.ApplyHelperAsync<U>(Task<OutputData<T>> dataTask, Func<T, Output<U>> func) x 2
       at async Task<OutputData<ValueTuple<T1, T2, T3, T4, T5, T6, T7, ValueTuple<T8>>>> Pulumi.Output<T>.TupleHelperAsync<T1, T2, T3, T4, T5, T6, T7, T8>(Input<T1> item1, Input<T2> item2, Input<T3> item3, Input<T4> item4, Input<T5> item5, Input<T6> item6, Input<T7> item7, Input<T8> item8)+GetData(?)
       at async Task<OutputData<ValueTuple<T1, T2, T3, T4, T5, T6, T7, ValueTuple<T8>>>> Pulumi.Output<T>.TupleHelperAsync<T1, T2, T3, T4, T5, T6, T7, T8>(Input<T1> item1, Input<T2> item2, Input<T3> item3, Input<T4> item4, Input<T5> item5, Input<T6> item6, Input<T7> item7, Input<T8> item8)
       at async Task<OutputData<U>> Pulumi.Output<T>.ApplyHelperAsync<U>(Task<OutputData<T>> dataTask, Func<T, Output<U>> func) x 2
       at async Task<OutputData<ValueTuple<T1, T2, T3, T4, T5, T6, T7, ValueTuple<T8>>>> Pulumi.Output<T>.TupleHelperAsync<T1, T2, T3, T4, T5, T6, T7, T8>(Input<T1> item1, Input<T2> item2, Input<T3> item3, Input<T4> item4, Input<T5> item5, Input<T6> item6, Input<T7> item7, Input<T8> item8)+GetData(?)
       at async Task<OutputData<ValueTuple<T1, T2, T3, T4, T5, T6, T7, ValueTuple<T8>>>> Pulumi.Output<T>.TupleHelperAsync<T1, T2, T3, T4, T5, T6, T7, T8>(Input<T1> item1, Input<T2> item2, Input<T3> item3, Input<T4> item4, Input<T5> item5, Input<T6> item6, Input<T7> item7, Input<T8> item8)
       at async Task<OutputData<U>> Pulumi.Output<T>.ApplyHelperAsync<U>(Task<OutputData<T>> dataTask, Func<T, Output<U>> func) x 2
       at async Task<OutputData<object>> Pulumi.Output<T>.Pulumi.IOutput.GetDataAsync()
       at async Task<object> Pulumi.Serialization.Serializer.SerializeAsync(string ctx, object prop, bool keepResources, bool keepOutputValues)
       at async Task<RawSerializationResult> Pulumi.Deployment.SerializeFilteredPropertiesRawAsync(string label, IDictionary<string, object> args, Predicate<string> acceptKey, bool keepResources, bool keepOutputValues)
       at async Task<SerializationResult> Pulumi.Deployment.SerializeFilteredPropertiesAsync(string label, IDictionary<string, object> args, Predicate<string> acceptKey, bool keepResources, bool keepOutputValues)
       at async Task<Struct> Pulumi.Deployment.SerializeAllPropertiesAsync(string label, IDictionary<string, object> args, bool keepResources, bool keepOutputValues)
       at async Task Pulumi.Deployment.RegisterResourceOutputsAsync(Resource resource, Output<IDictionary<string, object>> outputs)

Example

Program.cs File:

using System.Collections.Generic;
using Eks = Pulumi.Eks;
using Kubernetes = Pulumi.Kubernetes;
using K8sCore = Pulumi.Kubernetes.Core.V1;
using K8sYaml = Pulumi.Kubernetes.Yaml;
using Aws = Pulumi.Aws;
using Pulumi;

return await Deployment.RunAsync(() =>
{
    // Create an EKS cluster with the default configuration.
    var cluster = new Eks.Cluster("cluster");

    var eksProvider = new Kubernetes.Provider("eks-provider", new()
    {
        KubeConfig = cluster.KubeconfigJson,
    });

    // Create resources from standard Kubernetes guestbook YAML.
    var guestbook = new K8sYaml.ConfigGroup("guestbook",
        new K8sYaml.ConfigGroupArgs
        {
            Files = new[] { "app/guestbook.yaml" },
        },
        new ComponentResourceOptions
        {
            Provider = eksProvider   /// error when I uncomment this line...
        });

    var service = guestbook.GetResource<K8sCore.Service>("frontend");
    
    var frontendIp = service.Apply(svc =>
        svc.Status.Apply(status =>
     {
         var ingress = status.LoadBalancer.Ingress[0];
         return ingress.Ip ?? ingress.Hostname;
     }));

    return new Dictionary<string, object?>
    {
        // Export the (cluster-private) IP address of the Guestbook frontend.
        ["FrontendIp"] = frontendIp,

    };
});

Source for the
app/guestbook.yaml File

Output of pulumi about

pulumi about
CLI          
Version      3.100.0
Go Version   go1.21.5
Go Compiler  gc

Host     
OS       darwin
Version  14.2.1
Arch     arm64

Additional context

From Fraser regarding the error message:

I think this is the fun that invokes still run even if a provider isn't configured but there's nothing to run so they just return null but none of the deserialisers can really do anything to deal with that so you get errors likes this

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[mjeffryes]

Internal discussion: https://pulumi.slack.com/archives/CPKMG4L3Z/p1704422635365439
Sounds like there's kind of a core bug, where we should, in general, present a better error when there's no provider configured. But possibly ConfigGroup could be updated to catch this specific problem as well.

[mjeffryes]

I'm calling this a "panic" because it's an error that spills our internally inconsistent state.

[EronWright]

I believe the issue to be a bug with how the SDK uses immutable arrays. One must check ImmutableArray.IsDefault to see whether it is initialized before using any methods including any on IEnumerable. Either the YAML parsing code should check IsDefault, and/or the core invoke code should return an empty array rather than the default array.

Some key background information:

• Getting rid of ImmutableArray<T>.IsDefault ? dotnet/runtime#26829
• https://devblogs.microsoft.com/dotnet/immutable-collections-are-now-rc/

To reproduce, I simply set the provider's KubeConfig property to an unknown value.

    var conf = new RandomPet("conf", new RandomPetArgs{});

    var eksProvider = new Kubernetes.Provider("eks-provider", new()
    {
        KubeConfig = conf.Id,
    });

[EronWright]

The deeper issue is that Pulumi skips over invokes during preview, and here's an open issue about that: pulumi/pulumi#10209

We can still fix the kubernetes provider without a pu/pu change, by detecting that the invoke was skipped and returning an Unknown value. The scope of the fix will be the dotnet SDK.