Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

C# crash: "HashSet capacity is too big", "System.NullReferenceException" and other HashSet errors when deploying Helm charts #1311

Closed
Aaronontheweb opened this issue Sep 10, 2020 · 3 comments
Closed

C# crash: "HashSet capacity is too big", "System.NullReferenceException" and other HashSet errors when deploying Helm charts #1311

Aaronontheweb opened this issue Sep 10, 2020 · 3 comments

Comments

@Aaronontheweb
Copy link

Problem description

When deploying the following services via Helm charts (on Azure Kubernetes Service, hence why I'm using Azure Active Directory) using Pulumi 2.9.2

/*
             * Need to configure managed Pod identities using Helm
             */

            var aadPodIdentity = new Chart("aad-pod-identity", new ChartArgs
            {
                Repo = "aad-pod-identity",
                Chart = "aad-pod-identity",
                Version = "1.6.0",
                //FetchOptions = new ChartFetchArgs
                //{
                //    Repo = "https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts/"
                //},
            }, new ComponentResourceOptions()
            {
                Provider = k8sProvider,
                Protect = IsProtected
            });

I run into a NullReferenceException during the deployment:

Running program 'D:\Repositories\sdkbin\sdkbin-azure\azure\bin\Debug\netcoreapp3.1\Azure.Deploy.dll' failed with an unhandled exception:
    System.NullReferenceException: Object reference not set to an instance of an object.
       at System.Collections.Generic.HashSet`1.AddIfNotPresent(T value)
       at System.Collections.Generic.HashSet`1.Add(T item)
       at Pulumi.Resource..ctor(String type, String name, Boolean custom, ResourceArgs args, ResourceOptions options)
       at Pulumi.CustomResource..ctor(String type, String name, ResourceArgs args, CustomResourceOptions options)
       at Pulumi.Kubernetes.KubernetesResource..ctor(String type, String name, DictionaryResourceArgs args, CustomResourceOptions options)
       at Pulumi.Kubernetes.Core.V1.Secret..ctor(String name, ImmutableDictionary`2 dictionary, CustomResourceOptions options)
       at Pulumi.Kubernetes.Yaml.Parser.<>c__DisplayClass3_0.<ParseYamlObject>b__55(String id)
       at Pulumi.Output`1.<>c__DisplayClass12_0`1.<Apply>b__0(T t)
       at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
       at Pulumi.Output`1.AllHelperAsync(ImmutableArray`1 inputs)
       at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
       at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
       at Pulumi.Output`1.<TupleHelperAsync>g__GetData|20_0[T1,T2,T3,T4,T5,T6,T7,T8,X](Input`1 input)
       at Pulumi.Output`1.TupleHelperAsync[T1,T2,T3,T4,T5,T6,T7,T8](Input`1 item1, Input`1 item2, Input`1 item3, Input`1 item4, Input`1 item5, Input`1 item6, Input`1 item7, Input`1 item8)
       at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
       at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
       at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
       at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
       at Pulumi.Output`1.Pulumi.IOutput.GetDataAsync()
       at Pulumi.Serialization.Serializer.SerializeAsync(String ctx, Object prop)
       at Pulumi.Deployment.SerializeFilteredPropertiesAsync(String label, IDictionary`2 args, Predicate`1 acceptKey)
       at Pulumi.Deployment.SerializeAllPropertiesAsync(String label, IDictionary`2 args)
       at Pulumi.Deployment.RegisterResourceOutputsAsync(Resource resource, Output`1 outputs)
       at Pulumi.Deployment.Runner.<>c__DisplayClass7_0.<<WhileRunningAsync>g__HandleCompletion|0>d.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at Pulumi.Deployment.Runner.WhileRunningAsync()

We did not have this issue on this same branch two weeks ago - so I suspect it's related to one of the underlying NuGet packages being upgraded. The error is also intermittent - doesn't always happen.

Errors & Logs

Affected product version(s)

Pulumi 2.9.2
Pulumi.Kubernetes 2.6.0

Reproducing the issue

Suggestions for a fix

Given the intermittent error of these issues, I'd look for a race condition in the code that modifies the HashSet<T>, since it appears that the code writes to it before it exists.
@mikhailshilkov
Copy link
Member

This is probably the line that fails: https://github.com/pulumi/pulumi/blob/848a98d5112ec93239501fd66b51c9f1f30353ac/sdk/dotnet/Pulumi/Resources/Resource.cs#L179

I would expect resources to be created sequentially, so I'm surprised to see a concurrency issue.

Where does "HashSet capacity is too big" from the issue title come from?

@Aaronontheweb
Copy link
Author

Here's the stack trace for HashSet capacity too big:

 System.ArgumentException: HashSet capacity is too big.
       at System.Collections.Generic.HashSet`1.IncreaseCapacity()
       at System.Collections.Generic.HashSet`1.AddIfNotPresent(T value)
       at Pulumi.Resource..ctor(String type, String name, Boolean custom, ResourceArgs args, ResourceOptions options)
       at Pulumi.CustomResource..ctor(String type, String name, ResourceArgs args, CustomResourceOptions options)
       at Pulumi.Kubernetes.KubernetesResource..ctor(String type, String name, DictionaryResourceArgs args, CustomResourceOptions options)
       at Pulumi.Kubernetes.Rbac.V1Beta1.ClusterRole..ctor(String name, ImmutableDictionary`2 dictionary, CustomResourceOptions options)
       at Pulumi.Kubernetes.Yaml.Parser.<>c__DisplayClass3_0.<ParseYamlObject>b__84(String id)
       at Pulumi.Output`1.<>c__DisplayClass12_0`1.<Apply>b__0(T t)
       at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
       at Pulumi.Output`1.AllHelperAsync(ImmutableArray`1 inputs)
       at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
       at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
       at Pulumi.Output`1.<TupleHelperAsync>g__GetData|20_0[T1,T2,T3,T4,T5,T6,T7,T8,X](Input`1 input)
       at Pulumi.Output`1.TupleHelperAsync[T1,T2,T3,T4,T5,T6,T7,T8](Input`1 item1, Input`1 item2, Input`1 item3, Input`1 item4, Input`1 item5, Input`1 item6, Input`1 item7, Input`1 item8)
       at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
       at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
       at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
       at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
       at Pulumi.Output`1.Pulumi.IOutput.GetDataAsync()
       at Pulumi.Serialization.Serializer.SerializeAsync(String ctx, Object prop)
       at Pulumi.Deployment.SerializeFilteredPropertiesAsync(String label, IDictionary`2 args, Predicate`1 acceptKey)
       at Pulumi.Deployment.SerializeAllPropertiesAsync(String label, IDictionary`2 args)
       at Pulumi.Deployment.RegisterResourceOutputsAsync(Resource resource, Output`1 outputs)
       at Pulumi.Deployment.Runner.<>c__DisplayClass7_0.<<WhileRunningAsync>g__HandleCompletion|0>d.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at Pulumi.Deployment.Runner.WhileRunningAsync()

Happened when we tried to deploy the following Helm chart, but the error is intermittent:

public static void DeploySeq(CustomResourceOptions k8sCustomResourceOptions, string seqDns)
        {
           /**
             * Enable Prometheus data storage on a persistent volume claim
             */
            var seqChartValues = new Dictionary<string, object>()
            {
                /* define Kubelet metrics scraping */
                ["ingress"] = new Dictionary<string, object>
                {
                    ["enabled"] = true,
                    ["annotations"] = new Dictionary<string, string>()
                    {
                        {"kubernetes.io/ingress.class", "azure/application-gateway"},
                        {"appgw.ingress.kubernetes.io/appgw-ssl-certificate", "sdkbin-ssl"}
                    },
                },
                ["ui"]  = new Dictionary<string, object>{
                    ["ingress"] = new Dictionary<string, object>
                    {
                        ["enabled"] = true,
                        ["hosts"] = new List<string>() { seqDns }
                    },                    
                },
                ["persistence"]  = new Dictionary<string, object>{
                    ["enabled"] = true,
                    ["size"] = "15Gi"
                },
                ["cache"]  = new Dictionary<string, object>{
                    ["targetSize"] = "0.2" // use a smaller cache size https://hub.helm.sh/charts/stable/seq
                },
            };

            var seqChart = new Chart("seq", new Pulumi.Kubernetes.Helm.ChartArgs
            {
                Chart = "seq",
                Version = "2.3.0",
                Namespace = "monitoring",
                Values = seqChartValues,
                FetchOptions = new ChartFetchArgs
                {
                    Repo = "https://kubernetes-charts.storage.googleapis.com/"
                },
            }, new ComponentResourceOptions()
            {
                Provider = k8sCustomResourceOptions.Provider,
            });
        }

@mikhailshilkov mikhailshilkov mentioned this issue Nov 12, 2020
Closed
@mikhailshilkov
Copy link
Member

This will be fixed in the next release (and is fixed in the dev build)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Aaronontheweb @mikhailshilkov