Grpc.Core.RpcException: "Failed to deserialize response message." when using "cert-manager.crds.yaml" for ConfigFile

[sfmskywalker]

When using the following code in my C# Pulumi project:

var customResourceDefinitions = new ConfigFile("cert-manager-crds", new ConfigFileArgs
{
    File = "https://github.com/jetstack/cert-manager/releases/download/v0.14.0/cert-manager.crds.yaml"
});

and do pulumi up, the process fails with:

Diagnostics:
  pulumi:pulumi:Stack (ya-infra-shared-dev):
    error: Running program 'C:\Projects\Ya\Shared\infra\ya-infra-shared\bin\Debug\netcoreapp3.1\Ya.Infra.Shared.dll' failed with an unhandled exception:
    Grpc.Core.RpcException: Status(StatusCode=Internal, Detail="Failed to deserialize response message.")
       at Pulumi.GrpcMonitor.InvokeAsync(InvokeRequest request)
       at Pulumi.Deployment.InvokeAsync[T](String token, InvokeArgs args, InvokeOptions options, Boolean convertResult)
       at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
       at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
       at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
       at Pulumi.Output`1.Pulumi.IOutput.GetDataAsync()
       at Pulumi.Serialization.Serializer.SerializeAsync(String ctx, Object prop)
       at Pulumi.Deployment.SerializeFilteredPropertiesAsync(String label, IDictionary`2 args, Predicate`1 acceptKey)
       at Pulumi.Deployment.SerializeAllPropertiesAsync(String label, IDictionary`2 args)
       at Pulumi.Deployment.RegisterResourceOutputsAsync(Resource resource, Output`1 outputs)
       at Pulumi.Deployment.Runner.WhileRunningAsync()

I'm trying to automate the installation of cert-manager as described here. One of its first steps mentions I should install some custom resource definitions separately.

Perhaps I should be using a different class to get the cert-manager.crds.yaml applied?

[lblackstone]

This worked using the NodeJS SDK, so it must be specific to .NET. Can you take a look @mikhailshilkov?

new k8s.yaml.ConfigFile("guestbook2",
    {
        file: "https://github.com/jetstack/cert-manager/releases/download/v0.14.0/cert-manager.crds.yaml"
    }
);

[mikhailshilkov]

Here is a minimal repro YAML to get the same exception:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
spec:
  validation:
    openAPIV3Schema:
      properties:
        spec:
          properties:
            solver:
              properties:
                http01:
                  properties:
                    ingress:
                      properties:
                        podTemplate:
                          properties:
                            spec:
                              properties:
                                affinity:
                                  properties:
                                    podAffinity:
                                      properties:
                                        preferredDuringSchedulingIgnoredDuringExecution:
                                          items:
                                            properties:
                                              podAffinityTerm:
                                                properties:
                                                  labelSelector:
                                                    properties:
                                                      matchExpressions:
                                                        items:
                                                          properties:
                                                            values:
                                                              type: array

The C# gRPC swallows the real exception, but I got it with a debugger:

Google.Protobuf.InvalidProtocolBufferException: Protocol message had too many levels of nesting.  May be malicious.  Use CodedInputStream.SetRecursionLimit() to increase the depth limit.
   at Google.Protobuf.CodedInputStream.ReadMessage(IMessage builder)
   at Google.Protobuf.FieldCodec.<>c__DisplayClass32_0`1.<ForMessage>b__0(CodedInputStream input)
   at Google.Protobuf.FieldCodec`1.Read(CodedInputStream input)
... (a very long call stack follows)

Unfortunately, I don't see a setting to configure the recursion limit in the gRPC library - the protobuf setting mentioned in the exception doesn't look available. I'll open an issue and ask there.

[mikhailshilkov]

The issue in gRPC: grpc/grpc#22682

[sfmskywalker]

I was wondering if anyone could recommend an alternative approach to installing CertManager, even as a temporary workaround (other than manually invoking helm)?

[gitfool]

I've got the same problem using dotnet/c# with cert manager. This is a showstopper! 😬

[sam-cogan]

I'm seeing the same issue when using the option to install Cert-Manager CRD's as part of the Helm chart. So currently there is no way to install cert-manager with Pulumi, as far as I can see

[gitfool]

I'm still puzzled by why the above minimal repro yaml and cert-manager.yaml causes ParsingPrimitivesMessages.ReadMessage to throw with error "too many levels of nesting" since DefaultRecursionLimit is 100:

• minimal repro yaml has 32 nested elements
• cert manager yaml has at most 37 nested elements (e.g. line 9593)

(using yaml indentation as guide)

What am I missing?

[NArnott]

We just ran into this to, as part of setting up the AWS Load Balancer controller in our EKS environment. We are fully integrated with Pulumi and need a solution.

[lblackstone]

@mikhailshilkov Any ideas on a workaround here?

[mikhailshilkov]

I don't have any workarounds that wouldn't require a fix in our SDK. Options of how this could be resolved:

• Wait until C#: getting RecursionLimitExceeded with no way to configure grpc/grpc#22682 or Provide a way to configure message marshallers grpc/grpc-dotnet#1983 is addressed in some way
• Fork Google.Protobuf, bump the limit up, publish our fork as a separate NuGet package, use it instead of the official version
• Add an option to the invoke to "flatten" the result structure, implement some funky pointer-style flattening so that any levels beyond 30 are replaced with a pointer at the top level, and unflatten in .NET
• Stop using the invoke in .NET SDK and implement parsing YAML inside .NET. A quick prototype of this with YamlDotNet immediately failed to parse the original repro file (it didn't like --- in the file)

None of the options look particularly appealing but forking may be the least evil. Ideas?

[mikhailshilkov]

@gitfool

minimal repro yaml has 32 nested elements
What am I missing?

So, for every layer of yaml, ReadMessage method would be called three times: once with message being a Struct, once with message being MapField<string, Value>, and once with Value. I guess that's just how untyped dictionaries are represented in protobuf.

[gitfool]

@mikhailshilkov Given the latest response, I’d expect the upstream issues to take too long to wait, and we need a fix now, so I’d fork to increase the default recursion limit in the meantime.

[mikhailshilkov]

I’d fork to increase the default recursion limit in the meantime.

Is there the best practice for publishing forked NuGet packages? Any examples in the wild?

[gitfool]

I don’t know about best practice but in the past we’ve done it by publishing to an internal NuGet repository and configuring NuGet.config to reference the internal NuGet feed before the standard NuGet feed.

You could do the same but for a public repository and feed, which could be hosted on MyGet or GitHub packages etc. (Note: GitHub packages requires auth for private packages and I’m not sure if it requires auth for public packages, which would make it useless.)

[mikhailshilkov]

Before resorting to a fork, I tried changing YAML-parsing invokes to return a string of JSON rather than a raw untyped protobuf map.

This fixed the issue at invoke time but the same issue re-appeared at resource invocation time: each CRD in that YAML is only 1 level less deep than the YAML overall, so resource creation hits the same max recursion threshold. So, I hit

Grpc.Core.RpcException: Status(StatusCode=Internal, Detail="Failed to deserialize response message.")
       at Pulumi.GrpcMonitor.RegisterResourceAsync(Resource resource, RegisterResourceRequest request)
       at Pulumi.Deployment.RegisterResourceAsync(Resource resource, Boolean remote, Func`2 newDependency, ResourceArgs args, ResourceOptions options)
       at Pulumi.Deployment.ReadOrRegisterResourceAsync(Resource resource, Boolean remote, Func`2 newDependency, ResourceArgs args, ResourceOptions options)

[mikhailshilkov]

Here is the draft of the forking plan:

1. Fork protocolbuffers/protobuf to Pulumi org (done): https://github.com/pulumi/protobuf
2. Checkout the latest stable tag (v3.13.0.1 as of today).
3. Create a pulumi-fork branch.
4. Adjust DefaultRecursionLimit in two spots from 100 to 1000.
5. Adjust the csproj file to something like

   <GeneratePackageOnBuild>true</GeneratePackageOnBuild>
   <Description>Pulumi fork to override DefaultRecursionLimit - C# runtime library for Protocol Buffers - Google's data interchange format.</Description>
   <GeneratePackageOnBuild>true</GeneratePackageOnBuild>

6. Build the package.
7. Publish it under Pulumi organization in NuGet as Pulumi.Protobuf.
8. Update the reference in pulumi/pulumi to the fork.
9. Publish a new version of Pulumi NuGet.
10. Update Pulumi.Kubernetes to use it.

I ran this sequence locally, without any publishing, and it does fix the original problem. I am able to deploy the CRDs from https://github.com/jetstack/cert-manager/releases/download/v0.14.0/cert-manager.crds.yaml.

Any concerns about this proposal, technically or from the IP perspective?

[gitfool]

Sounds good. Only concern would be to ensure you follow any upstream updates.

Also, I thought Google.Protobuf was a transitive dependency not a direct dependency, in which case won’t the forked package name need to be the same?

[mikhailshilkov]

Thanks to everyone for waiting for so long: it took a while to investigate the issue, find an issue in the Protobuf .NET library, discuss it with maintainers, and finally come up with a workaround.

The issue is now fixed in the latest alpha release (dotnet add package Pulumi.Kubernetes --version 2.8.0-alpha.1605734708) and will be shipped in the next release. cc @lblackstone - we may want to cut 2.8.1 2.7.2 whenever that works for you.

[sfmskywalker]

That’s great news! Thanks for the effort that went into this.

[NArnott]

Awesome! Can't wait to get this built out.

[sam-cogan]

Thanks for sorting this @mikhailshilkov I can confirm it works as expected with the Alpha release, looking forward to getting it in once 2.8.1 is out!

[mikhailshilkov]

I think the next release is going to be 2.7.2 - I messed it up due to our alpha naming. I'll post an update here whenever the fix is released.

[lblackstone]

https://github.com/pulumi/pulumi-kubernetes/releases/tag/v2.7.2 is out now with the fix

[gitfool]

I also want to thank @mikhailshilkov for implementing the work around for this unfortunate issue with dotnet grpc! ❤️