Client Throws 'Error while copying content to a stream' on 401 API Response

[oatsoda]

When making a call to SendGridClient.SendEmailAsync using a deliberately invalid API Key, rather than receiving a Response with a StatusCode of 401 Unauthorized, an exception is thrown:

System.Net.Http.HttpRequestException occurred
  HResult=0x80131620
  Message=Error while copying content to a stream.
  Source=<Cannot evaluate the exception source>
  StackTrace:
   at System.Net.Http.HttpContent.<LoadIntoBufferAsyncCore>d__49.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__58.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at SendGrid.SendGridClient.<MakeRequest>d__23.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at SendGrid.SendGridClient.<RequestAsync>d__24.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at SendGrid.SendGridClient.<SendEmailAsync>d__25.MoveNext()

Technical details:

• sendgrid-csharp Version: 9.9.0
• Azure Function running Net462

Looking at the data received from SendGrid, this may or may not be due to the fact that the Content-Length header is > 0 but there is no response content:

[oatsoda]

Upon further investigation it appears that this is somehow linked to the original content-length of the POST request.

If the original content is small (i.e. less in the email body) then I get the correct response content:

{
    "errors": [
        {
            "message": "The provided authorization grant is invalid, expired, or revoked",
            "field": null,
            "help": null
        }
    ]
}

[thinkingserious]

Hi @oatsoda,

That's an interesting bug, thanks for sharing it with us!

I've added this to our backlog for further investigation.

With Best Regards,

Elmer

[oatsoda]

@thinkingserious Thanks. I have also opened a support ticket with SendGrid directly regarding the lack of content returned as it may be an API issue rather than a client issue. I will update here once I receive a response.

[oatsoda]

Source.zip
SendGridTest.zip

Attached are Source and Binaries of a test app demonstrating the failure. I really suspect this is an API issue rather than a SendGridClient issue, but updating it here anyway.

Run the app with a value to repeat the lines in the email body. Once you exceed 56 it gets the exception. Tested on multiple machines and also from an Azure VM.

[oatsoda]

@thinkingserious SendGrid Support have sent me your way :)

It looks like you've been working with one of our Engineers via the github repository. He is definitely your best resource for this particular issue. This bug falls a bit outside of the scope of Support, but regardless of whether it's client related or API related, the engineer you were in contact with via GitHub should be able to assist.

Any ideas on what the issue could be? Can you reproduce it using my test app?

[thinkingserious]

Hello @oatsoda,

Thank you for the updated information. This issue is still working its way up our backlog.

In the mean time, you may want to try the API call directly to officially rule out the client as the culprit.

[oatsoda]

Hi @thinkingserious

Yes, I can reproduce it directly to the API in Postman too. However, I sent SendGrid Support the sample Postman call and it didn't reproduce for them! I suspect this is API related rather than client related. The API just doesn't return any content if the original request is above a certain size. I'd be really interested to know if the sample app reproduces it for you - I have tried on multiple machines in multiple geographic locations and it reproduces it each time. Quite why it didn't work via Postman for SendGrid Support I don' know - perhaps a network issue which isn't occurring inside the SendGrid network? (I vaguely remember coming across something similar years ago regarding MTU sizes on network devices and responses being split over multiple packets being lost - but I think that is clutching at straws!)

[Jericho]

I can confirm some of the observations that @oatsoda made regarding this issue. Please note that I used StrongGrid instead of SendGrid's C# client in order help determine if this issue is specific to the client or not:

Observation 1:
Using the same email content included in @oatsoda's sample code (including the large text and HTML content), I observed that the response from SendGrid's API is HTTP/1.1 401 Unauthorized (this is expected since the sample code is intentionally using a bogus api key). I also observe that the content of the response is empty. Finally, I observe that the response contains the following header: Content-Length: 116 which is contradictory with the fact that the response content is empty. This points to a server-side issue.

Observation 2:
Using the email content included in @oatsoda's sample code with much smaller text and HTML content, I observed that the response from SendGrid's API is HTTP/1.1 401 Unauthorized and the content of the response is {"errors":[{"message":"The provided authorization grant is invalid, expired, or revoked","field":null,"help":null}]} which is 116 characters long and is consistent with the Content-Length: 116 header that I also observe in this scenario. There is no issue here, everything is working as expected.

Observation 3:
I am unable to reproduce the Error while copying content to a stream exception that @oatsoda experienced when the response is empty. Since the only difference between my setup and his is that I am using an alternate C# client, this leads me to conclude that this issue is specific to SendGrid's C# client.

Conclusion:
There seems to be two distinct issues:

1. The server is not returning the expected content with the response in the scenario described in observation 1 (combined with the fact that it returns incorrect Content-Length)
2. The client is not properly handling situations where the body of the response is empty and/or relying on the incorrect Content-Length to determine if the body of the response contains any content.

[oatsoda]

Thanks @Jericho for confirming!

Strange that the error doesn't occur in other clients. The stack trace indicates that it is an error in HttpClient.FinishSendAsyncBuffered - but I suppose it could be that other clients may not use this...

[oatsoda]

@thinkingserious Any confirmation of an API problem regarding the empty response? Causing me quite a lot of grief as authentication failures not being detected.

I suspect the secondary issue regarding the client side exception is actually just net core http client expecting responses to be well formed - hence relying on content-length.

[thinkingserious]

@oatsoda,

I can check on that for you. What's the support ticket number?

[oatsoda]

@thinkingserious The number was #1331298, but it is closed because they replied with:

It looks like you've been working with one of our Engineers via the github repository. He is definitely your best resource for this particular issue. This bug falls a bit outside of the scope of Support, but regardless of whether it's client related or API related, the engineer you were in contact with via GitHub should be able to assist.

Hence I am asking you for an update :)

[thinkingserious]

Ah thanks @oatsoda,

It looks like there are 2 issues here as Jericho has illustrated. Issue #2 is something I can attempt to fix and so I'll leave this open for a fix. For issue #1, I'll ping the relevant engineering team to see if it's on deck for a fix.

Our backlog is pretty full at the moment, so in order for this to gain some priority, additional +1's or a PR helps greatly.

With Best Regards,

Elmer

[oatsoda]

@thinkingserious

Can you give me details about what the actual problem in part 1 is - it might be something I can assist with?

Regarding part 2 do you know if they have found the issue?

[thinkingserious]

@oatsoda,

There is an internal bug happening here where the API is not returning the correct content-length. I'm going to circle back with product and engineering to see where we are on a fix. Thanks!

[janus007]

I've had the same error after moving a website to Azure.

I fixed it by changing SendGridClient.cs, MakeRequest method.

Instead of:

Line 250: 
HttpResponseMessage response = await client.SendAsync(request, cancellationToken).ConfigureAwait(false);

to

HttpResponseMessage response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead).ConfigureAwait(false);

[thinkingserious]

Hi @janus007,

Thanks for sharing this solution, you are awesome! I will look into it and see if I can confirm the solution.

With Best Regards,

Elmer

[DanielStout5]

Any updates on this? I have just experienced this same issue with version 9.9.0 of the SendGrid client library - wasted a lot of time trying to figure out what was going on before I realized I had just accidentally removed a character from the API key!

[thinkingserious]

Not yet @DanielStout5, but I appreciate you taking the time to add a comment.

Each +1/comment adds priority to issues in our backlog.

[tony-ho]

I can take this one as part of #749 as this error causes integration test failures for me when running against Prism 2.x.

[contactsamie]

+1 same stacktrace @AmbroiseCouissin

We ended up using just straight SMTP due the reliability issues of this library.

would you like to share your solution?

[sebandres]

+1 same stacktrace @AmbroiseCouissin
We ended up using just straight SMTP due the reliability issues of this library.

would you like to share your solution?

@contactsamie I believe he means this: https://sendgrid.com/docs/for-developers/sending-email/getting-started-smtp/

[sebandres]

I also got here due to this cryptic error and in my case it turned out to be a missing variable for the Api Key in the CI/CD pipeline that resulted into the same message when trying to send an email without an ApiKey.

[crowbarsolutions]

Getting this intermittently as well, our application is configured to retry failed requests. We have noticed that in cases where this error gets thrown, and an identical request is attempted immediately after the failure, the request will work properly. Everything on the client side remains the same, it's just the response from the server that is the culprit.

[jwisener]

@thinkingserious Geeze, 2 years and it's clearly a bug. So are you saying that in order to work on a "bug" you treat it like "new functionality". I wish we could do that. When there is a production bug, we make a priority to hotfix it as soon as possible.

[andre-castro-garcia]

I had exact same error when running my application with invalid public IP inside an AKS cluster (inside a docker container obviously).

To fix this issue I needed:

1. Add a static public IP in my AKS cluster (https://docs.microsoft.com/azure/aks/egress);
2. Add this public IP in Sendgrid IP whitelist;

Now everything works!!!!!!! 😄

[dhtatsch]

We have the same problem with our azure application here. We opened a support call, but they say it's a "connection" issue.

[tedvanderveen]

Just like @janus007 reported before, the issue arises when sending a request that is big and chopped into chunks over the wire and miss out any important response status codes/headers. You can fix it like this:
tedvanderveen@eb100f3
See also https://docs.microsoft.com/en-us/uwp/api/windows.web.http.httpcompletionoption#fields

[oznurco]

I use sendgrid v9.12.0 in my application and I have the same problem(Error while copying content to a stream). I can see that the problem is too old, have you any idea about that when the problem will be solved?
Also, the sendgrid worked fine on locally, the problem occured in the published application.

[lurumad]

Hi @oznurco

Same error for me in Azure Function v3 even in local.

Regards!

[oznurco]

Hi @lurumad . I solved my problem. SendGrid keys were wrong in the production environment.

[chriswill]

This is really sad that this error so obscures the actual error.

My html body is only 6349 bytes. Only when I turned off the html body entirely did I get to see the actual error, which was:

{"errors":[{"message":"The requestor's IP Address is not whitelisted","field":null,"help":null}]}

I had to chase through non-relevant logs for hours for what should have been a simple fix.

[AndreduToit]

I also experienced the "Error while copying content to a stream." exception on incorrect key. Not a big issue to handle it but the exception is not really helpful. Still the same bug after what looks like about 32 months - not good.

[childish-sambino]

Similar to sendgrid/sendgrid-java#259 and sendgrid/sendgrid-java#641

Closing this as a non-library issue since it cannot/should not be remedied in client-side code. I've opened an internal ticket for tracking (reference ID: CL-2667). Please reference that ticket when contacting support regarding this issue.

Note that support and backend product teams do not use GitHub to prioritize work. Contacting support is the best way to push this issue along.

[oatsoda]

@childish-sambino

• There was already an support request for this.

• If you see this post, there is technically a weakness in the library too.

[childish-sambino]

@oatsoda This is something the needs to be handled via support so I recommend opening a new ticket there. The engineers that manage these libraries do not handle backend issues and support is the path forward.

For the item related to this library handling invalid API responses, I'm not convinced this library should be doing something different. It would basically be just failing in a different way. If the backend issue is resolved and a proper response is returned then there would be no special error handling here.

[AndreduToit]

Oh, I see but disagree. Getting a response of 'Error while copying content to a stream' when the failure is an invalid API – well this response does not help any developer. From: childish-sambino [mailto:notifications@github.com] Sent: 02 September 2020 23:19 To: sendgrid/sendgrid-csharp <sendgrid-csharp@noreply.github.com> Cc: AndreduToit <dutoita1011@gmail.com>; Comment <comment@noreply.github.com> Subject: Re: [sendgrid/sendgrid-csharp] Client Throws 'Error while copying content to a stream' on 401 API Response (#648) @oatsoda <https://github.com/oatsoda> This is something the needs to be handled via support so I recommend opening a new ticket there. The engineers that manage these libraries do not handle backend issues and support is the path forward. For the item related to this library handling invalid API responses, I'm not convinced this library should be doing something different. It would basically be just failing in a different way. If the backend issue is resolved and a proper response is returned then there would be no special error handling here. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#648 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHOBDK2OV3IM34ZCGYSMOFDSDZO5RANCNFSM4EJCAKXQ> .

[AndreduToit]

Oh, I see but respectfully disagree. Getting a response of 'Error while copying content to a stream' when the failure is an invalid API – well this response does not help any developer. The one thing that the library may do differently is to provide a response which would aid sendgrid developer customers to track down their mistakes more efficiently. Maybe something along the lines of "Error - API key ......"?

[childish-sambino]

We use the standard HttpClient which is where the exception emanates from. Adding logic/handling for issues where the response is not valid (e.g., content length header does not match actual content length) is better suited there. I suspect the reason this was not seen in the other repo is because of the
Pathoschild/FluentHttpClient client wrapper.

Still, given that this is an API-level bug that affects multiple libraries/languages, I don't see creating workarounds in multiple places a viable solution for handling API bugs regarding large payloads and bad API keys.

[joanna1010]

I can't believe this is still an issue after 3 years, and there's no clear documentation on how to fix this one.
We are a paid customer, and the PO has escalated to suggest switching the integration.
Also if you can't support/maintain your API, please do not open one. just waste people's time and money.

[joanna1010]

HI we have submitted a support ticket and they asked us to open a github issue here.
Would you please unify your response?

@oatsoda This is something the needs to be handled via support so I recommend opening a new ticket there. The engineers that manage these libraries do not handle backend issues and support is the path forward.

For the item related to this library handling invalid API responses, I'm not convinced this library should be doing something different. It would basically be just failing in a different way. If the backend issue is resolved and a proper response is returned then there would be no special error handling here.

[nmg196]

Still getting this problem in 2021. I don't understand why the client cannot correctly process the result and expose the real error rather than just throwing an obscure exception. I can see in Postman that a perfectly valid 401 Unauthorised response comes back (due to non whitelisted IP), except instead of just setting IsSuccessStatusCode to false it throws an unhelpful exception.