SendMessage returns null

[gwynjudd]

var twilio = new TwilioRestClient(twilioAccountSID, twilioAuthToken);
var message = twilio.SendMessage(twilioFromPhoneNumber, twilioToPhoneNumber, 
  subjectText + Environment.NewLine + messageText);

On one of our machines, the message result is returned as null. This seems to be due to the security certificate for the server "https://api.twilio.com/" not being trusted.

I am using Twilio.Api version 3.4.1.0 and RestSharp version 104.4.0.0.

See also:
http://stackoverflow.com/questions/14614335/twilios-twiliorestclient-sendsmsmessage-returns-null

[gwynjudd]

[gwynjudd]

I fixed the certificate error on that server, but the issue still persists. How can I debug this issue? Is it due to some kind of transport failure (e.g. networking issue?)

[devinrader]

It could be a transport issue. RestSharp eats transport errors, so its hard to debug them. I'd suggest breaking out a tool like Fiddler so you can watch the actually HTTP requests being made by the library to Twilio. This should give you some insite into whether there is a connectivity issue. I'd love to know if you find anything there so I can work on better surfacing these kinds of issues.

[gwynjudd]

Thanks @devinrader, I'm trying to use fiddler, but I can't see any requests relating to the twilio webservice in the tool - is there anything special I have to to in my code to allow the api to go through the fiddler proxy?

[devinrader]

What OS are you running? Do you see requests in Fiddler to other URL's? Are you trying to make a request to localhost?

There shouldn't be anything special you need in your code to make Fiddler work.

[gwynjudd]

It is Windows Server 2008 R2. I can see plenty of other requests in fiddler. I can also see requests to api.twilio.com if I just browse to it from within (say) chrome

[devinrader]

Are you loading the site you want to test using localhost? If you are you might read this:

http://docs.telerik.com/fiddler/observe-traffic/troubleshooting/notraffictolocalhost

[gwynjudd]

I don't think that could be the problem, on the one hand I can see plenty of requests to localhost in the fiddler output, and on the other hand isn't the twilio API trying to connect outbound to the server api.twilio.com? Aren't we interested in those requests?

Could it be because the API request goes via https? Would the API request bypass the fiddler proxy in that case?

[gwynjudd]

Sample fiddler output shown below at about the time of the issue. The highlighted request is when I went to "https://api.twilio.com" in the chrome browser.

[devinrader]

Its possible this is applicable to you:

http://rothmanshore.com/2011/07/13/debugging-use-fiddler-and-webproxy-to-debug-httpwebrequest-httpwebresponse-issues-in-asp-net/

If it is, you should be able to set the proxy on TwilioRestClient.

HTTPS requests should still be logged by Fiddler.

[gwynjudd]

I set the proxy in web.Config and now I can see a request for api.twilio.com at the point the issue happens. Based on this, could you see what is going wrong? Note there is only a single request shown

[gwynjudd]

By comparison, a successful request from my developer machine which does not have the issue has two requests:

[devinrader]

Based on the screenshots it seems like there is still an issue with the library hitting the server with with HTTPS. You should be seeing two requests like you see on your development machine. I've never see this issue before though.

The Twilio library uses RestSharp to make the HTTP requests so there certainly could be an issue in that library. RestSharp itself uses HttpWebRequest to actually make its request. I did a bit of googling for RestSharp and SSL and found a number of people talking about authentication failures. I wonder if thats what you are running into.

Depending on how far the rabbit hole you want to go I think you've got a couple of options all of which might end up surfacing the root cause:

1. Change your app to not use the Twilio library, but still use the RestSharp library to make the HTTP request. You could literally copy the source from here and just paste it into your app. This may let you surface an exception if its happening by looking at the RestResponse.Status property.

2. Change your app to just use the HttpWebRequest/HttpWebResponse classes (or better yet HttpClient if your using .NET 4 or newer) to call the Twilio API.

[gwynjudd]

I am trying to use the RestSharp API directly and came up against the following issue that "AsString()" in the following code is not defined:

var content = resp.RawBytes.AsString(); //get the response content

I presume it is an extension method, but I can't see where it is defined. For now I just went with this, which I thought was probably right:

UTF8Encoding enc = new UTF8Encoding();
var content = new UTF8Encoding().GetString(resp.RawBytes);

[devinrader]

I believe you can just add:

using RestSharp.Extensions;

[gwynjudd]

So I did some more work, and found that the HTTP StatusCode from the request is coming back as 0. From what I can tell, this should only happen if the request is cancelled before going anywhere, possibly some kind of firewall interference (http://stackoverflow.com/questions/3825581/does-an-http-status-code-of-0-have-any-meaning).

This is what I added:

    internal class TwilioSender
    {
        public string AccountSid { get; set; }
        public string AuthToken { get; set; }

        public virtual Message SendMessage(string from, string to, string body)
        {
            log("TwilioSender.SendMessage: " + from + " - " + to + " - " + body);

            var request = new RestRequest(Method.POST);
            request.Resource = "Accounts/{AccountSid}/Messages.json";
            request.AddParameter("From", from);
            request.AddParameter("To", to);

            request.AddParameter("Body", body);

            return Execute<Message>(request);
        }

        public virtual T Execute<T>(RestRequest request) where T : new()
        {
            log("TwilioSender.Execute");

            var _client = new RestClient();
            _client.UserAgent = "twilio-csharp/" + "3.4.1.0" + " (.NET " + Environment.Version.ToString() + ")";
            _client.Authenticator = new HttpBasicAuthenticator(AccountSid, AuthToken);

#if FRAMEWORK
_client.AddDefaultHeader("Accept-charset", "utf-8");
#endif

            _client.BaseUrl = string.Format("{0}{1}", "https://api.twilio.com/", "2010-04-01");
            _client.Timeout = 30500;

            // if acting on a subaccount, use request.AddUrlSegment("AccountSid", "value")
            // to override for that request.
            _client.AddDefaultUrlSegment("AccountSid", AccountSid);

            request.OnBeforeDeserialization = (resp) =>
            {
                log("TwilioSender.OnBeforeDeserialization: " + (int)resp.StatusCode);

                // for individual resources when there's an error to make
                // sure that RestException props are populated
                if (((int)resp.StatusCode) >= 400)
                {
                    // have to read the bytes so .Content doesn't get populated
                    string restException = "{{ \"RestException\" : {0} }}";
                    var content = resp.RawBytes.AsString(); //get the response content

                    log("Response content: " + content);

                    var newJson = string.Format(restException, content);

                    resp.Content = null;
                    resp.RawBytes = Encoding.UTF8.GetBytes(newJson.ToString());

                    if (resp.ErrorException != null)
                    {
                        log("Response exception: " + resp.ErrorException.ToString());
                    }

                    if (resp.ErrorMessage != null)
                    {
                        log("Response error message: " + resp.ErrorMessage);
                    }

                    if (resp.StatusDescription != null)
                    {
                        log("Response status description: " + resp.StatusDescription);
                    }

                    if (resp.Headers != null)
                    {
                        log("Headers...");

                        foreach (var param in resp.Headers)
                        {
                            log("Header: " + param.ToString());
                        }

                        log("Headers finished");
                    }
                }
            };

            request.DateFormat = "ddd, dd MMM yyyy HH:mm:ss '+0000'";

            var response = _client.Execute<T>(request);
            return response.Data;
        }
    }

...

                var sender = new TwilioSender();
                sender.AccountSid = twilioAccountSID;
                sender.AuthToken = twilioAuthToken;
                Message message = sender.SendMessage(twilioFromPhoneNumber, twilioToPhoneNumber, body);

                if (message == null)
                {
                    log("Twilio message was null");
                    return false;
                }

                if (message.RestException != null)
                {
                    Exception e = new Exception(message.RestException.Message);
                    e.HelpLink = message.RestException.MoreInfo;
                    e.Data.Add("restException.Code", message.RestException.Code);
                    e.Data.Add("restException.MoreInfo", message.RestException.MoreInfo);
                    e.Data.Add("restException.Status", message.RestException.Status);
                    e.Data.Add("restException.Message", message.RestException.Message);

                    ErrorSignal.FromCurrentContext().Raise(e);
                    return false;
                }

In my log, I saw the following:

TwilioSender.SendMessage ...
TwilioSender.Execute
TwilioSender.OnBeforeDeserialization: 0
Twilio message was null

[devinrader]

What happens if you move this code:

 if (resp.ErrorException != null)
{
      log("Response exception: " + resp.ErrorException.ToString());
}

Out of the if block, right after this line of code:

log("TwilioSender.OnBeforeDeserialization: " + (int)resp.StatusCode);

Is the ErrorException property populated?

[gwynjudd]

Hi Devin,

I tried this and got some interesting results. The first attempt, the message send actually succeeded. I don't know why this happened, I hadn't made any configuration or code changes that would make me expect it to have worked from that machine.

I tried it again and this time it failed as it had been doing last week. I got the following results in the log:

[gwynjudd]

I noticed something odd. On my developer machine, if I browse to api.twilio.com, I see the following certificate information:

On the other hand, on the machine with the problem, I see this information:

They are obviously different. Could this be the cause of the problem?

[devinrader]

Yes, it could. Is the FortiGate your corporate firewall? I don't know enough about firewalls to be able to give you much advice, but Twilios SSL cert should be a Thawte cert as your seeing on your dev machine.

[gwynjudd]

I'll have to do some investigation, the machine with the problem is in our India office so I'm not sure how they have it set up there.

[gwynjudd]

Hi Devin,

I have figured out that the problem is that the fortigate certificate was not trusted by the local machine account - although I had trusted it by the logged in user, IIS was running as a different user.

ref:
http://blogs.msdn.com/b/jpsanders/archive/2009/09/16/troubleshooting-asp-net-the-remote-certificate-is-invalid-according-to-the-validation-procedure.aspx?Redirected=true

By trusting the root certificate at the local machine account, it enabled the service to work correctly.

[gwynjudd]

In order to fix the issue where SendMessage is returning null, do you think this kind of thing would be appropriate to add? Just basically to drag out information regarding the network issue:

                if (((int)resp.StatusCode == 0))
                {
                    string networkException = "{{ \"NetworkException\" : {0} }}";
                    string content = null;

                    if (resp.ErrorException != null)
                    {
                        content = resp.ErrorException.ToString();
                    }

                    if (String.IsNullOrEmpty(content) && resp.ErrorMessage != null)
                    {
                        content = resp.ErrorMessage;
                    }

                    if (String.IsNullOrEmpty(content) && resp.StatusDescription != null)
                    {
                        content = resp.StatusDescription;
                    }

                    var newJson = string.Format(networkException, content);

                    resp.Content = null;
                    resp.RawBytes = Encoding.UTF8.GetBytes(newJson.ToString());
                }

...

And then in the calling code:

var client = new TwilioRestClient(twilioAccountSID, twilioAuthToken);
var message = twilio.SendMessage(twilioFromPhoneNumber, twilioToPhoneNumber, body);

if (message.NetworkException != null)
{
  ...
}

Or would it "fit" in the existing RestException?

[devinrader]

Possibly. Though for errors like transport and network exceptions (eg DNS resolution failure, request time out, etc) I'd prefer to find a way to just bubble out an underlying exception if there is one instead of checking for that status code, since that's not really a status code.

When the error happens, are you seeing it get caught by RestSharp and exposed via the ErrorException property? IIRC that should be getting populated if things like transport or network errors cause HttpWebRequest to throw an exception.

Either here if its bubble out as a WebException:

Sync: https://github.com/restsharp/RestSharp/blob/master/RestSharp/Http.Sync.cs#L196-L213
Async: https://github.com/restsharp/RestSharp/blob/master/RestSharp/Http.Sync.cs#L181-L191

or here is its any other exception:

Sync: https://github.com/restsharp/RestSharp/blob/master/RestSharp/Http.Sync.cs#L181-L191
Async: https://github.com/restsharp/RestSharp/blob/master/RestSharp/Http.Async.cs#L337-L355

The thing I may need to figure out is if checking for that exception in OnBeforeDeserialization is the right place for it.

[gwynjudd]

Hi Devin,

the ErrorException property is being populated - it is a WebException - just check the screenshot here (#126 (comment)). It looks like it would generally be populated in either of those two places

[devinrader]

OK, since this has somewhat turned into a different issue than SendMessage being null I'm going to close this and open up another tissue for exposing the underlying transport exceptions.