Http Utils

Demis Bellot edited this page Oct 25, 2016 · 37 revisions


  1. Getting Started

    1. Creating your first project
    2. Create Service from scratch
    3. Your first webservice explained
    4. Example Projects Overview
    5. Learning Resources
  2. Designing APIs

    1. ServiceStack API Design
    2. Designing a REST-ful service with ServiceStack
    3. Simple Customer REST Example
    4. How to design a Message-Based API
    5. Software complexity and role of DTOs
  3. Reference

    1. Order of Operations
    2. The IoC container
    3. Configuration and AppSettings
    4. Metadata page
    5. Rest, SOAP & default endpoints
    6. SOAP support
    7. Routing
    8. Service return types
    9. Customize HTTP Responses
    10. Customize JSON Responses
    11. Plugins
    12. Validation
    13. Error Handling
    14. Security
    15. Debugging
    16. JavaScript Client Library (ss-utils.js)
  4. Clients

    1. Overview
    2. C#/.NET client
      1. .NET Core Clients
    3. Add ServiceStack Reference
      1. C# Add Reference
      2. F# Add Reference
      3. VB.NET Add Reference
      4. Swift Add Reference
      5. Java Add Reference
    4. Silverlight client
    5. JavaScript client
      1. Add TypeScript Reference
    6. Dart Client
    7. MQ Clients
  5. Formats

    1. Overview
    2. JSON/JSV and XML
    3. HTML5 Report Format
    4. CSV Format
    5. MessagePack Format
    6. ProtoBuf Format
  6. View Engines 4. Razor & Markdown Razor

    1. Markdown Razor
  7. Hosts

    1. IIS
    2. Self-hosting
    3. Messaging
    4. Mono
  8. Security

    1. Authentication
    2. Sessions
    3. Restricting Services
    4. Encrypted Messaging
  9. Advanced

    1. Configuration options
    2. Access HTTP specific features in services
    3. Logging
    4. Serialization/deserialization
    5. Request/response filters
    6. Filter attributes
    7. Concurrency Model
    8. Built-in profiling
    9. Form Hijacking Prevention
    10. Auto-Mapping
    11. HTTP Utils
    12. Dump Utils
    13. Virtual File System
    14. Config API
    15. Physical Project Structure
    16. Modularizing Services
    17. MVC Integration
    18. ServiceStack Integration
    19. Embedded Native Desktop Apps
    20. Auto Batched Requests
    21. Versioning
    22. Multitenancy
  10. Caching

  11. Caching Providers

  12. HTTP Caching

  13. CacheResponse Attribute

  14. Cache Aware Clients

  15. Auto Query

  16. Overview

  17. Why Not OData

  18. AutoQuery RDBMS

  19. AutoQuery Data

  20. AutoQuery Memory

  21. AutoQuery Service

  22. AutoQuery DynamoDB

  23. Server Events

    1. Overview
    2. JavaScript Client
    3. C# Server Events Client
    4. Redis Server Events
  24. Service Gateway

    1. Overview
    2. Service Discovery
  25. Encrypted Messaging

    1. Overview
    2. Encrypted Client
  26. Plugins

    1. Auto Query
    2. Server Sent Events
    3. Swagger API
    4. Postman
    5. Request logger
    6. Sitemaps
    7. Cancellable Requests
    8. CorsFeature
  27. Tests

    1. Testing
    2. HowTo write unit/integration tests
  28. ServiceStackVS

    1. Install ServiceStackVS
    2. Add ServiceStack Reference
    3. TypeScript React Template
    4. React, Redux Chat App
    5. AngularJS App Template
    6. React Desktop Apps
  29. Other Languages

    1. FSharp
      1. Add ServiceStack Reference
    2. VB.NET
      1. Add ServiceStack Reference
    3. Swift
    4. Swift Add Reference
    5. Java
      1. Add ServiceStack Reference
      2. Android Studio & IntelliJ
      3. Eclipse
  30. Amazon Web Services

  31. ServiceStack.Aws

  32. PocoDynamo

  33. AWS Live Demos

  34. Getting Started with AWS

  35. Deployment

    1. Deploy Multiple Sites to single AWS Instance
      1. Simple Deployments to AWS with WebDeploy
    2. Advanced Deployments with OctopusDeploy
  36. Install 3rd Party Products

    1. Redis on Windows
    2. RabbitMQ on Windows
  37. Use Cases

    1. Single Page Apps
    2. HTML, CSS and JS Minifiers
    3. Azure
    4. Connecting to Azure Redis via SSL
    5. Logging
    6. Bundling and Minification
    7. NHibernate
  38. Performance

    1. Real world performance
  39. Other Products

    1. ServiceStack.Redis
    2. ServiceStack.OrmLite
    3. ServiceStack.Text
  40. Future

    1. Roadmap
Clone this wiki locally

This page has moved to docs.servicestack.net/http-utils


The recommended way to call ServiceStack services is to use any of the C# Service Clients which have a nice DRY and typed API optimized for this use. However when doing server programming you will often need to consume 3rd Party HTTP APIs, unfortunately the built-in way to do this in .NET doesn't make for a good development experience since it makes use of WebRequest - one of the legacy classes inherited from the early .NET days. WebRequest is an example of a class that's both versatile but also suffers from exposing an out-dated and unpleasant API for your application code to bind to.

Creating a pleasant DRY API with just extension methods

Rather than taking the normal .NET approach of wrapping WebRequest inside a suite of proxy and abstraction classes, we prefer to instead encapsulate any unnecessary boilerplate behind extension methods DRYing common access patterns behind terse, readable and chained APIs without any loss of flexibility since the underlying WebRequest remains accessible whenever it's needed.

The PocoPower project shows some good examples of what this looks like in Practice. Here's how you can retrieve a typed list of GitHub User Repos from GitHub's JSON REST API:

List<GithubRepo> repos = $"https://api.github.com/users/{user}/repos"
    .GetJsonFromUrl()
    .FromJson<List<GithubRepo>>();

Url Extensions

You can make use of the accompanying String Extensions to programatically construct a url as seen in this Twitter API example:

var url = $"http://api.twitter.com/statuses/user_timeline.json?screen_name={name}";
if (sinceId != null)
    url = url.AddQueryParam("since_id", sinceId);
if (maxId != null)
    url = url.AddQueryParam("max_id", maxId);

var tweets = url.GetJsonFromUrl()
    .FromJson<List<Tweet>>();

In both these cases it uses WebRequest to make a HTTP GET request asking for the "application/json" Content-Type, that's preferably compressed with gzip or deflate encoding (if the remote web server supports it).

Alternative Content-Type

In addition to GetJsonFromUrl there's also GetXmlFromUrl covering the 2 widely used content-types used for data containers:

List<User> users = "http://example.org/xml-rpc/users"
    .GetXmlFromUrl()
    .FromXml<List<User>>();

For any other Content-Type you can specify it with the optional accept param:

var csv = "http://example.org/users.csv"
    .GetStringFromUrl(accept:"text/csv");

Customizing the WebRequest

Although most extension methods start on string urls, you can customize the HttpWebRequest used to make the request by specifying a requestFilter. e.g:

var json = "http://example.org/users".GetJsonFromUrl(requestFilter:webReq =>{
    webReq.Headers["X-Api-Key"] = apiKey;
});

Parsing Custom Responses

This also works for Response Filters as well where if you need access to the Response HTTP Headers as well as the body you can add a callback for the response filter:

List<GithubRepo> repos = "https://api.github.com/users/{0}/repos".Fmt(user)
    .GetJsonFromUrl(responseFilter: httpRes => {
        var remaining = httpRes.Headers["X-Api-Remaining"];
    })
    .FromJson<List<GithubRepo>>();

Downloading Raw Content

Use the GetStringFromUrl extension to download raw text:

string csv = "http://example.org/sales.csv".GetStringFromUrl();

and the GetBytesFromUrl extension to download raw bytes:

byte[] imgBytes = "http://example.org/photo.jpg".GetBytesFromUrl();

POSTing data

Another common HTTP Request is to POST data to REST APIs. The most common way to post data to HTTP APIs is to post x-www-form-urlencoded key value pairs which you can do with:

var response = "http://example.org/login"
    .PostToUrl("Username=mythz&Password=password");

Or using a POCO Type:

var response = "http://example.org/login"
    .PostToUrl(new Login { Username="mythz", Password="password" });

An Anonymous Type:

var response = "http://example.org/login"
    .PostToUrl(new { Username="mythz", Password="password" });

Or a string Dictionary:

var login = new Dictionary<string,string> { 
    {"Username","mythz"}, {"Password","password"} };
var response = "http://example.org/login".PostToUrl(login);

Or a NameValueCollection:

var login = new NameValueCollection { 
    {"Username","mythz"}, {"Password","password"} };
var response = "http://example.org/login".PostToUrl(login.ToFormUrlEncoded());

Although POST'ing other Content-Types are also easily supported. An example using JSON:

Either as any serializable JSON object:

var response = "http://example.org/login"
    .PostJsonToUrl(new Login { Username="mythz", Password="password" });

Or as a raw JSON string:

var response = "http://example.org/login"
    .PostJsonToUrl(@"{""Username"":""mythz"",""Password"":""p@ssword""}");

And an example of sending any other arbitrary content types:

var response = "http://example.org/login"
  .PostStringToUrl("<User>mythz</User><Pass>p@ss</Pass>", contentType:"application/xml");

The above API's also apply to PUT or PATCH data as well by using the PutToUrl and PatchToUrl extension methods.

The same HTTP Utils extension methods for Post and Put also have Patch() equivalents.

PATCH Example

We use the new Patch() support in Gistlyn's GitHubServices.cs to update contents of existing Gists:

var GithubApiBaseUrl = "https://api.github.com/";
var updateResponse = GithubApiBaseUrl.CombineWith("gists", gist)
    .PatchJsonToUrl(new UpdateGithubGist {
        description = request.Description,
        files = request.Files,
    }, 
    requestFilter: req => {
        req.UserAgent = "Gistlyn";
        req.Headers["Authorization"] = "token " + github.AccessTokenSecret;
    });

Creating a Proxy using HTTP Utils

As the HTTP Utils offers a flexible API it becomes trivial to create a generic HTTP Proxy which you can implement with the ServiceStack Service below:

[Route("/proxy")]
public class Proxy : IRequiresRequestStream, IReturn<string>
{
    public string Url { get; set; }
    public Stream RequestStream { get; set; }
}

public object Any(Proxy request)
{
    if (string.IsNullOrEmpty(request.Url))
        throw new ArgumentNullException("Url");

    var hasRequestBody = base.Request.Verb.HasRequestBody();
    try
    {
        var bytes = request.Url.SendBytesToUrl(
          method: base.Request.Verb,
          requestBody: hasRequestBody ? request.RequestStream.ReadFully() : null,
          contentType: hasRequestBody ? base.Request.ContentType : null,
          accept: ((IHttpRequest)base.Request).Accept,
          requestFilter: req => req.UserAgent = "Gistlyn",
          responseFilter: res => base.Request.ResponseContentType = res.ContentType);

        return bytes;
    }
    catch (WebException webEx)
    {
        var errorResponse = (HttpWebResponse)webEx.Response;
        base.Response.StatusCode = (int)errorResponse.StatusCode;
        base.Response.StatusDescription = errorResponse.StatusDescription;
        var bytes = errorResponse.GetResponseStream().ReadFully();
        return bytes;
    }
}

Async HTTP Utils

Many of HTTP Utils also have async versions allowing them to participate in C#'s async/await workflows. The available async APIs include:

Task<string> GetStringFromUrlAsync(...)
Task<string> PostStringToUrlAsync(...)
Task<string> PostToUrlAsync(...)
Task<string> PostJsonToUrlAsync(...)
Task<string> PostXmlToUrlAsync(...)
Task<string> PutStringToUrlAsync(...)
Task<string> PutToUrlAsync(...)
Task<string> PutJsonToUrlAsync(...)
Task<string> PutXmlToUrlAsync(...)
Task<string> DeleteFromUrlAsync(...)
Task<string> OptionsFromUrlAsync(...)
Task<string> HeadFromUrlAsync(...)
Task<string> SendStringToUrlAsync(...)

Uploading Files

In the ServiceStack.ServiceClient.Web namespace there are more HTTP extensions available including the UploadFile extension methods to upload files using multi-part/formdata:

var httpRes = "http://example.org/upload"
    .PostFileToUrl(new FileInfo("/path/to/file.xml"), "application/xml");

Exception handling

Exception handling is another area we can DRY up using extension methods. Rather than wrapping them in our own Custom exception classes or repeating the boilerplate required to find the underlying issue on every request, we provide typed APIs over the native WebRequest Exceptions, providing typed DRY APIs to handle common HTTP Faults:

try 
{
    var response = "http://example.org/fault".GetStringFromUrl();
} 
catch (Exception ex) 
{
    var knownError = ex.IsBadRequest() 
        || ex.IsNotFound() 
        || ex.IsUnauthorized() 
        || ex.IsForbidden() 
        || ex.IsInternalServerError();

    var isAnyClientError = ex.IsAny400();
    var isAnyServerError = ex.IsAny500();

    HttpStatusCode? errorStatus = ex.GetStatus();
    string errorBody = ex.GetResponseBody();
}

HTTP Utils are mockable

The one disadvantage of using Extension methods is that by default they can be hard or impossible to mock which is why we've added explicit support for support for Mocking in OrmLite and now the above HTTP Util extension methods, e.g:

using (new HttpResultsFilter {
    StringResult = "mocked"
})
{
    //All return "mocked"
    "http://google.com".GetJsonFromUrl();
    "http://google.com".GetXmlFromUrl();
    "http://google.com".GetStringFromUrl(accept: "text/csv");
    "http://google.com".PostJsonToUrl(json: "{\"postdata\":1}");
}

See the HttpUtilsMockTests.cs for more examples showing how the HTTP Apis can be mocked.

HTTP API Reference

The above description should give you a good idea of how to make use of the APIs, although for a more complete reference we'll post the full signatures here.

Most of the APIs are located in the ServiceStack.Text namespace:

string GetJsonFromUrl(this string url, Action<HttpWebRequest> requestFilter=null,
  Action<HttpWebResponse> responseFilter=null)

string GetXmlFromUrl(this string url, Action<HttpWebRequest> requestFilter=null, 
  Action<HttpWebResponse> responseFilter=null)

string GetStringFromUrl(this string url, string accept = "*/*", 
  Action<HttpWebRequest> requestFilter=null, Action<HttpWebResponse> responseFilter=null)

string PostStringToUrl(this string url, string requestBody = null,
    string contentType = null, string accept = "*/*",
    Action<HttpWebRequest> requestFilter = null, Action<HttpWebResponse> responseFilter = null)

string PostToUrl(this string url, string formData=null, string accept="*/*",
  Action<HttpWebRequest> requestFilter=null, Action<HttpWebResponse> responseFilter=null)

string PostToUrl(this string url, object formData = null, string accept="*/*",
  Action<HttpWebRequest> requestFilter=null, Action<HttpWebResponse> responseFilter=null)

string PostJsonToUrl(this string url, string json,
    Action<HttpWebRequest> requestFilter = null, Action<HttpWebResponse> responseFilter = null)

string PostJsonToUrl(this string url, object data,
    Action<HttpWebRequest> requestFilter = null, Action<HttpWebResponse> responseFilter = null)

string PostXmlToUrl(this string url, string xml,
    Action<HttpWebRequest> requestFilter = null, Action<HttpWebResponse> responseFilter = null)

string PostXmlToUrl(this string url, object data,
    Action<HttpWebRequest> requestFilter = null, Action<HttpWebResponse> responseFilter = null)

string PutStringToUrl(this string url, string requestBody = null,
    string contentType = null, string accept = "*/*",
    Action<HttpWebRequest> requestFilter = null, Action<HttpWebResponse> responseFilter = null)

string PutToUrl(this string url, string formData=null, string accept="*/*",
  Action<HttpWebRequest> requestFilter=null, Action<HttpWebResponse> responseFilter=null)

string PutToUrl(this string url, object formData = null, string accept = "*/*",
  Action<HttpWebRequest> requestFilter=null, Action<HttpWebResponse> responseFilter=null)

string PutJsonToUrl(this string url, string json,
    Action<HttpWebRequest> requestFilter = null, Action<HttpWebResponse> responseFilter = null)

string PutJsonToUrl(this string url, object data,
    Action<HttpWebRequest> requestFilter = null, Action<HttpWebResponse> responseFilter = null)

string PutXmlToUrl(this string url, string xml,
    Action<HttpWebRequest> requestFilter = null, Action<HttpWebResponse> responseFilter = null)

string PutXmlToUrl(this string url, object data,
    Action<HttpWebRequest> requestFilter = null, Action<HttpWebResponse> responseFilter = null)

string DeleteFromUrl(this string url, string accept = "*/*",
    Action<HttpWebRequest> requestFilter=null, Action<HttpWebResponse> responseFilter=null)

string OptionsFromUrl(this string url, string accept = "*/*",
    Action<HttpWebRequest> requestFilter=null, Action<HttpWebResponse> responseFilter=null)

string HeadFromUrl(this string url, string accept = "*/*",
    Action<HttpWebRequest> requestFilter=null, Action<HttpWebResponse> responseFilter=null)

string SendStringToUrl(this string url, string method = null,
    string requestBody = null, string contentType = null, string accept = "*/*",
    Action<HttpWebRequest> requestFilter=null, Action<HttpWebResponse> responseFilter=null)

byte[] GetBytesFromUrl(this string url, string accept = "*/*", 
    Action<HttpWebRequest> requestFilter=null, Action<HttpWebResponse> responseFilter=null)

byte[] PostBytesToUrl(this string url, byte[] requestBody = null, string contentType = null, 
    string accept = "*/*",
    Action<HttpWebRequest> requestFilter=null, Action<HttpWebResponse> responseFilter=null)

byte[] PutBytesToUrl(this string url, byte[] requestBody = null, string contentType = null, 
    string accept = "*/*",
    Action<HttpWebRequest> requestFilter=null, Action<HttpWebResponse> responseFilter=null)

byte[] SendBytesToUrl(this string url, string method = null,
    byte[] requestBody = null, string contentType = null, string accept = "*/*",
    Action<HttpWebRequest> requestFilter=null, Action<HttpWebResponse> responseFilter=null)

bool IsAny300(this Exception ex)
bool IsAny400(this Exception ex)
bool IsAny500(this Exception ex)
bool IsBadRequest(this Exception ex)
bool IsNotFound(this Exception ex)
bool IsUnauthorized(this Exception ex)
bool IsForbidden(this Exception ex)
bool IsInternalServerError(this Exception ex)

HttpStatusCode? GetResponseStatus(this string url)
HttpStatusCode? GetStatus(this Exception ex)
HttpStatusCode? GetStatus(this WebException webEx)
string GetResponseBody(this Exception ex)

bool HasStatus(this WebException webEx, HttpStatusCode statusCode)

Whilst some additional HTTP APIs can be found in the ServiceStack.ServiceClient.Web namespace:

HttpWebResponse GetErrorResponse(this string url)

WebResponse PostFileToUrl(this string url, FileInfo uploadFileInfo, string uploadFileMimeType,
    string accept = null, Action<HttpWebRequest> requestFilter = null)

WebResponse UploadFile(this WebRequest webRequest, FileInfo uploadFileInfo, string uploadFileMimeType)

UploadFile(this WebRequest webRequest, Stream fileStream, string fileName, string mimeType,
    string accept = null, Action<HttpWebRequest> requestFilter = null)

void UploadFile(this WebRequest webRequest, Stream fileStream, string fileName)

Which is located in the ServiceStack.Text NuGet package