REST Client & Serverside APIs RFC

[thekid]

Scope of Change

The goal is to create consistent and clean REST Client & Serverside APIs.

Rationale

After using the APIs for a while lots of deficiencies were noted.

Functionality

This document is divided into two sections - one of the serverside implementation, one on the client.

Server

Requirements

• Annotation-based, reuse @webmethod from SOAP 🆗
• Built-in default serialization for all types 🆗
• Extensible serialization 🆗
• Default exception handling 🆗
• Exception mapper 🆗
• Default content type = JSON 🆗
• Handles XML and JSON via Accept negotiation 🆗
• Custom mime type support, graceful degradation 🆗
• Access to HTTP request and response headers 🆗
• Payload versioning support 🆗
• Command line testing server for easy setup 🆗

Current usage example:

<?php
  #[@webservice]
  class TheService extends Object {

    #[@webmethod(verb= 'GET', path= '/items')]
    public function listSomething() { }

    #[@webmethod(verb= 'POST', path= '/items', inject= array('payload'))]
    public function addSomething($data) {

    #[@webmethod(verb= 'GET', path= '/item/{id}')]
    public function getSomething($id) { }

    #[@webmethod(verb= 'PUT', path= '/item/{id}', inject= array('id', 'payload'))]
    public function saveSomething($id, $data) { }
  }
?>

Class-level changes

• The current approach doesn't allow specifying a base path. To allow this, the class level @webservice annotation will allow an optional "path" key. If it supplies a value, it will be prepended to all "path" keys defined in methods.
• To allow accessing headers, the "inject" key will be changed to four annotations: @param for query parameters, @path for path parameters, @header for headers, @cookie for cookies. These annotations will be method-level annotations but written in a way to become forward-compatible with the parameter annotations described in RFC Parameter annotations #218.
• Parameters without annotation will be fetched from the payload. The special-case handling for the string "payload" will be removed.
• To allow modifying the response other than its status code - to 200 (OK) and 500 (when an exception is raised) - methods may also return a Response instance.

The annotations are defined as follows:

@$<param>: <annotation> [ <name> ]
@$p: param                   // Use the request parameter "p"
@$p: param('id')             // Use the request parameter "id"

The Response builder class:

public class Response extends Object {
  public static self ok() {}                      // 200
  public static self created(string $location) {} // 201
  public static self noContent() {}               // 204
  public static self see(string $location) {}     // 302
  public static self notModified() {}             // 304
  public static self notFound() {}                // 404
  public static self notAcceptable() {}           // 406
  public static self error() {}                   // 500
  public static self status(int $status) {}       // ($status)

  public self withHeader(string $name, string $value) {}
  public self withPayload(var $data) {}
  public self withType(string $mediaType) {}
  public self withCookie(peer.http.Cookie $cookie) {}
  public self withBody(peer.http.RequestData $data) {}
}

Applying these changes to the above example would yield the following:

<?php
  #[@webservice(path = '/resource')]
  class TheService extends Object {

    #[@webmethod(verb= 'GET', path= '/items')]
    public function listSomething() { }

    #[@webmethod(verb= 'POST', path= '/items')]
    public function addSomething($data) {
      // TBI
      return Response::created($url);
    }

    #[@webmethod(verb= 'GET', path= '/item/{id}'), @$id: path]
    public function getSomething($id) { }

    #[
    #  @webmethod(verb= 'PUT', path= '/item/{id}'),
    #  @$identifier: path('id'),
    #  @$ref: header('X-Reference-Number')
    #]
    public function saveSomething($identifier, $data, $ref= NULL) { }
  }
?>

Serialization changes

When deserializing parameters from strings, the parameter type is taken into account. If the parameter type is a value object, the reflected class is checked for either a public one-arg constructor or a public static valueOf() method.

Example:

<?php
  // Either this...
  class Identifier extends Object {
    public function __construct($str) { }
  }

  // ...or that will work!
  class Identifier extends Object {
    public static function valueOf($str) { 
      return new self($str);
    }
  }

  #[@webservice(path = '/resource')]
  class TheService extends Object {

    #[@webmethod(verb= 'GET', path= '/item/{id}'), @$id: path]
    public function getSomething(Identifier $id) { }
  }
?>

Exception mappers

To map exceptions, we will reuse the Response object as seen above. Exception mappers can be registered by binding them to the context, which can be retrieved via the well-known @inject annotation.

Example implementation:

<?php
  #[@generic(implements= array('remote.RemoteException'))]
  class RemoteExceptionMapper extends Object implements ExceptionMapper {
    public function asResponse(Throwable $e) {
      return Response::status(503)->withPayload($e->getMessage());  // "Temporarily not available"
    }
  }

  #[@webservice(path = '/resource')]
  class TheService extends Object {

    #[@inject]
    public function registerException(RestContext $context) {
      $context->addExceptionMapping(new RemoteExceptionMapper());
    }
  }
?>

By default, the following exceptions are mapped:

Class                               #   Message
----------------------------------- --- --------------------------
lang.IllegalAccessException         403 Forbidden
lang.IllegalArgumentException       400 Bad request
lang.IllegalStateException          409 Conflict
lang.ElementNotFoundException       404 Not found
lang.MethodNotImplementedException  501 Method not implemented
lang.FormatException                422 Unprocessable entity

*                                   500 Internal Server Error

Mime type support

The default to map the mime type supplied with the client's Content-Type request header as follows: The pattern */json will map to the JSON serialization and */xml to the XML serialization. This means, custom mime types such as application/vnd.example.v3+json will work out of the box and will be mapped to the JSON serialization mechanism.

The Accept header will be parsed and returned media type will be determined by following the client-set preferences: The list is sorted by preference and the value with the highest preference then chosen. This value again is mapped against the above rules and will select the JSON or XML serialization. For example, if the client sends: Accept: text/xml;q=0.3, application/vnd.example.v3+json, the second (custom) mime type will be chosen, as its q-value is 1.0.

Extensible marshalling

The context may be extended to support marshalling from and to a given class. Builtin support for primitives, enums, value objects and arrays thereof exist, though you might want to handle your type specially.

<?php
  class Identifier extends Object {
    public function __construct($id) { ... }
    public function intValue() { ... }
  }

  #[@webservice(path = '/resource')]
  class TheService extends Object {

    #[@inject]
    public function changeMarshalling(RestContext $context) {
      $context->addSerializer(newinstance('TypeMarshaller<com.example.types.Identifier>', array(), '{
        public function marshal(Identifier $instance) {
          return $instance->intValue();
        }

        public function unmarshal($input) {
          return new Identifier((int)$input);
        }
      }');
    }
  }
?>

Payload versioning

Versioning is done via custom mime types, e.g.: application/vnd.example.v3+json. The version is simply part of the mime type string (here: "application/vnd.example.v3", and not handled in any special way). To overwrite a method with a different version, we use the "accepts" key of the @webmethod annotation for incoming payload and the "returns" key for returned data:

<?php
  #[@webservice(path = '/resource')]
  class TheService extends Object {

    #[@webmethod(verb= 'PUT', path= '/item/{id}', accepts= 'application/vnd.example.v1'), @$id: path]
    public function save_v1($id, $data) { }

    #[@webmethod(verb= 'GET', path= '/item/{id}', returns= 'application/vnd.example.v1'), @$id: path]
    public function get_v1($id) { }

    #[@webmethod(verb= 'PUT', path= '/item/{id}'), @$id: path]
    public function save($id, $data) { }

    #[@webmethod(verb= 'PUT', path= '/item/{id}'), @$id: path]
    public function get($id) { }
  }
?>

Keeping methods for all versions in one class encourages developers to clean up! If this is a definite problem, there is an alternative: Use the "accepts" key in the @webservice annotation:

<?php
  #[@webservice(path = '/resource', accepts= 'application/vnd.example.v1', returns= 'application/vnd.example.v1']
  class TheService_v1 extends Object {

    #[@webmethod(verb= 'PUT', path= '/item/{id}', @$id: path]
    public function save($id, $data) { }

    // ...
  }

  #[@webservice(path = '/resource')]
  class TheService extends Object {

    #[@webmethod(verb= 'PUT', path= '/item/{id}'), @$id: path]
    public function save($id, $data) { }

    // ...
  }
?>

If a given mime type cannot be matched directly, a method without "accepts" key is chosen (actually, its "accept" value is */*, which matches anything). If the acceptable mime type is not matched directly, a methd without "returns" key is chosen.

Methods can accept multiple mime types by using an array for the "accepts" key.

Client

Requirements

• Common use-cases use simple source paths (e.g. "GET" is default method if omitted)
• Authentication via Basic-Auth built-in 🆗
• Optional type-casting to userland objects
• Access to HTTP request and response headers 🆗
• Built-in serialization to JSON and XML for all types 🆗
• Extensible serialization and deserialization
• Exception mapping
• HTTP Errors => Exceptions as default 🆗
• Default content type = JSON 🆗
• Custom mime type support, graceful degradation
• ...

Current usage example:

<?php
  $client= new RestClient('http://api.example.com/');

  $request= new RestRequest('/resource/{id}');
  $request->addSegment('id', 5000);          // Replaces token in resource
  $request->addParameter('details', 'true'); // POST or querystring

  $response= $client->execute($request);
  $content= $response->content();            // Raw data as string
  $content= $response->data();               // Deserialize to map
?>

Security considerations

Speed impact

Dependencies

For a forward-compatible way of handling parameter annotations, extension methods could be used as seen in https://github.com/thekid/xp-framework/compare/rfc218-fc

Related documents

XP Framework

• Inject HttpRequestAdapter/HttpResponseAdapter into webmethods xp-framework#142 - Inject HttpRequestAdapter/HttpResponseAdapter into webmethods
• Rest client features xp-framework#135 - Rest client features
• Some improvements of the RestDataCaster xp-framework#144 - Some improvements of the RestDataCaster
• Json serialization for Generic objects xp-framework#119 - Json serialization for Generic objects
• Add webservices.rest xp-framework#90 - Add webservices.rest
• Feature rest: Support for ArrayType and Primitives xp-framework#78 - Feature rest: Support for ArrayType and Primitives
• REST implementation fixes xp-framework#74 - REST implementation fixes
• RFC REST server API implementation #216 - Original webservices.rest API

Java

• http://docs.jboss.org/resteasy/docs/1.1.GA/userguide/html_single/index.html
• http://wiki.restlet.org/docs_2.1/13-restlet/27-restlet/130-restlet.html - Mapping HTTP headers

C#

• http://msdn.microsoft.com/de-de/magazine/dd315413.aspx - Eine Einführung in REST-Dienste mit WCF
• http://joseoncode.com/2011/04/08/content-negotiation-with-nancy/ - Content Negotiation with Nancy

[thekid]

The wrapping XML element could be set via xmlwrapped annotation - see https://github.com/thekid/xp-framework/compare/xmlwrapped

Update (Nov 27th): Now supported

[thekid]

Matrix parameters

Origin

The document Matrix URIs describes the original intent, and bears a note stating "relative Matrix URLs are not generally implemented so this is just a theoretical discussion."

How's it done elsewhere?

On matrix parameters, I found the following in a JAX-RS implementations' documentation:

URI matrix parameters, like URI query parameters, are name/value pairs that can provide additional information selecting a resource. Unlike query parameters, matrix parameters can appear anywhere in a URI and they are separated from the hierarchical path segments of the URI using a semicolon(;). /mostersforhire/daikaiju;id=jonas has one matrix parameter called id and /monstersforhire/japan;type=daikaiju/flying;wingspan=40 has two matrix parameters called type and wingspan

Note: Matrix parameters are not evaluated when computing a resource's URI. So, the URI used to locate the proper resource to handle the request URI /monstersforhire/japan;type=daikaiju/flying;wingspan=40 is /monstersforhire/japan/flying.

Note: JAX-RS evaluates all of the matrix parameters in a URI at once, so it cannot enforce constraints on a matrix parameters location in a URI. For example /monstersforhire/japan;type=daikaiju/flying;wingspan=40, /monstersforhire/japan/flying;type=daikaiju;wingspan=40, and /monstersforhire/japan;type=daikaiju;wingspan=40/flying are all treated as equivalent by a RESTful Web service implemented using the JAX-RS APIs.

One possible implementation could extract them out of the URL into a map and then rebuild the URL without them. Another possibility would be to handle this in the peer.URL class, adding accessors for matrix params (this could prove to be very tough!)

Bound to path segments

Matrix parameters belong to path segments, which in this implementation, are pulled in to a method via path(...) source. Give the following URI:

http://example.com/res/categories;name=foo/objects;name=green/?page=1

...the name=foo belongs to a category and name=green to an object inside it, and might just be a fuzzier variant of:

http://example.com/res/categories;id=6100/objects;id=1/?page=1

...addressing the entities not by its primary key but by a name (which can include some overhead).

Alongside this argument, we should be able to write a single handler method but retrieve the entities using different - e.g. rdbms.Finder - methods.

  #[@webservice(verb= 'GET', path= '/categories{;category}/objects{;object}')]
  public function findObjects($category, $object) {
    // $category is now array('name' => 'foo') or array('id' => '6100')
    // $object is array('name' => 'green') or array('id' => '1')
  }

One syntax?

On further thought, the id-example above is almost like saying:

http://example.com/res/categories/6100/objects/1/?page=1

...which would be handled by

  #[@webservice(verb= 'GET', path= '/categories/{category}/objects/{object}')]
  public function getObjects($category, $object) {
    // $category is 6100
    // $object is 1
  }

Can we come up with one syntax for all of them?

[thekid]

Announced in the XP Blog - http://news.planet-xp.net/article/486/2012/12/08/

[thekid]

Released 2012-12-21 together with 5.8.10-RELEASE RC1

[thekid]

Fixes in 5.8.10-RELEASE RC2 include:

• The default exception mapping has been extended as described here
• REST handlers can now have util.Properties injected via [@inject] just like "xpcli"s
• Exceptions from marshalling / unmarshalling parameters are handled
• Unmarshalling to value objects with single-arg constructor / static valueOf() methods is only performed with simple payloads (primitives, an array with one element or a map with one key-value pair).
• Parameter reflection was fixed for inherited methods - this bug has been in the framework for ages and hadn't been discovered previously!

See also xp-framework/xp-framework@r5.8.10RC1...r5.8.10RC2