Toward Catalyst Restfullness - Improving Catalyst use for web services
This is a document which outlines a full proposal of achievable development goals that would improve Catalyst for web service development. It also reviews the deficiencies in existing strategies and outlines the reasons for why we believe such features belong in Catalyst core. We will also note limitations to our outlined approach.
Catalyst is a superb web development framework which has demonstrated itself via many successful projects. However, when it was originally designed the dominant use for web frameworks was to hook a database (or similar data store) to the internet via HTML and server side generated web pages (typically using a server side templating system such as Template Toolkit or Mason). Javascript and client side (in browser) programming was usually limited to progressive enhancement effects. There was very little use case for server to client communication initiated via Javascript on the clientside. In addition, the market for web services (exposing one's business logic API or data endpoints via HTTP, using SOAP, XML/JSON-RPC or similar) was limited.
As a result, the design of Catalyst stressed features of use to development of the primary use cases of the time.
Since contemporary web development has many use cases for RESTful style API and AJAX, it makes sense for Catalyst to offer some minimal level of built in support, such that newcomers to the framework would get a base amount of features that could be counted on. In addition, this base support could be improved upon via framework extensions such that the community could continue the conversation at a higher level.
The scope of this project is to improve the baseline support for the most common use cases web developers encounter requiring support for lightweight web services. Typically this involves exposing some data via JSON for consumption in a web brower, when a front end developer is adding web page interactive features, or even building 'one page javascript apps' as is becoming a more common use case. Additionally we'd like to enable programmers to have a common approach to building more complex web services whose typical use case is providing data endpoints for applications beyond client side web pages. Case could include support for mobile apps (such as Android or iPhone apps), server to server communications (for build software as a service systems, or for data exchange with external partners) and situations where an architecture is heterogenous and web services offers a possible integration approach.
Out of scope is to include all the features required to build fully REST compliant web services (although one could fairly say building the additional required pieces would just require the coding commitment as Catalyst already exposes enough API to fill in the missing peices on your own).
We would add the follow API
The core Catalyst application class would get some additional API so that we could register handlers for incoming HTTP body data beyond the existing 'classic' HTML forms and file upload. The core class would do no parsing of its own, but rather 'register' and offer these parsers to a requesting class. A user would configure this via Catalyst configuration (although API will be documented, it will not be considered proper for public use. Here's the proposed example:
package MyApp::Web;
use Catalyst;
use JSON;
__PACKAGE__->config(
'body_data_parsers' => {
'application/json' => sub { JSON::decode $_ }
},
);We defined a new top level configuration key called body_data_parsers whose value MUST be a Hashref, which itself contains a keys (which matches a particular incoming body MIME type) and coderefs ( handlers that receive the flattened body content as a string and are expected to return a Perl data reference).
The value coderef is evaluated with $_ localized to the string form of the request content. You should use $_ since we currently do not document the meanings of @_ in this context. The return value is expected to be a Perl reference.
For the first version, we consider parsing body input streams, or iterative parsing of large sized incoming request bodies, to be out of scope.
Catalyst will come bundled with the following built in body parsers:
application/json
application/x-www-form-urlencoded
application/octetAlthough you may of course overide the built-in settings in local configuration.
For JSON, Catalyst will use JSON::PP unless JSON is found to be installed (if you want the XS based parser, you should include it in your application Makefile.PL). For Form data, this will incorporate the features of Catalyst::Plugin::Params::Nested as to better support how common clientside Javascript frameworks attempt to encode complex incoming data. Please note that in some cases this could lead to 'double parsing' of the request body, so it is recommended that if you plan to make use of this feature, to enable the Catalyst global configuration parse_on_demand, which defers all request content parsing until its requested. Regardless of the configuration setting, the data parsers should always try to do the most optimal this (reading from $psgi_env->{input} or $request->body as given the current state of things). See Request object below.
application/octet parsing will return a filehandle like object, similar to how the current approach using HTTP::Body works.
Multipart content is considered out of scope for this iteration (but we do note that support for this already does exist in Catalyst).
When matching an incoming request to a body parser, we will chose the correct parser using a 'best match' algorithm similiar to
HTTP::Headers::ActionPack::ContentNegotiation method choose_media_type
That way you can set quality and standard mime type pattern matching.
For this first version, errors are propogated upwards. We do not attempt to set any response statuses.
The following describes the proposed approach to enable Catalyst to automate some types of response body encoding. The current scope for this proposal does not include methods to support streaming and evented / non blocking interfaces.
In traditional Catalyst Applications, one either sets the request body to a string with is assumed to be encoded according to the header content type or a filehandle, which allows one way to stream a response.
We will add the ability to set the response body as a reference either to an array ref or a hashref. A reference value like this will be an indicator to Catalyst that the body value must be encoded via one of the globally defined body format handlers (see below). For example:
package MyApp::Web::Controller::User;
use base 'Catalyst::Controller';
sub myaction :Local {
my ($self, $ctx) = @_;
$ctx->res->content_type('application/json');
$ctx->res->body( { a => 1, b => 2 });
}
1;We propose allowing one to establish body 'format' handlers, as well as the establishment of several common built in formaters. Here's the proposed example:
package MyApp::Web;
use Catalyst;
use JSON;
__PACKAGE__->config(
'body_data_formatters' => {
'application/json' => sub { JSON::encode $_ }
},
);This works similarly to the produced body data decoders, in that we allow for a new top level configuration key that takes a hash ref, whose keys are strings indicating a content type (or a regular expression to a content type) and whos values are a subroutine ref that receive the reference value placed in the response body localized to $_.
Matching a reference body value to a format handler will take place by inspecting the response content type. If no content type is set for the response AND the configuration value alway_do_global_content_negotiation is true (it is false by default) then we will use standard content negotiation, inspecting the Request and choosing the best matching formatter. If not formater matches, this will raise a yet to be determined error.
Catalyst will come bundled with the following built in body formatters:
application/json
application/x-www-form-urlencodedAs with the data parsers, we bundle JSON:PP but use a faster, XS based parser if one is already installed and available.
The following new API is proposed
Given a list of one or more MIME types, examine the incoming request ACCEPTS HTTP header and return the one that is the best match using standard techniques for content negotiation
Works like "accepts" except it examines the HTTP language and charset accepts.
Check if the request was issued with the "X-Requested-With" header field set to "XMLHttpRequest" (jQuery etc).
returns true if the request appears to originate from a web browser
When called, it will try to parse the request body, if any exists by comparing the request content type to the registered body parsers (defined in the previous section). Depending on the status of parse_on_demand it will either read from $psgi_env->{input} or the previous read value in $c->request->body.
This is allowed to be undefined in the case where there is not body content.
The following new API is proposed
This method will now accept an array or hash reference, which is later formatted via an encoded defined and indicated by the existing set http content type header, or via standard content negotiation if the global configuration alway_do_global_content_negotiation is true.
Allows one to indicate the response body can be one of several content types which should be choosen using standard content negotiations techniques. For example:
package MyApp::Web::Controller::User;
use base 'Catalyst::Controller';
sub example :Local {
my ($self, $ctx) = @_;
$ctx->response->format(
'application/json' => sub { +{ a=>1, b=>2 }},
'text/xml' => sub { ... },
);
}
1;The format method takes a hash, where the key is a content type being formated (the response content type is set to this, and it may not be a regular expression it must be a real content type) and the value is a sub ref whose value is stored in the response body.
Arguments for the subref have not yet been defined.
This approach also will set the reponse http accept header based on the allowed formats.
The main use case for this approach is more compact control when you wish to vary the actual content based on the content type.
Given a valid psgi response (either tuple or delayed coderef), set the LCatalyst) response accordingly.
We defined the two new Controller attributes to assist in controlling which actions handle the request
package MyApp::Web::Controller::User;
use base 'Catalyst::Controller';
sub example :Local Provides('application/json') {
my ($self, $ctx) = @_;
}
1;Indicates the the action, or following action chain, can provide the specified content. We should inspect the request HTTP Accept to make sure that the client that initatiated the request is willing to accept this content type. An action may indicate more than one Provides.
Please not that it is not in scope for this iteration to make the Provides subroutine attribute conform to standard content negotiation rules. We just follow the standard Catalyst, the first and best match just wins.
package MyApp::Web::Controller::User;
use base 'Catalyst::Controller';
sub example :Local Consumes('application/json') {
my ($self, $ctx) = @_;
}
1;Indicates that the action, or following action chain is willing to process certain request content types (or list of content types).
We check the request content type fo a match. This value is allowed to be a string that is formatted as a regular expression.
Please not that it is not in scope for this iteration to make the Consumes subroutine attribute conform to standard content negotiation rules. We just follow the standard Catalyst, the first and best match just wins.
12 August 2013
- Configuration key names changed to match existing Catalyst standards
- Body parsers can no longer match request content types using regular
expressions since this vastly complicates how to merge configurations
and possible is more fine grained for the global object.
- misc. spelling and grammer fixes