Streamed Body

Chris Hagan edited this page May 3, 2013 · 4 revisions

Webmachine allows the resource developer to handle request and response bodies as either whole units (binary or iolist) or as many hunks of data.

The body-handling functions are:

  • wrq:req_body/1
  • wrq:stream_req_body/2
  • wrq:set_resp_body/2

The last of these, wrq:set_resp_body/2, is also called implicitly with the return value of any content-producing function such as to_html/2.

The first of these (req_body) is the simplest. It will provide the whole incoming request body as a binary. (Unless the body is too large, as set by wrq:set_max_recv_body/2 or defaulting to 50M). For the majority of resources, this is the easiest way to handle the incoming request body.

If a resource wants to handle the incoming request body a hunk at a time, it may call wrq:stream_req_body/2 instead. Instead of a binary, this produces a StreamBody structure.

It is an error to call both wrq:req_body/1 and wrq:stream_req_body/2 in the execution of a single resource.

A StreamBody is a pair of the form {Data,Next} where Data is a binary and Next is either the atom done signifying the end of the body or else a 0-arity function that, when called, will produce the "next" StreamBody structure.

The integer parameter to wrq:stream_req_body/2 indicates the maximum size in bytes of any hunk from the resulting StreamBody.

When a resource provides a body to be sent in the response, it should use wrq:set_resp_body/2. The first parameter to this function may be either an iolist, representing the entire body, or else a pair of the form {stream, StreamBody}.

As of 1.10.0, the first parameter to wrq:set_resp_body/2 may also be {{known_length_stream, ResourceLength, StreamBody}. The purpose of ths form is to avoid buffering the entire content of a large response in memory when not using multipart/mixed or some other chunked format.

Some examples may make the usage of this API clearer.

A simple one way resource which generates string of a specified length, composed entirely of the letter $a:

-export([init/1, to_html/2]).


    {ok, Config}.

send_streamed_body(Remaining) ->
    case Remaining of
        Partial when Partial =< PacketSize ->
        Full ->
            {string:chars($a,Full), fun() -> send_streamed_body(Remaining - PacketSize) end}

    PathInfo = wrq:path_info(ReqData),
    {ok,SizeString} = dict:find(size,PathInfo),
    {Size,[]} = string:to_integer(SizeString),
    {{halt,200}, wrq:set_resp_body({stream,send_streamed_body(Size)},ReqData), State}.

And a complete and working resource module using this API in both directions:

-export([init/1, allowed_methods/2, process_post/2]).


init([]) -> {ok, undefined}.

allowed_methods(ReqData, State) -> {['POST'], ReqData, State}.

process_post(ReqData, State) ->
    Body = get_streamed_body(wrq:stream_req_body(ReqData, 3), []),
    {true, wrq:set_resp_body({stream, send_streamed_body(Body,4)},ReqData), State}.

send_streamed_body(Body, Max) ->
    case Body of
        <<Hunk:HunkLen/bitstring, Rest/binary>> ->
            io:format("SENT ~p~n",[Hunk]),
            {Hunk, fun() -> send_streamed_body(Rest,Max) end};
        _ ->
            io:format("SENT ~p~n",[Body]),
            {Body, done}

get_streamed_body({Hunk,done},Acc) ->
    io:format("RECEIVED ~p~n",[Hunk]),
get_streamed_body({Hunk,Next},Acc) ->
    io:format("RECEIVED ~p~n",[Hunk]),

If you use this resource in place of the file /tmp/mywebdemo/src/mywebdemo_resource.erl in the quickstart setup, you should then be able to issue curl -d '1234567890' on the command line and the io:format calls will show you what is going on.

Obviously, a realistic resource wouldn't use this API just to collect the whole binary into memory or break one up that is already present

  • you'd use wrq:req_body and put a simple iolist into wrq:set_resp_body instead. Also, the choices of 3 and 4 bytes as hunk size are far from optimal for most reasonable uses. This resource is intended only as a demonstration of the API, not as a real-world use of streaming request/response bodies.