Nowadays, numerous web applications offer their features as web services, APIs that can be used by other web applications or native clients. This is how Twitter, GitHub, Google Maps or countless others can be scripted by third-party applications, using cleanly defined and easily accessible protocols.
With Opa, offering a web service is just as simple as creating any other form of web application. In this chapter, instead of writing a new application, we will extend our wiki to make it accessible through such a web API. This task will lead us through REST web service design, command-line testing of Opa services, management of URI queries and more.
|
Tip
|
Several protocols share the landscape of web services, in particular REST (Representational State Transfer, a simple standard which does not specify how messages should be formated, only how they should be exchanged), SOAP (a more complex standard imposing conventions on the formatting of messages) and WSDL (a higher-level protocol). In this chapter, we’ll only cover REST. |
In this chapter, we will modify our wiki to make it accessible by a REST API. This involves few changes from the original wiki, only the addition of a few cases to differentiate between several kinds of requests that can be sent by a client — which does not need to be a browser anymore.
If you are curious, this is the full source code of the REST wiki server:
link:hello_wiki_rest_client.opa[role=include]We will now walk through the concepts introduced in this listing.
In the rest of this chapter (pun intended), we will want to be able to delete a topic previously added to the wiki. Adding this feature (without showing it in the user interface) is just the matter of one line, as follows:
remove_topic(topic) = Db.remove(@/wiki[topic])In this extract, we use function Db.remove, a function whose sole role is to remove the contents of a database path. Notice the @ before /wiki[topic]? This symbol signifies that we are not working with the value /wiki[topic] but with the path itself. If we had omitted this symbol, the Opa compiler would have complained that Db.remove cannot work with a string — which is absolutely true.
A web service behaves much like a web application, without the client part. In other word, as any Opa web application, it starts with a server:
topic_of_path(path) = String.capitalize(String.to_lower(List.to_string_using("", "", "::", path)))
start =
| {path = [] ... } -> display("Hello")
| {path = ["_rest_" | path] ...} -> rest(topic_of_path(path))
| {~path ...} -> display(topic_of_path(path))
server = Server.of_bundle([@static_include_directory("resources")])
server = Server.simple_dispatch(start)In this version of start, we have slightly altered our pattern-matching to handle the case of paths starting with "\_rest_". We decide that such paths are actually entry points for REST-based requests and handle them as such. Here, we delegate the management to function rest, which we write immediately:
As you may see, this function is also quite simple:
rest(topic) =
match HttpServer.get_method() with
| {some = {post}} ->
_ = save_source(topic,
match HttpServer.get_body() with
| ~{some} -> some
| {none} -> ""
)
Resource.raw_status({success})
| {some = {delete}}->
do remove_topic(topic)
Resource.raw_status({success})
| {some = {get}} ->
Resource.raw_response(load_source(topic), "text/plain", {success})
| _ ->
Resource.raw_status({method_not_allowed})First, notice that rest is based on pattern-matching. Expect to meet pattern-matching constantly in Opa. The first three patterns are built from some of to the distinct verbs of the standard vocabulary of REST (these verbs are called Http methods):
-
\{post\} is used to place information on a server, here to add some content to the wiki;
-
\{delete\} is used to remove information from the server, here remove a topic from the wiki;
-
\{get\} is used to get information from a server, here to download the source code of an entry.
From these verbs, we build the following patterns:
-
\{some = \{post\}\}, i.e. the Http method is defined and is post;
-
\{some = \{delete\}\}, i.e. the Http method is defined and is delete;
-
\{some = \{get\}\}, i.e. the Http method is defined and is get;
-
_, i.e. any other case, whether the Http method is not defined or whether it is a method that we do not wish to handle.
Everything else in rest is simply function calls. You can find the definition of each function in the API documentation, so we will just introduce quickly the functions you have not seen yet:
-
Function HttpRequest.get_method has type -> option(method). If the function is called from a request and this request has a method m, it produces \{some = m\}. Otherwise, it produces \{none\}.
-
Similarly, HttpRequest.get_body has type -> option(string). If the function is called from a request containing a body b, it produces \{some = b\}. Otherwise, it produces \{none\}.
-
Function Resource.raw_response has type string, string, status -> resource. It produces a resource with a body from its body, its MIME type, and a status. This function is commonly used to reply to REST requests.
-
Finally, function Resource.raw_status has type status -> resource. It produces an empty resource, and is generally used to return an error to a REST request.
As pattern-matching against an option is very common, Opa provides an operator ? that can be used to make the above extract shorter and more readable. Expression a?b is equivalent to the following three lines:
match a with
| {none} -> b
| ~{some} -> someWith this expression, we may rewrite our extract as follows:
rest(topic) =
match HttpServer.get_method() with
| {some = {post}} ->
_ = save_source(topic, HttpServer.get_body() ? "")
Resource.raw_status({success})
| {some = {delete}}->
do remove_topic(topic)
Resource.raw_status({success})
| {some = {get}} ->
Resource.raw_response(load_source(topic), "text/plain", {success})
| _ ->
Resource.raw_status({method_not_allowed})And with this, we are done! Our wiki can now be scripted by external web applications:
link:hello_wiki_rest.opa[role=include]All in all, the changes required a dozen lines of code.
Exercises will show you how to introduce more complex forms of scripting.
The simplest way of testing a REST API is to use a command-line tool that lets you place requests directly, for instance curl or wget. Assuming that curl is installed on your system, the following command-line will test the result of placing a \{get\} request at address \_rest_/hello:
curl localhost:8080/_rest_/hello
Execute this command-line and curl will show you the result of the call.
Similarly, the following command-line will test the result of placing a \{post\} request at the same address:
curl localhost:8080/_rest_/hello -d "I've just POSTed to change the contents of my wiki"
Now, we are not here to learn about curl, but to learn about Opa. And what best way to test the REST API of a wiki than by writing a web front-end that does not rely on its own database but on that of the wiki we have just defined?
We will do just this in the next chapter.
As mentioned, functions HttpServer.get_method and HttpServer.get_body can produce result \{none\} if the http method (respectively the body) does not exist.
This may be surprising, as, by definition of the protocols, every request has a method (not all have a body). Indeed, the only case in which HttpServer.get_method returns \{none\} is when there is no request, i.e. when the function has been called by the server for its own use and not during the execution of a request on behalf of a web browser or a distant web server.
On the other hand, many requests do not have a body. Function HttpServer.get_body returns \{none\} when there is no body, or when there is no request, as above.
If you have started thinking about large applications, at this stage, you might start worrying about having to centralize all your path management into only one pattern-matching, which could hurt modularity and hamper your work.
Well, you can stop worrying. With Opa, you may combine any number of servers in an application. If you look at the API documentation of Opa, you can find function Server.make_dispatch, which is designed to let you write separate servers for subpaths, subdomains, etc.
Add a REST API to your chat, with the following feature:
-
use a \{post\} request send a message for immediate display into the chat (for the moment, we will assume that the message has been written by author "ghost").
|
Tip
|
To deal with several entry points, you will have to rewrite your server and replace one_page_bundle by a dispatcher. For these exercises, we decide that any request placed on path \_rest_ is a REST request. |
For testing, use the following command-line (assuming that curl is installed on your system):
curl localhost:8080/_rest_ -d "Whispers..."
If you have not done so yet, update your chat to maintain conversation logs in the database.
Now, add the following REST API:
-
use a \{get\} request to get the log of messages as string containing one message per line.
Remember, use function List.to_string_using to convert a list to a string.
For this exercise, we wish to extend the REST API for the chat to be able to send a message and give a name to the author of the message. For this purpose, we need to send more informations than simply \{post\}. In the REST world, there are typically two ways of passing additional informations: either in the URI itself or in the body of the request. For this exercise, we will see the first option:
-
if a \{post\} request is received on \_rest_ and if the query of the request contains a pair ("author", x), use the value of x as the author name;
-
otherwise, use name "ghost", as above.
|
Tip
|
About queries
A query is an element of a URI. From the user’s perspective, queries look like ?author=name&arg2=val2&arg3=val3. From the developer’s perspective, the query is contained in field query of the URI, just as path. This field contains a list of pairs with the name of the argument and its value. So, for the previous query, the list will look like: [("author", "name"), ("arg2", "val2"), ("arg3", "val3")]Note that the order of these arguments is meaningless. |
|
Tip
|
About association lists
Lists of pairs containing a name and a value (or, more generally, a key and a value) are generally called "association lists". In Opa, the most common function to extract a value from an association list is List.assoc. This function takes two arguments: the key to search and the list in which to search. Its result is an option which may contain either \{none\} (if the key does not appear in the list) or \{some = v\} (if the key appears in the list, associated to value v). |
Another common technique used among REST services is to pass additional information as part of the body of the request, often formated using the JavaScript Object Notation language (or JSON). The objective of this exercise is to use JSON instead of the URI to send the author name to the server.
-
if a \{post\} request is received on \_rest_ and if the body of the request is a valid JSON construction containing a field "author", use the value associated to this field as the author name;
-
otherwise, use name "ghost", as above.
|
Tip
|
JSON requests
To obtain the JSON body of a request, use function HttpRequest.get_json_body. |
|
Tip
|
About JSON
JSON is a format of strings which can be interpreted as simple data structures. In Opa, a string in JSON format can be transformed into a value with type RPC.Json.json by using function Json.deserialize: string -> option(RPC.Json.json)Note that this function can return \{none\} if the string was incorrectly formated. The opposite operation is implemented by function Json.serialize: RPC.Json.json -> stringType RPC.Json.json is defined as follows: type RPC.Json.json =
{ Int: int}
/ { Float: float}
/ { String: string}
/ { Bool: bool}
/ { List: list(RPC.Json.json)}
/ { Record: list((string, RPC.Json.json))}As above, case Record corresponds to a list of associations. |
With Opa, accessing a distant web service is as simple as creating one. In this chapter, we will develop a variant of our wiki which, instead of using its own database, will serve as a front-end for the wiki developed in the previous chapter. This task will lead us through the other side of REST: how to connect to a distant server, send commands and interpret results. Somewhere along the way, we will also see how to handle command-line arguments in Opa, how to analyze text and some interesting features of the language.
The general idea behind REST is to use the well-known HTTP protocol to send/receive commands through the web. In other words, a REST client is just a web application that has a few of the features of a browser, i.e. a web client: the functions that we will meet in this chapter can be used just as well for purposes unrelated to REST, for instance to write a web crawler, to post the contents of a web form automatically, or to download a distant image from an Opa application.
In this chapter we modify further our wiki to make it use a distant REST API instead of its own database. As previously, this involves few changes from the original wiki: we remove the database, we handle error cases in case of communication issues, and we introduce command-line options to let users specify where to find the distant REST server.
If you are curious, this is the full source code of the REST wiki client (which also acts as a server):
link:hello_wiki_rest_client_customizable.opa[role=include]To connect to distant servers and services, Opa offers a module called WebClient. The following extract adapts load_source to perform loading from a distant service:
@publish load_source(topic) =
match WebClient.Get.try_get(uri_for_topic(topic)) with
| {failure = _} -> "Error, could not connect"
| {~success} ->
match WebClient.Result.get_class(success) with
| {success} -> success.content
| _ -> "Error {success.code}"
endAs in previous variants of the wiki, this version of load_source attempts to produce the source code matching a topic. The main difference is that, instead of reading the database, it performs a \{get\} request on a distant server. This is the role of function WebClient.Get.try_get — of course, module WebClient offers similar functions other operations other than \{get\}. This function takes as argument a URI — here, provided by a function uri_for_topic that we will need to write at some point — and produces as result a sum type, containing either \{failure = f\} or \{success = s\}.
Failures take place when the operation could not proceed at all, for instance because of network issues, or because the distant server is down. In such case, f contains more details about the exact error. Any other case means that the request was successful. Note that, depending on what you are trying to do, the result of the request could still be something that is no use to your application. For instance, the server may have returned some content along with a status of "404 Not Found", to indicate that this content is a default page and that it actually does not know what to do with your URI. It could be a "100 Continue", to indicate that you should now send more information before it can proceed. All these responses are successes at the level of WebClient, although many applications decide to treat them as failures.
Here, for our simple protocol, we use function WebClient.Result.get_class to perform a rough decoding of the server response and categorize it as a success (case \{success\}) or anything else (redirection, client error, server error, etc.) (case _). In case of success, we return the content of the response, e.g. success.content.
|
Tip
|
There’s more to distribution than REST
Do not forget that this web client is a demonstration of REST. In Opa, REST is but one of the many ways of handling distribution. Indeed, as long as your application is written only in Opa, Opa can perform distribution automatically, using protocols that are largely more efficient for this purpose than REST. |
Function remove_topic is even simpler (we ignore the result of the operation):
remove_topic(topic) =
_ = WebClient.Delete.try_delete(uri_for_topic(topic))
voidWe can similarly adapt load_rendered, with a slight change to use the API we have previously published:
@publish load_rendered(topic) =
source = load_source(topic)
Markdown.xhtml_of_string(Markdown.default_options, source)Finally, we can adapt save_source, as follows:
@publish save_source(topic, source) =
match WebClient.Post.try_post(uri_for_topic(topic), source) with
| { failure = _ } ->
{failure = "Could not reach the distant server"}
| { success = s } ->
match WebClient.Result.get_class(s) with
| {success} -> {success = load_rendered(topic)}
| _ -> {failure = "Error {s.code}"}
endThis version of save_source differs slightly from the original, not only because it uses a \{post\} request to send the information, but also because it either returns the result \{success=...\} or indicates an error with \{failure=...\}.
We take this opportunity to tweak our UI with a box meant to report such errors:
We add a <div> called show_messages to the HTML-like user interface, and we update it in edit and save, as follows:
display(topic) =
Resource.styled_page("About {topic}", ["/resources/css.css"],
<div id=#header><div id=#logo></div>About {topic}</div>
<div class="show_content" id=#show_content ondblclick={_ -> edit(topic)}>{load_rendered(topic)}</>
<div class="show_messages" id=#show_messages />
<textarea class="edit_content" id=#edit_content style="display:none" cols="40" rows="30" onblur={_ -> save(topic)}></>
)
edit(topic) =
do Dom.transform([#show_messages <- <></>])
do Dom.set_value(#edit_content, load_source(topic))
do Dom.hide(#show_content)
do Dom.show(#edit_content)
do Dom.give_focus(#edit_content)
void
save(topic) =
match save_source(topic, Dom.get_value(#edit_content)) with
| { ~success } ->
do Dom.transform([#show_content <- success]);
do Dom.hide(#edit_content);
do Dom.show(#show_content);
void
| {~failure} ->
do Dom.transform([#show_messages <- <>{failure}</>])
voidAnd that is all for the user interface.
We have already been using URIs by performing pattern-matching on them inside dispatchers. It is now time to build new URIs for our function uri_for_topic.
|
Tip
|
About absolute URIs
Many languages consider that a URI is simply a string. In Opa, URIs come in several flavors. So far, we have been using absolute uris, as defined by the following type: type Uri.absolute =
{ schema: option(string)
; credentials: Uri.uri_credentials
; domain: string
; port: option(int)
; path: list(string)
; query: list((string,string))
; fragment: option(string)
}
type Uri.uri_credentials =
{ username : option(string)
; password : option(string)
}Other flavor exists, e.g. to handle e-mail addresses, relative URIs, etc. The most general form of URI is Uri.uri, whose definition looks like: type Uri.uri = Uri.absolute / Uri.relative / ...To cast an Uri.absolute into a Uri.uri, use function Uri.of_absolute. To build a Uri.absolute, you can either construct a record manually or derive one from Uri.default_absolute. |
To match the API we have defined earlier, we need to place requests for topic at URI http://myserver/\_rest_/topic. In other words, we may write:
uri_for_topic(topic) =
Uri.of_absolute({ schema = {some = "http"}
credentials = {username = {none} password = {none}}
domain = "localhost" //Assume server is launched locally
port = {some = 8080}//Assume server is launched on port 8080
path = ["_rest_", topic]
query = []
fragment = {none}
})It is, however, a tad clumsy to provide query, fragment, port, etc. only to mention that they are not used. So we will prefer to derive a uri from Uri.default_absolute, as follows:
uri_for_topic(topic) =
Uri.of_absolute({Uri.default_absolute with
schema = {some = "http"}
domain = "localhost"
port = {some = 8080}
path = ["_rest_", topic]
})|
Tip
|
Record derivation
Use record derivation to construct a record from another one by modifying several fields. For instance, if we have foo = {a = 1; b = 2}we may write bar = {foo with b = 17}This is equivalent to the following bar = {a = foo.a; b = 17}Using record derivation is a good habit, as it is not only more readable than copying field values from one record to another, but also faster. |
With this, your client wiki is complete:
link:hello_wiki_rest.opa[role=include]Launch the server wiki, launch the client wiki on a different port (use option -p or --opa-server-port to select a port) and behold, you can edit your wiki from two distinct ports. Or two distinct servers, if you replace "localhost" by the appropriate server name.
On the other hand, replacing a magic constant by another equally magic constant is not very nice. Would it not be better to decide that the server name and port are options that can be configured without recompiling?
Opa is a higher-order language. Among other things, this means that there are many ways of defining a function. So far, our function definitions have been quite simple, but if we wish to define a function whose behavior depends on a command-line option or on an option somehow defined at start-up, the best and nicest way is to expand our horizon.
In this case, expanding our horizon starts by rewriting uri_for_topic as follows:
uri_for_topic = topic ->
Uri.of_absolute({Uri.default_absolute with
schema = {some = "http"}
domain = "localhost"
port = {some = 8080}
path = ["_rest_", topic]
})So far, this is absolutely equivalent to what we had written earlier. Note that we can move the parenthesis as follows:
uri_for_topic = (topic ->
Uri.of_absolute({Uri.default_absolute with
schema = {some = "http"}
domain = "localhost"
port = {some = 8080}
path = ["_rest_", topic]
})
)While we have not changed the behavior of the function at all, this rewrite is a nice opportunity to split the construction URI in two parts, as follows:
uri_for_topic =
(
base_uri = {Uri.default_absolute with
schema = {some = "http"}
domain = "localhost"
port = {some = 8080}
}
topic -> Uri.of_absolute({base_uri with
path = ["_rest_", topic]
})
)Suddenly, things have changed a little: uri_for_topic is still a function that takes a topic and returns a URI, but with a twist. At some point, when the function itself is built, it first initializes a (local) value called base_uri which it uses whenever the function is called. This is an example use of closures.
|
Tip
|
About closures
You have already met closures in previous chapters. Indeed, most of the event handlers we have been using so far are closures. Rigorously, a closure is a function which uses some values that are local but defined outside of the function itself. Closures are a very powerful mechanism used in many places in Opa, in particular for event handlers. |
With this rewrite, the only task we still have ahead of us is changing base_uri so that it uses options specified on the command-line or in an option file. For both purposes, Opa offers a module CommandLine:
uri_for_topic =
(
default_uri =
{Uri.default_absolute with
domain = "localhost"
schema = {some = "http"}
}
base_uri =
CommandLine.filter({
title = "Wiki arguments"
init = default_uri
parsers = []
anonymous = []
})
topic -> Uri.of_absolute({base_uri with
path = ["_rest_", topic]
})
)This variant on uri_for_topic calls CommandLine.filter to instruct the option system to take into account a family of arguments to progressively construct base_uri, starting from default_uri. We name this family "Wiki arguments" and we specify its behavior with fields parsers (used for named arguments) and anonymous (used for anonymous arguments) which are both empty for the moment. As long as both fields are empty, this family has no effect and base_uri is always going to be equal to default_uri — we will change this shortly. Also, for the moment, if you compile your application and launch it with command-line argument --help, you will see an empty entry for a family called "Wiki arguments".
Let us add one command-line option (or, more precisely, a command-line parser) to our family, as follows:
port_parser =
{CommandLine.default_parser with
names = ["--wiki-server-port"]
description = "The server port of the REST server for this wiki. By default, 8080."
on_param(x) = parser y=Rule.natural -> {no_params = {x with port = {some = y}}}
}As you can see, a command-line parser is a record (it has type CommandLine.parser), and here, we derive it from CommandLine.default_parser. In this extract, we only specify the bare minimum.
Firstly, a command-line parser should have at least one name, here "--wiki-server-port".
Secondly, Opa needs to know what it should do whenever it encounters something along the lines of "--wiki-server-port foo" on the command-line. This is the role of field on_param. Argument x is the value we are currently building — here, initially, default_uri. The body of this field is a text parser, i.e. a construction that should analyze a text and either extract information or reject it. Here, we just want a non-negative integer (aka a "natural number"), a construction for which the library offers a predefined text parser called Rule.natural. We call the result y.
|
Tip
|
About text parsers
Opa offers a powerful text analysis feature with text parsers. Text parsers have roughly the same role as regular expressions engines found in many web-related languages, but they are considerably more powerful. A text parser is introduced with keyword parser and has a syntax roughly comparable to pattern-matching: parser y=Rule.natural -> //do something with y
| y=Rule.hex -> //do something with y
| "none" -> //...This parser will accept any non-negative integer and execute the first branch, or any hexadecimal integer and execute the second branch, or the character string "none" and execute the third branch. If none of the branches matches the text, parsing fails. The core function for applying a text parser to some text is Parser.try_parse. You can find a number of predefined parsing functions in module Rule. Additional modules offer custom parsing, e.g. Uri.uri_parser. |
The result of on_param must have one of three shapes:
-
\{no_params = v\}, if the option parser does not expect any additional argument and is now ready to produce value v;
-
\{params = v\}, if the option parser expects at least one other argument;
-
\{opt_params = v\}, if the option parser can handle additional arguments but is also satisfied if no such argument is provided.
Here, we expect only one argument after "--wiki-server-port" so we just produce a value with \{no_params = ...\}. As for the result itself, we derive from x the same absolute URI, but with a new content in field port.
We can now define in the exact same manner the command-line parser for the host:
domain_parser =
{CommandLine.default_parser with
names = ["--wiki-server-domain"]
description = "The REST server for this wiki. By default, localhost."
on_param(x) = parser y=Rule.consume -> {no_params = {x with domain = y}}
}The main difference is that we use predefined text parser Rule.consume (which accepts anything) instead of Rule.natural (which only accepts non-negative integers).
Once we have added both our parsers to parsers, we are ready. With a little additional documentation, we obtain:
uri_for_topic =
domain_parser =
{CommandLine.default_parser with
names = ["--wiki-server-domain"]
description = "The REST server for this wiki. By default, localhost."
on_param(x) = parser y=Rule.consume -> {no_params = {x with domain = y}}
}
port_parser =
{CommandLine.default_parser with
names = ["--wiki-server-port"]
description = "The server port of the REST server for this wiki. By default, 8080."
on_param(x) = parser y=Rule.natural -> {no_params = {x with port = {some = y}}}
}
base_uri =
CommandLine.filter(
{title = "Wiki arguments"
init = {Uri.default_absolute with domain = "localhost" schema = {some = "http"}}
parsers = [domain_parser, port_parser]
anonymous = []
}
)
topic -> Uri.of_absolute({base_uri with path = ["_rest_", topic]})This completes our REST client. We now have a full-featured REST client that can also act as a server and supports command-line configuration.
The full source code follows:
link:hello_wiki_rest_client_customizable.opa[role=include]Modify the wiki so that it acts both as a database-backed wiki and as a REST client:
-
by default, behave as the REST client wiki;
-
whenever information is downloaded from the REST server, store the information to the local database;
-
whenever information is updated locally, store the information to the local database and upload it to the REST server;
-
if connection fails for some reason, fallback to the database.
Modify the wiki of the previous exercise so that:
-
the REST server can be specified from the command-line;
-
if no server is specified from the command-line, it behaves exactly as the non-REST wiki;
-
otherwise, behave as the wiki of the previous exercise.
|
Tip
|
Using tuples
For this exercise, you may need to define not just one function using the command-line but several. In this case, it will probably be interesting to use a tuple definition, such as (a, b) =
x = 50
(x, x+1)This tuple definition defines both a = 50 and b = 51. You can, of course, use more complex expressions instead of 50. |
Architecting a REST chat ^^^^^^^^^^ How would you design a chat distributed among servers using only REST for communications between servers?
|
Tip
|
A REST chat?
While it is definitely possible to write a REST-based chat in Opa, this is not the preferred way of implementing a multi-server application. But it is an interesting exercise, if only to experience the contrast between manual REST-style distribution and automated Opa-style distribution. |