hello_recaptcha.omd

//[[hello_recaptcha]]
Hello, reCaptcha, and the rest of the world
===========================================

In this chapter, we will see how to plug an external API to the Opa platform --
here, Google reCaptcha, an API used to protect forms against spammers by
attempting to determine whether the person filling the form is actually a human
being. Along the way, we will introduce the Opa Binding System Library (or BSL)
and some of the mechanisms provided by Opa to permit modular, safe programming.

As we will be interacting with JavaScript, some notions of the JavaScript syntax
are useful to understand this chapter.

Overview
--------

As in previous chapters, let us start with a picture of the application we will develop in this chapter:

![Final version of the Hello reCaptcha application](/resources/manual/img/hello_recaptcha/result.png)

This simple application prompts users to recognize words that are considered too
difficult to read by a computer in the current state of the art of character
recognition -- typically because Google's own resources have failed to make
sense of these words -- and determines that a user is indeed a human being if
the answer corresponds to that of sufficient other users through the world.
The feature is provided by Google, as an API called reCaptcha. This API involves
some code that must be executed on the client (to display the user interface
and offer some interactivity) and some code that must be executed on the
server (to contact Google's servers and check the validity of the answer).

If you are curious, this is the full source code of our application:

[opa|fork=hello_recaptcha|run=http://recaptcha.tutorials.opalang.org]file://hello_recaptcha/hello_recaptcha_app.opa

Of course, since the features are provided by Google reCaptcha, the most
interesting aspects of the code are not to be found in the source of the
application itself, but in how we define the binding of reCaptcha for Opa.

This is done in two parts. At high-level, we find the Opa package:

[opa]file://hello_recaptcha/hello_recaptcha.opa

At low level, we find the JavaScript binding:

[js]file://hello_recaptcha/recaptcha.js

In the rest of the chapter, we will walk you through all the concepts and
constructions introduced in these listings.

Populating the BSL
------------------

The [documentation of the reCaptcha API](http://code.google.com/apis/recaptcha/docs/display.html)
details five methods of a JavaScript object called `Recaptcha`,
that should be called at distinct stages of the use of the API. Our first step
will therefore be to bind each of these methods to Opa, through the mechanism of
the [Binding System Library](/manual/Hello--bindings----Binding-other-languages).

{block}[TIP]
### About the BSL
The Binding System Library was developed to allow _binding_ external features
to Opa. The mechanism is used at low-level in Opa itself, to bind Opa to
browser features, but also to server features.

At the time of this writing, the BSL provides native bindings with JavaScript
and OCaml, and can be used to bind to most languages, including C.

This chapter shows how to plug Opa with some Javascript client code.
The chapter about the [BSL](/manual/Hello--bindings----Binding-other-languages) will also show how to
[bind Opa and Ocaml](/manual/Hello--bindings----Binding-other-languages/Binding-Ocaml-and-Javascript), and
[how to use C primitives from Opa](/manual/Hello--bindings----Binding-other-languages/Binding-C).
{block}

For this purpose, we will create a file called `recaptcha.js`, and that we
will populate with Opa-to-JavaScript bindings.

Let us start with initialization. The documentation states that any use of
the JavaScript API must initialize the `Recaptcha` object as follows:

```
Recaptcha.create(pubkey,
   id,
   {
      theme: theme,
      callback: Recaptcha.focus_response_field
   }
);
```

where `pubkey` is a public key obtained from the reCaptcha admin site,
`id` is the identifier of the UI component that will host the reCaptcha
and `theme` is the name of the visual theme.

For our purpose, as we are binding the API, these three values should
be function arguments, for a function accepting three character strings
and returning nothing meaningful.

We specify this as follows:

###### Binding initialization (extract of `recaptcha.js`)

```
##register init: string, string, string -> void
##args(id, pubkey, theme)
{
  Recaptcha.create(pubkey,
      id,
      {
        theme: theme,
        callback: Recaptcha.focus_response_field
      }
  );
}
```

The first line registers a function called `init` (we could have called it
`create`, as the JavaScript method, but `init` fits better with the Opa
conventions). This function is implemented in JavaScript and its type
is specified as `string, string, string -> void`.

The second line gives names to arguments, respectively `id`, `pubkey`
and `theme`. If you are familiar with JavaScript, you can think of
BSL keyword `##args` as a counterpart to `function`, with stricter
checks.

The rest of the extract is regular JavaScript, copied directly from the
documentation of reCaptcha.

{block}[CAUTION]
### About `void`
The only surprise may be that there is no final
`return`. Indeed, in regular JavaScript, functions with no `return` value actually
return `undefined`. By opposition, Opa is stricter and does not allow
`undefined` values. Since our definition states that `init` always
returns a `void`, the BSL will apply an automatic transformation
of the Javascrit code so that once bined in Opa, the function returns
`void`.

In Opa, functions always return a value, even if this value is `void`. This is
true even of functions implemented in JavaScript.

Therefore, if a BSL JavaScript function has type `... -> void`, once in Opa,
it returns `void`.
{block}

The rest of the API is quite similar. Functions `reload` and `destroy`,
which serve respectively to display a new challenge or to clean the
screen once the reCaptcha has become useless, are bound as follows:

###### Binding `reload`, `destroy` (extract of `recaptcha.js`)

```
##register reload: -> void
##args()
{
    Recaptcha.reload();
}

##register destroy: -> void
##args()
{
    Recaptcha.destroy();
}
```

These bindings should not surprise you. Simply note that we write `##args()`
if a function does not take any argument.

Binding functions `get_challenge` and `get_response` is quite similar.  The
first of these functions returns an opaque string that can be used by the
reCaptcha server to determine which image has been sent to the user. The
second returns the text entered by the user.

###### Binding `get_challenge` and `get_response` (extract of `recaptcha.js`)

```
##register get_challenge: -> string
##args()
{
    return (Recaptcha.get_challenge()||"")
}

##register get_response: -> string
##args()
{
    return (Recaptcha.get_response()||"")
}
```

Note that we do not return simply `Recaptcha.get_challenge()` or
`Recaptcha.get_response()`. Indeed, experience with the reCaptcha API shows
that, in some (undocumented) cases, these functions return value `null`,
which is definitely not a valid Opa `string`. For this purpose, we normalize
the `null` value to the empty string `""`.

{block}[CAUTION]
### About `null`
In Opa, the JavaScript value `null` is meaningless. An Opa function implemented
as JavaScript and which returns `null` (or an object in which some fields are `null`)
is a programmer error.
{block}

With this, the source code for the [BSL bindings](/manual/Hello--bindings----Binding-other-languages) is complete. Before proceeding
to the Opa side, we just need to compile this source code:

    opa-plugin-builder recaptcha.js -o recaptcha

//For more details about _opa-plugin-builder_, you can refer to <|opa_plugin_builder, its documentation|>.

We are now done with JavaScript.

Typing the API
--------------

The next step is to connect the BSL to the server component and wrap the
features as a nice Opa API.

Looking at the documentation of reCaptcha, we may see that a reCaptcha
accepts exactly three meaningful arguments:

* a private key, which we need to obtain manually from [reCaptcha](https://www.google.com/recaptcha/admin/create), and which should never be transmitted to the client, for security reasons;
* a public key, which we obtain along with the private key;
* an optional theme name.

We group these arguments as a record type, as follows:

###### The reCaptcha configuration

```
type Recaptcha.config =
{
   {
     string privkey
   }
   cfg_private,

   {
      string pubkey,
      option(string) theme
   }
   cfg_public
}
```

By convention, records which simply serve to group arguments under meaningful
names are called _configurations_ and their name ends with `.config`. Here,
in order to avoid confusions, we have split this record in two subrecords,
called respectively `cfg_private`, for information that we want to keep on the
server, and `cfg_public`, for information that can leave the client without
breaching security.

Also, looking at the
[documentation of the reCaptcha server-side API](http://code.google.com/apis/recaptcha/docs/verify.html),
we find out that communications with the reCaptcha
server can yield the following results:

* a success;
* a failure, which may either mean that the user failed to identify the text, or that some other issue took place, including a communication error between Google servers.

Two additional error cases may appear:

* a communication error between your application and Google (typically, due to a network outage);
* a message returned by Google which does not match the documentation (unlikely but possible);
* an empty answer provided by the user, in which case communication should not even take place.

While these distinct results are represented as strings in the API, in Opa,
we will prefer a sum type, which we define as follows:

###### reCaptcha results

```
type Recaptcha.success = {captcha_solved} /**The captcha is correct.*/
type Recaptcha.failure =
    { WebClient.failure captcha_not_reachable } /**Could not reach the distant server.*/
 or { string upstream }      /**Upstream failure. Could be a user error, but the code is not meant to be exploited.*/
 or { list(string) unknown } /**Server could be reached, but produced an error that doesn't match the specifications
                                 provided by Google. Possible cause: proxy problem.*/
 or { empty_answer }         /**Recaptcha guidelines mention that we should never send answers that are empty.*/

type Recaptcha.result = { Recaptcha.success success } or { Recaptcha.failure failure }
```

And finally, we define one last type, that of the reCaptcha _implementation_, as follows:

###### Type of the reCaptcha implementation (first version)

```
type Recaptcha.implementation = {
      /**Place a request to the reCaptcha server to verify that user entry is correct.
        @param challenge
        @param response
        @param callback*/
     (string, string, (Recaptcha.result -> void) -> void) validate,
      /**Reload the reCaptcha, displaying a different challenge*/
     (-> void) reload,
      /**Destroy the reCaptcha*/
     (-> void) destroy
}
```

The role of this type is to encapsulate the functions of the reCaptcha after construction.

This type offers three fields. The first two will map respectively to the
`reload` method we have bound earlier and to the `destroy` method we have bound
earlier. The third, `validate`, is a more powerful function whose role will be
to send to the reCaptcha server the challenge, the response entered by the user,
to wait for the server response and to trigger a function with the result.

{block}[TIP]
### Objects in Opa
Opa is not an Object-Oriented Programming Language in the traditional sense of the term.
However, it is a _Higher-Order Programming Language_, which means among other things that
all the major features of Object-Oriented Programming can be found in Opa, and more.

In our listing, values of type `Recaptcha.implementation` contain fields, some
of which are functions -- not unlike Objects in OO languages may contain fields
and methods.

Although this is a slight misues of the term, we often call such records, that
is records containing fields, some of which are functions, _Objects_.
{block}

Now that the types are ready, we can write the functions that manipulate them.

Implementing the API
--------------------

The documentation of reCaptcha mentions a JavaScript library, provided by reCaptcha, which needs to be loaded
prior to initializing the reCaptcha. We handle this in a function `onready` which we will use shortly:

###### Function `onready` (first version)

```
function void onready(string id, string pubkey, string theme)
{
   Client.Script.load_uri_then(path_js_uri,
     function()
     {
       (%% Recaptcha.init %%)(id, pubkey, theme)
     }
   )
}
```

This function makes use of `Client.Script.load_uri_then`, a function provided by
the library to load a distant script and, once loading is complete, to invoke a
second function. First argument, `path_js_uri`, is a constant representing the
URI at which the JS is available. We will define it a bit later.  The second
argument is a function that makes use of a construction you have never seen:

    (%% Recaptcha.init %%)

This is simply function `init`, as we defined it a few minutes ago in JavaScript.

{block}[TIP]
### Calling the BSL
To use an Opa value defined in the BSL (that is, in JavaScript, C, OCaml, or any
other language), the syntax is

    (%% NameOfThePlugin.name_of_the_value %%)

Replace `NameOfThePlugin` by the name of the file you have passed
to `opa-plugin-builder` and `name_of_the_value` by the name you have
registered with `##register`. Capitalization of plug-in names is ignored.

The contains between the two `%%` is called the _key_ of the external primitive.
A part of the binding system reference details
[how keys are associated to primitives](/manual/Hello--bindings----Binding-other-languages/key_normalization).
{block}

In other words, this function `onready` loads the JavaScript provided by Google,
then initializes the reCaptcha. To call this function, we will make use of a
`Recaptcha.config` and a `ready` event, to ensure that it is called only on a
browser that actually makes use of the reCaptcha. We will need this
initialization in our constructor for `Recaptcha.implementation`.

{block}[TIP]
### Multiple loads
Module `Client.Script` provides several functions for loading JavaScript.
These functions ensure that a JavaScript URI will only be loaded once.
In other words, you do not have to worry about the same JavaScript being
loaded and executed multiple times.
{block}

The second important function, which we will also need to build our
`Recaptcha.implementation`, is the validation. We implement it as follows:

```
function validate(challenge, response, (Recaptcha.result -> void) callback)
{
  //By convention, do not even post a request if the data is empty
  if (String.is_empty(challenge) || String.is_empty(response))
  {
    callback({failure: {empty_answer}})
  }
  else
  {
    /**POST request, formatted as per API specifications*/
    data = [ ("privatekey", privkey)
           , ("remoteip",   "{HttpRequest.get_ip()?(127.0.0.1)}")
           , ("challenge",  challenge)
           , ("response",   response)
           ]
    /**Handle POST failures, decode reCaptcha responses, convert this to [reCaptcha.result].*/
    function with_result(res)
    {
      match (res)
      {
        case ~{failure}:
          callback({failure: {captcha_not_reachable: failure}})
        case ~{success}:
          details = String.explode("\n", success.content)
          match (details)
          {
            case ["true" | _]: callback({success: {captcha_solved}})
            case ["false", code | _]: callback({failure: {upstream: code}})
            default: callback({failure: {unknown: details}})
          }
      }
    }
    /**Encode arguments, POST them*/
    WebClient.Post.try_post_with_options_async(path_validate_uri,
          WebClient.Post.of_form({WebClient.Post.default_options with content: {some: data}}),
          with_result)
  }
}
```

Although this listing uses several functions that you have not seen yet, it should not surprise you.
The first part responds immediately if the challenge or the response is negative. This complies with
the specification of reCaptcha, in addition to saving precious resources on your server. We then
build `data`, a list of association of the arguments expected by the reCaptcha API, and `with_result`,
a function that handles the return of our `{post}` request by analyzing the resulting `string`.
Note the use of function `WebClient.Post.of_form`, which converts a request using a list of
associations, as are built by HTML forms, into a raw, `string`-based request. Also note function
`String.explode`, which splits a string in a list of substrings.

From `validate`, as well as our JavaScript implementations of `reload` and `destroy`, we may now
construct our `Recaptcha.implementation`, as follows:

```
function Recaptcha.implementation make_implementation(string privkey)
{
   function validate(challenge, response, (Recaptcha.result -> void) callback)
   {
     //By convention, do not even post a request if the data is empty
     if (String.is_empty(challenge) || String.is_empty(response))
     {
       callback({failure: {empty_answer}})
     }
     else
     {
       /**POST request, formatted as per API specifications*/
       data = [ ("privatekey", privkey)
              , ("remoteip",   "{HttpRequest.get_ip()?(127.0.0.1)}")
              , ("challenge",  challenge)
              , ("response",   response)
              ]
       /**Handle POST failures, decode reCaptcha responses, convert this to [reCaptcha.result].*/
       function with_result(res)
       {
         match (res)
         {
           case ~{failure}:
             callback({failure: {captcha_not_reachable: failure}})
           case ~{success}:
             details = String.explode("\n", success.content)
             match (details)
             {
               case ["true" | _]: callback({success: {captcha_solved}})
               case ["false", code | _]: callback({failure: {upstream: code}})
               default: callback({failure: {unknown: details}})
             }
         }
       }
       /**Encode arguments, POST them*/
       WebClient.Post.try_post_with_options_async(path_validate_uri,
             WebClient.Post.of_form({WebClient.Post.default_options with content: {some: data}}),
             with_result)
     }
   }
   {~validate, reload:cl_reload, destroy:cl_destroy}
}
```

With this function, we implement a new function `make`, to construct both the
`Recaptcha.implementation` and the user interface `xhtml` component that connects
to this implementation:

```
function (Recaptcha.implementation, xhtml) make(Recaptcha.config config) {
    id = Dom.fresh_id();
    xhtml = <div id={id} onready={function(_) { onready(id, config.cfg_public.pubkey, config.cfg_public.theme?"red") }}/>;
    (make_implementation(config.cfg_private.privkey), xhtml);
}
```

One more utility function will be useful, to read the user interface and clean it up immediately:

```
function {string challenge, string response} get_token() {
    result = ({ challenge: (%%Recaptcha.get_challenge%%)()
                , response: (%%Recaptcha.get_response%%)()
              });
    (%%Recaptcha.destroy%%)();
    result;
}
```

With this function, we now have exposed all the features we need to use a
reCaptcha. However, at this stage, we are exposing a great deal of the
implementation. Consequently, our next step will be to make the implementation
abstract and to encapsulate the features in a module.

Modularizing the features
-------------------------

It is generally a good idea to split large (or even small) projects into _packages_,
as follows:

###### Wrapping our code as a package

    package tutorial.recaptcha

{block}[TIP]
### About packages
In Opa, a _package_ is a unit of compilation and abstraction. Packages
can be distributed separately as compiled code, packages can hold private
values, abstract types, etc.

The following declaration states that the current file is part of
package `package_name`:

    package package_name

Conversely, the following declaration states that the current file makes
use of package `package_name`:

    import package_name

Generally, in Opa, packages are assembled into package hierarchies,
using reverse domain notation.
{block}

Now that we have placed our code in a package, we may decide that
some of our types are _abstract_, by adding an `@abstract` directive as follows:

```
abstract type Recaptcha.implementation = { ... }
```

{block}[TIP]
### About _abstract_ types

An _abstract_ type is a type whose _definition_ can only be used in the
package in which it was defined, although its _name_ might be used outside of the
package.

This feature provides a very powerful mechanism for avoiding errors and ensuring
that data invariants remain unbroken, but also to ensure that third-party developers
do not base their work on characteristics that may change at a later stage.

Consider the following example:

```
package example

abstract type cost = float
```

In this example, type `cost` is defined as a synonym of `float`. In package
`example`, any `float` can be used as a `cost` and reciprocally.  However,
outside of package `example`, `cost` and `float` are two distinct types.  In
particular, we have ensured that only the code of package `example` can create
new values of type `cost`. If our specifications expect that a `cost` is always
strictly positive, we have succesfully restriced the perimeter of the code we
need to check to ensure that this invariant remains unbroken: only package
`example` needs to be verified.
{block}

With this change, methods `validate`, `destroy` and `reload` can now be called only
from our package. As these methods offer important features, we certainly wish to
provide some form of access to the methods, as follows:

###### Exporting features (first version)

```
function void validate(Recaptcha.implementation implementation, (Recaptcha.result -> void) callback) {
    t = get_token();
    implementation.validate(t.challenge, t.response, callback);
}

function void reload(Recaptcha.implementation implementation) {
    implementation.reload();
}

function void destroy(Recaptcha.implementation implementation) {
    implementation.destroy();
}
```

For further modularization, we will group all our functions as a _module_, and take the opportunity
to hide the function and constants that should not be called from the outside of the module:

```
module Recaptcha
{
   function (Recaptcha.implementation, xhtml) make(Recaptcha.config config) {
       id = Dom.fresh_id();
       xhtml = <div id={id} onready={function(_) { onready(id, config.cfg_public.pubkey, config.cfg_public.theme?"red") }}/>;
       (make_implementation(config.cfg_private.privkey), xhtml);
   }

   function void validate(Recaptcha.implementation implementation, (Recaptcha.result -> void) callback) {
       t = get_token();
       implementation.validate(t.challenge, t.response, callback);
   }

   function void reload(Recaptcha.implementation implementation) {
       implementation.reload();
   }

   function void destroy(Recaptcha.implementation implementation) {
       implementation.destroy();
   }

   private path_validate_uri =
        Option.get(Parser.try_parse(Uri.uri_parser, "http://www.google.com/recaptcha/api/verify"));

   private path_js_uri =
        Option.get(Parser.try_parse(Uri.uri_parser, "http://www.google.com/recaptcha/api/js/recaptcha_ajax.js"));

   private function void onready(string id, string pubkey, string theme) {
      Client.Script.load_uri_then(path_js_uri,
        function()
        {
          (%% Recaptcha.init %%)(id, pubkey, theme)
        }
       );
   }

   private cl_reload =
     %%Recaptcha.reload%% /**Implementation of [reload]*/

   private cl_destroy =
     %%Recaptcha.destroy%% /**Implementation of [destroy]*/

   private function Recaptcha.implementation make_implementation(string privkey) {
      function validate(challenge, response, (Recaptcha.result -> void) callback) {
        //By convention, do not even post a request if the data is empty
        if (String.is_empty(challenge) || String.is_empty(response)) {
          callback({failure: {empty_answer}})
        } else {
          /**POST request, formatted as per API specifications*/
          data = [ ("privatekey", privkey)
                 , ("remoteip",   "{HttpRequest.get_ip()?(127.0.0.1)}")
                 , ("challenge",  challenge)
                 , ("response",   response)
                 ]
          /**Handle POST failures, decode reCaptcha responses, convert this to [reCaptcha.result].*/
          function with_result(res) {
              match (res) {
              case ~{failure}:
                  callback({failure: {captcha_not_reachable: failure}})
              case ~{success}:
                  details = String.explode("\n", success.content)
                  match (details) {
                  case ["true" | _]: callback({success: {captcha_solved}})
                  case ["false", code | _]: callback({failure: {upstream: code}})
                  default: callback({failure: {unknown: details}})
                  }
              }
          }
          /**Encode arguments, POST them*/
          WebClient.Post.try_post_with_options_async(path_validate_uri,
                WebClient.Post.of_form({WebClient.Post.default_options with content: {some: data}}),
                with_result)
        }
      }
      {~validate, reload:cl_reload, destroy:cl_destroy}
   }

   private function {string challenge, string response} get_token() {
     result = ({ challenge: (%%Recaptcha.get_challenge%%)()
               , response: (%%Recaptcha.get_response%%)()
               });
     (%%Recaptcha.destroy%%)();
     result;
   }

}
```

{block}[TIP]
### About modules
A _module_ is an extended form of record introduced with `module ModuleName { ... }` instead of `{ ... }`.

Modules offer a few syntactic enrichments:
- any field may be declared as `private`, which forbids from using it outside of the module;
- fields can be declared in any order, even if they have dependencies (including circular dependencies).

In addition, modules are typed slightly differently from regular records. We will detail this in
another chapter.
{block}

And with this, our binding is complete.

We may compile our package with

    opa recaptcha.opp hello_recaptcha.opa

Let us recapitulate the Opa source code:

[opa|fork=hello_recaptcha|run=http://recaptcha.tutorials.opalang.org]file://hello_recaptcha/hello_recaptcha.opa

Testing the API
---------------

To test the API, we may write a simple application:

[opa]file://hello_recaptcha/hello_recaptcha_app.opa

With the exception of directives `server protected`, this listing should not
surprise you. Here, we placed directives `server protected` as a sanity check, to
be absolutely certain that the compiler would keep this value on the server
and not expose it to the clients.

Note that the public and private key provided here are registered for domain
"example.com". They will work for the example, but should you wish to use the
reCaptcha, you should register your own public/private key pair.

We may now compile the application, as usual

    opa recaptcha.opp hello_recaptcha.opa

Questions
---------

### Why an object?

As mentioned, we have defined `Recaptcha.implementation` as an object. This is a
good reflex when extending the Opa platform through additional BSL bindings that
use data structures can be implemented only on one side.

In Opa, data can be transmitted transparently between the client and the server.
This is impossible for data that is meaningful only on the client. This is the
case here, as JavaScript object `Recaptcha`, by definition, exists only on the
client. However, wrapping the JavaScript data structure and the functions that
manipulate it as an object ensures that the user only ever needs to access
methods -- and such methods can always be transmitted from the client to the
server, making the object data structure side-independent.

{block}[TIP]
### Making objects
When a BSL extension to the Opa platform introduces a data structure implemented
only on one side, the user _must_ never manipulate this data structure directly.
_Always_ hide this data structure behind an object, whose only fields are functions.
{block}