README.mlidl

-*-text-*-

MLIDL
=====

This  is a program which takes  as  input a file containing  a  (limited) set of
OCaml type definitions plus some annotations and generates code in OCaml and OPA
to transmit  these values between networked instances.    In theory, the encoded
types should be compatible with the current JSON types and could be used to send
data to javascript endpoints but this has not been tested.

The mlidl program
=================

Recognised types
----------------

This  takes as input  the following subset  of OCaml types, defined in pseudo-ML
types:

<code>
  open MLIDL_Module

  external external_ty : Mod.external_type = "<complex set of implementation functions>"

  type ty =
     unit
   | int
   | float
   | bool
   | string
   | ty option
   | ty list
   | (ty * ty * ... )
   | { lab1:ty; lab2:ty; ... }
   | (Cons1 of ty | Cons2 of ty | Cons3)
   | MLIDL_Module.ty
   | external_ty
</code>

Note that there are no:

  - Type variables: 'a ty
  - References: ty ref
  - Functional types: ty -> ty -> ...
  - Type labels: ~label:ty

Although some support is  planned for a simplified  form of type variables which
will require a complex set of definitions as used by the external type syntax.

You can import types from other IDL files with the  syntax indicated above.  You
have  to "open" the  name of the  generated module by the  included IDL file and
then prefix *all* of the types you import from other files.

There  is   also provision  for importing   externally-defined  types from OCaml
modules but the syntax for this  is highly complex.   Simply put, you have to do
all  the work done  for you by the  IDL code generator and  indicate  to the IDL
function names which are the equivalent of its own generated functions for these
types.  This is a lot of work so you should maybe consider using the IDL program
to generate these types and then use the import mechanism if  you have access to
the sources of these types.  One  point, however, is  that this mechanism allows
you  to define a completely separate  type representation  between the OCaml and
OPA types.  See the "External Types" section below for more details.

Basic functionality generated
-----------------------------

The types are translated more-or-less literally into OCaml types  on the ML side
and  equivalent types on   the OPA  side.   The  OCaml  output code  includes an
interface file.  The following functions are generated:

  String input/output
  ...................

    Firstly, for OCaml this functionality is wrapped  in functors (MakeInput and
    MakeOutput) where the input and output functions are abstract.  However, for
    convenience, an instantiation  of these modules  is provided  generating the
    functions:

      - <type>_of_string
      - string_of_<type>

    for each <type> in  the  IDL file.  These   are  the functions used by   the
    network  layer code.  If  you wish to  output to  Buffer.t  or whatever then
    instantiate the functors.

    For OPA the output functions are called:

      - output_<type>

    and are wrapped in a module called STR_<name>, where <name> is the base name
    of  the IDL  file.   There is  currently  no input_<type>  functionality for
    native OPA (due to the slowness of OPA parsers).

    For BSL output, the <type>_of_string and string_of_<type> functions are made
    available to OPA  where  the  types  remain  external OCaml  types.    Note,
    however, that the  wrap_<type> and  unwrap_<type>  functions used by   these
    routines are also made available to OPA.

    Generation of these functions can be prevented with the option:

    --string-functions false

  JSON input/output
  .................

    These are functions to  convert the types  into JsonTypes types in OCaml and
    to RPC.Json.json types for OPA:

      - tojson_<type>
      - fromjson_<type>

    These functions are   available  for both  OCaml  and Native  OPA.  The  OPA
    functions are included in a module called "JSON_<name>".

    Currently, the BSL generated code does not include JSON conversion.

    There used to be support for low-level JSON types in OPA (OCaml JsonTypes as
    external types in the  OPA file) but  this has fallen out  of date since the
    Json types were updated.  It could be resurrected on request.

    You can individually defeat  the tojson_<type> and fromjson_<type> functions
    with:

    --tojson-functions false
    --fromjson-functions false

  Create types
  ............

    There is primitive  support for generating types from  constant types in the
    constructors:

      IDL file:    type date = { year : int; month : int; day : int; }

      MLI file:    val create_date : int -> int -> int -> date

      Application: let date = create_date 1961 8 24;;

    These can be switched off by:

      --create-functions false

Network layer code generated
----------------------------

The IDL code generator also understands a small set of annotations (disguised as
OCaml  "val" statements) which indicate  that  it should generate  code for  the
network layer communications which use the basic conversion functions:

<code>
val sendreceive_profile : profile -> profile
val send_date : hare
val receive_gender : gender
val protocol_cat : cat -> string
val responder_dog : dog -> string
</code>

Don't be fooled by  the ML-like syntax of  these annotations, there are  no such
values actually generated by the IDL code generator.

The most important  of these is the "responder"  annotation which generates  all
the code  needed to implement  a  server/client combination between two  (Hlnet)
endpoints.  The other routines provide low-level network operations which may be
useful where the responder pattern is not what is required.   A brief summary of
these operations:

 - "val sendreceive_<name> : <type1> -> <type2>"

   This is an Hlnet.sendreceive  call where the  types are encoded  according to
   the encoding defined in the IDL file (currently there is only one encoding).

 - "val send_<name> : <type>"

   Generates an Hlnet.send call with the encoded type.

 - "val receive_<name> : <type>"

   Generates an Hlnet.receive call with the encoded type.

 - "val protocol_<name> : <type1> -> <type2>"

   Generates an  Hlnet.Aux.easy_protocol   value with  query type   <type1>  and
   response type <type2>.  Note that you can set the name and version number for
   the   protocol by annotations  in the  IDL file.  Note   also that if you set
   either type to "string" then no type encoding is applied to that type.

 - "val responder_<name> : <type1> -> <type2>"

   This simple statement  generates all   the  apparatus needed to  implement  a
   server/client pair using the indicated  types.  See the "Responders"  section
   below for details.

Command line and in-built arguments
-----------------------------------

The output of the IDL file can be controlled by either command-line arguments or
with annotations built into the IDL files themselves.

One important  point  is that the  in-built parameters  take precedence.  If you
wish to  control  the  output from  the  command  line,  the  in-built  argument
equivalents must not be present.  Here are the command line arguments:

  -output-suffix <string>       Output suffix (default: "types").

    The generated files (ml,  mli, opa, bsl) will  all have this suffix appended
    to the base name of the IDL file (eg. test.mlidl -> testtypes.ml).

  -bsl-prefix <string>          Bsl prefix (default: "bsl").

    This prefix will be prepended to the base name of generated BSL files.  This
    is to prevent overwriting the ml files of the  same name.  Note that if your
    code is not destined for the OPA BSL mechanism, a more suitable prefix might
    be "plugin".

  -ocaml-wrap-options <bool>    Wrap option around OCaml input (default: "true").
  -opa-wrap-options <bool>      Wrap option around Opa input (default: "true").

    Redundant, do not use.

  -native-parser <bool>         Use native parser instead of TRX wrappers (default: "true").

    Redundant, do not use.

  -hlnet-logging <bool>         Add logger statements to Hlnet wrappers (default: "false").

    Network-layer code can optionally generate debug  logging message, this flag
    switches this functionality on and off.

  -logger-function <string>     Logger function (default: "Logger.log").

    The logger function to use for the --hlnet-logging feature.

  -protocol-version <int>       Protocol version number (default: 1).

    The protocol number to install  in any protocols  generated for the  network
    layer.

  -default-port <int>           Default port number (default: 49152).

    The default port to be built into the network layer endpoint description.

  -default-addr <string>        Default inet address (default: "Unix.inet_addr_loopback").

    The default address for the endpoint.

  -create-functions <bool>      Output create value from type functions (default: true).

    Add the create functions to the output.

  -tojson-functions <bool>      Output type to json functions (default: true).
  -fromjson-functions <bool>    Output type from json functions (default: true).

    Enable/disable JSON output/input functions.

  -string-functions <bool>      Output type to/from string functions (default: true).

    Enable/disable to/from string functions.

  -bsl-file <bool>              Output BSL file for OCaml functions (default: true).

    Generate BSL wrapper code.  Note: you need  -no-ocaml to be false since only
    the wrappers are generated, not the referenced OCaml code.

  -no-ocaml <bool>              Don't generate OCaml output (default: false).

    Do not generate OCaml output.

  -no-opa <bool>                Don't generate OPA output (default: false).

    Do not generate native OPA output.

The in-built option have the following syntax:

  let module_name = "test2"
  let protocol_version = 2
  let verbose = true

The value must be of the correct type.  Mostly these are exactly the same as the
command-line arguments:

  In-built argument   Type    Command-line option
  -----------------   ------  -------------------
  module_name         String  <none>
  output_suffix       String  --output-suffix       
  bsl_prefix          String  --bsl-prefix          
  encoding_number     Int     --encoding-number     
  ocaml_wrap_opt      Bool    --ocaml-wrap-opt      
  opa_wrap_opt        Bool    --opa-wrap-opt        
  native_parser       Bool    --native-parser       
  hlnet_logging       Bool    --hlnet-logging       
  logger_function     String  --logger-function     
  protocol_version    Int     --protocol-version    
  default_port        Int     --default-port        
  default_addr        String  --default-addr        
  create_functions    Bool    --create-functions    
  tojson_functions    Bool    --tojson-functions    
  fromjson_functions  Bool    --fromjson-functions  
  string_functions    Bool    --string-functions    
  bsl_file            Bool    --bsl-file            
  no_ocaml            Bool    --no-ocaml            
  no_opa              Bool    --no-opa              
  verbose             Bool    --v             
  debug               Bool    --g               

The "module_name" argument does not appear on the command line since the name of
the output module will apply  to each IDL file, it  would not make sense to  set
the same module name for all included files as well.

External Types
==============

This is  a difficult feature to  use but is  potentially rewarding in that, once
you  have  generated all the  necessary support  code  by  hand, you can include
external types in your IDL files with the same status as IDL types.

To define an external type, use:

  external ip : Ip.ip = "[(<name>,<value>);...]"

This defines  an   IDL type called  "ip"  with  OCaml  type Ip.ip.   The  string
definition is   an assoc list of  name-value  pairs which  define the   names of
functions to be used as replacement functions for those not generated by the IDL
code generator.  Note that the type of the OPA-side value is also defined in the
assoc list.  Here is the current list of names:

  Name           Type of value
  -------------  -------------
  ocamlstringof  <type> -> string
  ocamlofstring  string -> <type>
  ocamltojson    <type> -> JsonTypes.json
  ocamlfromjson  JsonTypes.json -> <type>

  bslwrap        <type> -> <opatype>
  bslunwrap      <opatype> -> <type>

  opatype        name of the OPA type
  opastringof    <opatype> -> string
  opaofstring    string -> <opatype>
  opatojson      <opatype> -> RPC.Json.json
  opafromjson    RPC.Json.json -> <opatype>
  opatojsonll    <opatype> -> (external) JsonTypes.json
  opafromjsonll  (external) JsonTypes.json -> <opatype>

Notes
-----

  1) The text provided by the <value> string is  technically OCaml code but note
     that you can't currently use complex code here because the string is parsed
     very simply with  String.slice, so you  can't have any semicolons or commas
     in your ML code!   The above  table  indicates the  types of the  resulting
     OCaml text.

  2) The <opatype>  type need in no  way  correspond to  the  <type> type.  This
     allows matching of unrelated  types (the Ip.ip   type is a good example  of
     this).

  3) For an  example  of the implementation  of  this  code, see the   ip.ml and
     ipopa.opa files in libnet/tests.  (Note: don't call your .ml and .opa files
     by the same name!).

  4) It is  unlikely   that you  will need   all of the    above values for  any
     particular application.  The IDL code generator  will inform you if you are
     missing any of the functions for a particular output code.

Compiling and running the IDL code generator
============================================

The IDL code  generator executable  is called  mlidl.native and  is installed by
build_tools into _build/protocols/mlidl.native.

The command itself  simply takes the name of  a single  IDL file (the  extension
should be .mlidl) but note that it may start reading  included IDL files.  These
should already have been  processed by the  IDL code generator before processing
the outer IDL file.

Compiling the  output  is somewhat problematical since  the  dependencies in the
generated  code are  quite  complex.  Note that  you need  to  include  any file
dependencies  on the command line,  particularly, the ML  and BSL  files for any
included types plus have the libraries  for any external types.  Hopefully, this
should be handled by opalang/bld but this is untested.

Responders
==========

This is currently  the main pay-off  for the IDL  code  generator.  From  a very
simple description file  it  is possible  to  generate  code which  reduces  the
complex business  of  managing  the transfer   of  typed data  from  one network
endpoint to another.  The IDL annotation:

  val responder_kind : kind -> string

generates the following values in the OCaml output file:

  val protocol_kind : (kind,string) Hlnet.protocol

  val entrypoint_protocol_kind : (unit option, (kind,string) Hlnet.channel) Hlnet.protocol

  val port_kind : int ref
  val addr_kind : Unix.inet_addr ref
  val endpoint_kind : Hlnet.endpoint ref
  val scheduler_kind : Scheduler.t ref

  val init_responder_kind : int -> Unix.inet_addr -> Scheduler.t -> unit

These allow the specification of endpoints for the resulting network layer.  The
port number and network address are actually used to define  the endpoint.  They
are included separately    so  they can  be   read  back more  easily.  Use  the
init_responder_kind  function  to set  all the  endpoint values  in  one go, for
example:

  init_responder_kind 12345 Unix.inet_addr_loopback Scheduler.default 

The endpoint  and protocol values  can then  be used  by your own communications
code but the IDL also generates server and client code which uses these values:

  val respond_server_kind :
    ('a * 'b) -> (('a * 'b) -> kind -> 'a * 'b * string option * bool) -> unit

  val respond_client_kind :
    'a -> ('a -> ('a -> kind option * ('a -> string -> ('a -> bool -> unit) -> unit) option -> unit) -> unit)
    -> ('a -> unit) -> unit

These  are a generic  server/client pair and  the rather complex types need some
explanation.  For the server, the actual call to the server looks like:

  respond_server_kind (server_data, connection_data) responder

This initialises a   server.  The user-supplied  data  is  divided up into   two
portions,   server_data  which  persists    while  the  server  is   active  and
connection_data  which is  reinitialised  to the value   given here  at each new
connection.

The "responder" function is  a callback function which  is called by the  server
when a value of the given type is received, for example:

  let responder (count,ud_conn) (msg:K.kind) =
    match msg with
    | K.Low str -> (count, ud_conn, Some (String.uppercase str), true)
    | K.Up str -> (count, ud_conn, Some (String.lowercase str), true)
    | K.Kill -> (sleep 2 @> fun _ -> ()); (count+1, ud_conn, None, count < 2)

This function is passed the user data (server,connection) plus the type received
from the client.  It is expected to return a quadruple:

  (<updated server data>,
   <updated connection data>,
   <response type>,
   <continue flag>)

Most of these are self-explanatory, the response  type is the  data sent back to
the client.  In this case the return type was just a  string so there is no type
conversion done in this case.  The continue flag  should be "true" if the server
is to continue.  If "false" then the server terminates.

The semantics of the client is more complex, the client call looks like:

  respond_client_kind userdata client_cont termination_cont

Here, the user data is a single value.  The client continuation parameter allows
control over the connection:

  let client_cont (list_message,_) k =
    match list_message with
                      (*data*)   (*msg*)         (*handler*)
    | []         -> k ([],true) ((Some K.Kill),  None)                  (* Send message and close connection *)
    | (msg::t)   -> k (t,false) ((Some msg),     (Some client_handler)) (* Send-receive message *)
  (*| <whatever> -> k userdata  (None,           (Some client_handler)) (* Receive message *)*)
  (*| <whatever> -> k userdata  (None,           None)                  (* Close connection immediately *)*)

This is a continuation function  and is passed the  user data and the  following
continuation ("k").  Based on the user  data, the client continuation can decide
to do various things, the following continuation is passed the updated user data
(in this case a list of messages and  a status code) plus  a pair of an optional
query type message for  the server and a continuation  handler for replies  from
the server.  Based on these values the client code will:

   Message   Handler       Action
   -------   -------       ------
   Some msg  Some handler  call Hlnet.sendreceive with the message and handler
   Some msg  None          call Hlnet.send but does not wait for a reply
   None      Some handler  do not send anything but wait with Hlnet.receive
   None      None          terminate the connection

The receive handler has the following semantics:

  let client_handler ((_,last) as userdata) str k =
    k userdata (not last) (* true=>continue, false=>close connection *)

It  is passed  the  user  data, the  message   received from  the  server and  a
continuation  function.  It should   pass the continuation function the  updated
user data plus a continuation flag, "false" means terminate the connection.

Although  quite complex,  this scheme allows  fine  control over the  server and
client by the application, without having to  deal with low-level network issues
although   server/client  coordination   is still   the   responsibility  of the
application code.

The  abstract client model allows the  implementation of the following auxiliary
client functions:

  val respond_client_single_kind : kind -> (string option -> unit) -> unit
  val respond_client_send_kind : kind -> (unit -> unit) -> unit
  val respond_client_receive_kind : (string option -> unit) -> unit

These are simple one-shot client operations:

  respond_client_single_kind:  make  a connection to the  server,  send a single
  message and handle the reply with the handler  function before terminating the
  connection.

  respond_client_send_kind:  connect, send message and close connection.

  respond_client_receive_kind:  connect, handle received message and close
  connection.

Note that the values  received  by the handlers are   always options.  A  "None"
value means an error in receiving the value.

Note also that there are significant differences  between the BSL implementation
of these functions and the OCaml version.  This was because I couldn't get OCaml
to call the OPA continuation functions without segfaults.   The OPA interface is
therefore only  "pseudo-cps" and works by  callback functions instead.  This may
change if I  can ever get OCaml  to call an  OPA continuation function.  See the
multi_clients_bsl.opa file for an example.

Test code and examples
======================

The standard test script
------------------------

In libnet/tests there is a script "testmlidl.sh" which allows testing of various
features  and is intended   to be run as  a  reftester test one  day.   This has
various options   for testing  specific phases  and  includes   two applications
mentioned below.

Note that this test script attempts to compile using the bare ocaml command line
and   does  not  work via    opalang/bld.   You need   a successfully  installed
opalang/mkinstall for this to work.

test.mlidl
..........

This is a bare-bones test which defines one type for each of the types currently
handled by the IDL  code generator.  Note, however,  that  there are actually  a
couple of special cases which look like duplicated  types but are actually there
to  test the peculiarities of  the OCaml type system  (eg.  Cons (int * bool) is
handled slightly strangely by OCaml).  The phase flags are absent from this file
to allow the test script to control the phases from the command-line.

kind.mlidl
..........

This  is a  trivial IDL  file but it   is used in  the multi_clients_bsl.opa and
multi_clients_idl.ml applications which are  IDL implementations of an old Hlnet
test  program.  They use the responder  mechanism and illustrate the differences
between the two implementations.

=========
N. Scaife
26-May-11