Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
Effi documentation updated
  • Loading branch information
joergen7 committed Aug 16, 2018
1 parent a446a08 commit 7aac82c
Show file tree
Hide file tree
Showing 9 changed files with 182 additions and 77 deletions.
6 changes: 3 additions & 3 deletions conf.py
Expand Up @@ -45,16 +45,16 @@

# General information about the project.
project = u'Cuneiform'
copyright = u'2013-2017, Jörgen Brandt'
copyright = u'2013-2018, Jörgen Brandt'

# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = '3.0.0'
version = '3.0.4'
# The full version, including alpha/beta/rc tags.
release = '3.0.0'
release = '3.0.4'

# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
Expand Down
12 changes: 9 additions & 3 deletions effi.rst
@@ -1,11 +1,17 @@
Erlang Foreign Function Interface (effi)
========================================

This section describes the Erlang Foreign Function Interface library (effi).
The Erlang foreign function interface (effi) is a way to run code in arbitrary foreign languages, e.g., Python or Octave from a native (Erlang) environment.

Herein, the foreign code is executed in an environment that binds a defined set of input variables to values defined from the outside. After completion of the foreign code, a set of output variables is expected to be bound to values.

The foreign code snippet along with input variable bindings is given to effi in the form of a key-value map in a format that can easily be decoded/encoded using the `jsone <https://github.com/sile/jsone>`_ JSON serialization library. Accordingly, effi comes with a command-line application that reads and writes JSON files.

.. toctree::
:maxdepth: 2

effi_usage
effi_configuration
effi_supported_languages
effi_integration
effi_synopsis
effi_api
effi_format
12 changes: 12 additions & 0 deletions effi_api.rst
@@ -0,0 +1,12 @@
Erlang API
==========

Effi can be driven from an Erlang API by using the function ``effi:handle_request/2``. Given an Erlang hash map using atoms as keys and binaries as values bound in the variable ``EffiRequest`` you can start effi by entering::

EffiRequest = #{ ... }.
Dir = "./".
effi:handle_request( EffiRequest, Dir ).

Effi starts evaluating the request, expecting input data in and writing output data to the working directory given in the variable ``Dir``.

The ``handle_request/2`` function returns a reply-map summarizing the result of the computation.
2 changes: 0 additions & 2 deletions effi_configuration.rst

This file was deleted.

136 changes: 69 additions & 67 deletions effi_format.rst
@@ -1,117 +1,119 @@
Exchange Formats
================
Effi JSON Exchange Format
=========================

Effi is driven by task instantiations or applications. Running effi on an application results in a reply. Here, we describe the application format and the reply format.

Example
-------

Below is an example for an Effi request (an application)

.. code-block:: json
{ "app_id": "1234",
"lambda": { "lambda_name": "bowtie2-build",
"arg_type_lst": [{ "arg_name": "fa",
"arg_type": "File",
"is_list": false }],
"ret_type_lst": [{ "arg_name": "idx",
"arg_type": "File",
"is_list": false }],
"lang": "Bash",
"script": "bowtie2-build $fa bt2idx\nidx=idx.tar\ntar cf $idx --remove-files bt2idx.*\n" },
"arg_bind_lst": [{ "arg_name": "fa",
"value": "chr22.fa" }] }
{ "app_id": "1234",
"lambda": { "lambda_name": "bowtie2-build",
"arg_type_lst": [{ "arg_name": "fa",
"arg_type": "File",
"is_list": false }],
"ret_type_lst": [{ "arg_name": "idx",
"arg_type": "File",
"is_list": false }],
"lang": "Bash",
"script": "bowtie2-build $fa bt2idx\nidx=idx.tar\ntar cf $idx --remove-files bt2idx.*\n" },
"arg_bind_lst": [{ "arg_name": "fa",
"value": "chr22.fa" }] }
.. code-block:: json
The following is an example for an Effi reply:

{ "app_id": "1234",
"stat": { "tstart": "...",
"duration": "..." },
"result": { "status": "ok",
"ret_bind_lst": [{ "arg_name": "idx",
"value": "idx.tar" }] } }

Application Format
------------------
.. code-block:: json
The application format is what effi consumes. An application :code:`App` has the following form:
{ "app_id": "1234",
"result": { "status": "ok",
"stat": { "run": { "t_start": "1523007609917834743",
"duration": "30391761645" },
"node": "cf_worker@x240" },
"ret_bind_lst": [{ "arg_name": "idx",
"value": "idx.tar" }] } }
.. code-block:: none
The start time `tstart` is given in nanoseconds from 1970-01-01 and also the the wall-clock running time `duration` is given in nanoseconds. So the example ran `bowtie2-build` for about 30.4 seconds. The `node` field identifies the Erlang node name of the worker instance that ran the task.

App ::= { "app_id": S,
"lambda": Lambda,
"arg_bind_lst": [Bind, ...] }
Request Format
--------------

The application's lambda expression :code:`Lambda` has the following form:
The Effi request (application) format is what is consumed. An application `App` has the following form:

.. code-block:: none
Lambda ::= { "lambda_name": S,
"arg_type_lst": [TArg, ...],
"ret_type_lst": [TArg, ...],
"lang": Lang,
"script": S }
App ::= { "app_id": S,
"lambda": Lambda,
"arg_bind_lst": [Bind, ...] }
A lambda expression's :code:`arg_type_lst` pair lists specifications for the input parameters while the :code:`ret_type_lst` pair lists specifications for the output parameters of a lambda. An input or output parameter specification :code:`TArg` has the following form:
The application's lambda expression `Lambda` has the following form:

.. code-block:: none
TArg ::= { "arg_name": S,
"arg_type": Type,
"is_list": B }
Lambda ::= { "lambda_name": S,
"arg_type_lst": [TArg, ...],
"ret_type_lst": [TArg, ...],
"lang": Lang,
"script": S }
The :code:`arg_type` pair provides the base type of the argument. The base type :code:`Type` has the following form:
A lambda expression's `arg_type_lst` pair lists specifications for the input parameters while the `ret_type_lst` pair lists specifications for the output parameters of a lambda. An input or output parameter specification `TArg` has the following form:

.. code-block:: none
Type ::= "Bool"
| "Str"
| "File"
TArg ::= { "arg_name": S,
"arg_type": Type,
"is_list": B }
The :code:`lang` pair provides the programming language in which the script is written. The language :code:`Lang` has the following form:
The `arg_type` pair provides the base type of the argument. The base type `Type` has the following form:

.. code-block:: none
Lang ::= "Bash"
| "Octave"
| "Perl"
| "Python"
| "R"
| "Racket"
Type ::= "Bool"
| "Str"
| "File"
A lambda expression contains a list of argument bindings :code:`Bind` of the following form:
The `lang` pair provides the programming language in which the script is written. The language `Lang` has the following form:

.. code-block:: none
Bind ::= { "arg_name": S, "value": S}
| { "arg_name": S, "value": [S, ...] }
Lang ::= "Bash"
| "Matlab"
| "Octave"
| "Python"
A lambda expression contains a list of argument bindings `Bind` of the following form:

.. code-block:: none
B ::= true
| false
Bind ::= { "arg_name": S, "value": S}
| { "arg_name": S, "value": [S, ...] }
.. code-block:: none
B ::= true
| false
S ::= "..."
S ::= "..."
Reply Format
------------

The reply format is what effi produces.
The Effi reply format is what is produced.

.. code-block:: none
Reply ::= { "app_id": S,
"result": Result }
Reply ::= { "app_id": S,
"result": Result }
Result ::= { "status": "ok",
"stat": { "run": { "t_start": S, "duration": S },
"node": S },
"ret_bind_lst": [Bind, ...] }
| { "status": "error", "stage": "run", "extended_script": S, "output": S }
| { "status": "error", "stage": "stagein", file_lst: [S, ...] }
| { "status": "error", "stage": "stageout", file_lst: [S, ...] }
.. code-block:: none
Result ::= { "status": "ok",
"stat": { "t_start": S, "duration": S },
"ret_bind_lst": [Bind, ...] }
| { "status": "error", "stage": "run", "extended_script": S, "output": S }
| { "status": "error", "stage": "stagein", file_lst: [S, ...] }
| { "status": "error", "stage": "stageout", file_lst: [S, ...] }
34 changes: 34 additions & 0 deletions effi_integration.rst
@@ -0,0 +1,34 @@
Integration and Build
=====================

Adding Effi to a Project
------------------------

Although Effi can be imported also directly from GitHub, we recommend adding a dependency via `hex.pm <https://hex.pm>`_. Here, we show how this can be done using the build tools `rebar3 <https://www.rebar3.org>`_ or mix.


rebar3
^^^^^^

To integrate effi into a rebar3-managed project change the ``deps`` entry in your application's ``rebar.config`` file to include the tuple ``{effi, "0.1.6"}``::

{deps, [{effi, "0.1.6"}]}.

mix
^^^

To integrate effi into a mix-managed project include the following::

{:effi, "~> 0.1.6"}

Compiling
---------

Having rebar3 available on your system, compile as an Erlang project by entering::

rebar3 compile

If you want to drive the project from the command line please compile it by entering::

rebar3 escriptize

16 changes: 16 additions & 0 deletions effi_supported_languages.rst
@@ -0,0 +1,16 @@
Supported Languages
===================

The following programming languages are supported by effi:



- `Bash <https://www.gnu.org/software/bash/>`_
- `Erlang <http://www.erlang.org/>`_
- `Java <https://www.java.com>`_
- `Matlab <https://www.mathworks.com/products/matlab.html>`_
- `Octave <https://www.gnu.org/software/octave/>`_
- `Perl <https://www.perl.org/>`_
- `Python <https://www.python.org/>`_
- `R <https://www.r-project.org/>`_
- `Racket <http://www.racket-lang.org/>`_
39 changes: 39 additions & 0 deletions effi_synopsis.rst
@@ -0,0 +1,39 @@
Command Line Synopsis
=====================

Compiling effi using ``escriptize`` creates an Erlang script file ``effi`` whcih allows starting it via the command line.

To display a help text enter::

./effi --help

This shows the command line synopsis, which looks like the following::

._,,,, ,,_,=_
W `_@__#__ The Erlang Foreign Function Interface (Effi) allows the
@P+# F @F @ execution of functions defined in different programming
_W y @ # qF languages (e.g., Bash, Python, or R) by specifying the
^^^^^ P qF ` function's arguments, body and output values.

Copyright 2015-2018 Jorgen Brandt <joergen.brandt@onlinehome.de>

Usage: effi [-v] [-h] [-d [<dir>]] [-i <input_file>] [-o <output_file>]

-v, --version Show effi version.
-h, --help Show command line options.
-d, --dir Working directory in which to look for input data and
run the request. [default: .]
-i, --input_file Input file holding the effi request (must be
specified).
-o, --output_file Output file into which to write the effi reply (must
be specified).


The input_file and output_file arguments must be specified.

To start Effi from the command line consuming the request file ``effi_request.json`` and let it produce the reply file ``effi_reply.json`` enter::

./effi -i effi_request.json -o effi_reply.json

The format of the request and reply is described below.

2 changes: 0 additions & 2 deletions effi_usage.rst

This file was deleted.

0 comments on commit 7aac82c

Please sign in to comment.