Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
Effi documentation updated
- Loading branch information
Showing
9 changed files
with
182 additions
and
77 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,11 +1,17 @@ | ||
Erlang Foreign Function Interface (effi) | ||
======================================== | ||
|
||
This section describes the Erlang Foreign Function Interface library (effi). | ||
The Erlang foreign function interface (effi) is a way to run code in arbitrary foreign languages, e.g., Python or Octave from a native (Erlang) environment. | ||
|
||
Herein, the foreign code is executed in an environment that binds a defined set of input variables to values defined from the outside. After completion of the foreign code, a set of output variables is expected to be bound to values. | ||
|
||
The foreign code snippet along with input variable bindings is given to effi in the form of a key-value map in a format that can easily be decoded/encoded using the `jsone <https://github.com/sile/jsone>`_ JSON serialization library. Accordingly, effi comes with a command-line application that reads and writes JSON files. | ||
|
||
.. toctree:: | ||
:maxdepth: 2 | ||
|
||
effi_usage | ||
effi_configuration | ||
effi_supported_languages | ||
effi_integration | ||
effi_synopsis | ||
effi_api | ||
effi_format |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
Erlang API | ||
========== | ||
|
||
Effi can be driven from an Erlang API by using the function ``effi:handle_request/2``. Given an Erlang hash map using atoms as keys and binaries as values bound in the variable ``EffiRequest`` you can start effi by entering:: | ||
|
||
EffiRequest = #{ ... }. | ||
Dir = "./". | ||
effi:handle_request( EffiRequest, Dir ). | ||
|
||
Effi starts evaluating the request, expecting input data in and writing output data to the working directory given in the variable ``Dir``. | ||
|
||
The ``handle_request/2`` function returns a reply-map summarizing the result of the computation. |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,117 +1,119 @@ | ||
Exchange Formats | ||
================ | ||
Effi JSON Exchange Format | ||
========================= | ||
|
||
Effi is driven by task instantiations or applications. Running effi on an application results in a reply. Here, we describe the application format and the reply format. | ||
|
||
Example | ||
------- | ||
|
||
Below is an example for an Effi request (an application) | ||
|
||
.. code-block:: json | ||
{ "app_id": "1234", | ||
"lambda": { "lambda_name": "bowtie2-build", | ||
"arg_type_lst": [{ "arg_name": "fa", | ||
"arg_type": "File", | ||
"is_list": false }], | ||
"ret_type_lst": [{ "arg_name": "idx", | ||
"arg_type": "File", | ||
"is_list": false }], | ||
"lang": "Bash", | ||
"script": "bowtie2-build $fa bt2idx\nidx=idx.tar\ntar cf $idx --remove-files bt2idx.*\n" }, | ||
"arg_bind_lst": [{ "arg_name": "fa", | ||
"value": "chr22.fa" }] } | ||
{ "app_id": "1234", | ||
"lambda": { "lambda_name": "bowtie2-build", | ||
"arg_type_lst": [{ "arg_name": "fa", | ||
"arg_type": "File", | ||
"is_list": false }], | ||
"ret_type_lst": [{ "arg_name": "idx", | ||
"arg_type": "File", | ||
"is_list": false }], | ||
"lang": "Bash", | ||
"script": "bowtie2-build $fa bt2idx\nidx=idx.tar\ntar cf $idx --remove-files bt2idx.*\n" }, | ||
"arg_bind_lst": [{ "arg_name": "fa", | ||
"value": "chr22.fa" }] } | ||
.. code-block:: json | ||
The following is an example for an Effi reply: | ||
|
||
{ "app_id": "1234", | ||
"stat": { "tstart": "...", | ||
"duration": "..." }, | ||
"result": { "status": "ok", | ||
"ret_bind_lst": [{ "arg_name": "idx", | ||
"value": "idx.tar" }] } } | ||
|
||
Application Format | ||
------------------ | ||
.. code-block:: json | ||
The application format is what effi consumes. An application :code:`App` has the following form: | ||
{ "app_id": "1234", | ||
"result": { "status": "ok", | ||
"stat": { "run": { "t_start": "1523007609917834743", | ||
"duration": "30391761645" }, | ||
"node": "cf_worker@x240" }, | ||
"ret_bind_lst": [{ "arg_name": "idx", | ||
"value": "idx.tar" }] } } | ||
.. code-block:: none | ||
The start time `tstart` is given in nanoseconds from 1970-01-01 and also the the wall-clock running time `duration` is given in nanoseconds. So the example ran `bowtie2-build` for about 30.4 seconds. The `node` field identifies the Erlang node name of the worker instance that ran the task. | ||
|
||
App ::= { "app_id": S, | ||
"lambda": Lambda, | ||
"arg_bind_lst": [Bind, ...] } | ||
Request Format | ||
-------------- | ||
|
||
The application's lambda expression :code:`Lambda` has the following form: | ||
The Effi request (application) format is what is consumed. An application `App` has the following form: | ||
|
||
.. code-block:: none | ||
Lambda ::= { "lambda_name": S, | ||
"arg_type_lst": [TArg, ...], | ||
"ret_type_lst": [TArg, ...], | ||
"lang": Lang, | ||
"script": S } | ||
App ::= { "app_id": S, | ||
"lambda": Lambda, | ||
"arg_bind_lst": [Bind, ...] } | ||
A lambda expression's :code:`arg_type_lst` pair lists specifications for the input parameters while the :code:`ret_type_lst` pair lists specifications for the output parameters of a lambda. An input or output parameter specification :code:`TArg` has the following form: | ||
The application's lambda expression `Lambda` has the following form: | ||
|
||
.. code-block:: none | ||
TArg ::= { "arg_name": S, | ||
"arg_type": Type, | ||
"is_list": B } | ||
Lambda ::= { "lambda_name": S, | ||
"arg_type_lst": [TArg, ...], | ||
"ret_type_lst": [TArg, ...], | ||
"lang": Lang, | ||
"script": S } | ||
The :code:`arg_type` pair provides the base type of the argument. The base type :code:`Type` has the following form: | ||
A lambda expression's `arg_type_lst` pair lists specifications for the input parameters while the `ret_type_lst` pair lists specifications for the output parameters of a lambda. An input or output parameter specification `TArg` has the following form: | ||
|
||
.. code-block:: none | ||
Type ::= "Bool" | ||
| "Str" | ||
| "File" | ||
TArg ::= { "arg_name": S, | ||
"arg_type": Type, | ||
"is_list": B } | ||
The :code:`lang` pair provides the programming language in which the script is written. The language :code:`Lang` has the following form: | ||
The `arg_type` pair provides the base type of the argument. The base type `Type` has the following form: | ||
|
||
.. code-block:: none | ||
Lang ::= "Bash" | ||
| "Octave" | ||
| "Perl" | ||
| "Python" | ||
| "R" | ||
| "Racket" | ||
Type ::= "Bool" | ||
| "Str" | ||
| "File" | ||
A lambda expression contains a list of argument bindings :code:`Bind` of the following form: | ||
The `lang` pair provides the programming language in which the script is written. The language `Lang` has the following form: | ||
|
||
.. code-block:: none | ||
Bind ::= { "arg_name": S, "value": S} | ||
| { "arg_name": S, "value": [S, ...] } | ||
Lang ::= "Bash" | ||
| "Matlab" | ||
| "Octave" | ||
| "Python" | ||
A lambda expression contains a list of argument bindings `Bind` of the following form: | ||
|
||
.. code-block:: none | ||
B ::= true | ||
| false | ||
Bind ::= { "arg_name": S, "value": S} | ||
| { "arg_name": S, "value": [S, ...] } | ||
.. code-block:: none | ||
B ::= true | ||
| false | ||
S ::= "..." | ||
S ::= "..." | ||
Reply Format | ||
------------ | ||
|
||
The reply format is what effi produces. | ||
The Effi reply format is what is produced. | ||
|
||
.. code-block:: none | ||
Reply ::= { "app_id": S, | ||
"result": Result } | ||
Reply ::= { "app_id": S, | ||
"result": Result } | ||
Result ::= { "status": "ok", | ||
"stat": { "run": { "t_start": S, "duration": S }, | ||
"node": S }, | ||
"ret_bind_lst": [Bind, ...] } | ||
| { "status": "error", "stage": "run", "extended_script": S, "output": S } | ||
| { "status": "error", "stage": "stagein", file_lst: [S, ...] } | ||
| { "status": "error", "stage": "stageout", file_lst: [S, ...] } | ||
.. code-block:: none | ||
Result ::= { "status": "ok", | ||
"stat": { "t_start": S, "duration": S }, | ||
"ret_bind_lst": [Bind, ...] } | ||
| { "status": "error", "stage": "run", "extended_script": S, "output": S } | ||
| { "status": "error", "stage": "stagein", file_lst: [S, ...] } | ||
| { "status": "error", "stage": "stageout", file_lst: [S, ...] } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,34 @@ | ||
Integration and Build | ||
===================== | ||
|
||
Adding Effi to a Project | ||
------------------------ | ||
|
||
Although Effi can be imported also directly from GitHub, we recommend adding a dependency via `hex.pm <https://hex.pm>`_. Here, we show how this can be done using the build tools `rebar3 <https://www.rebar3.org>`_ or mix. | ||
|
||
|
||
rebar3 | ||
^^^^^^ | ||
|
||
To integrate effi into a rebar3-managed project change the ``deps`` entry in your application's ``rebar.config`` file to include the tuple ``{effi, "0.1.6"}``:: | ||
|
||
{deps, [{effi, "0.1.6"}]}. | ||
|
||
mix | ||
^^^ | ||
|
||
To integrate effi into a mix-managed project include the following:: | ||
|
||
{:effi, "~> 0.1.6"} | ||
|
||
Compiling | ||
--------- | ||
|
||
Having rebar3 available on your system, compile as an Erlang project by entering:: | ||
|
||
rebar3 compile | ||
|
||
If you want to drive the project from the command line please compile it by entering:: | ||
|
||
rebar3 escriptize | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
Supported Languages | ||
=================== | ||
|
||
The following programming languages are supported by effi: | ||
|
||
|
||
|
||
- `Bash <https://www.gnu.org/software/bash/>`_ | ||
- `Erlang <http://www.erlang.org/>`_ | ||
- `Java <https://www.java.com>`_ | ||
- `Matlab <https://www.mathworks.com/products/matlab.html>`_ | ||
- `Octave <https://www.gnu.org/software/octave/>`_ | ||
- `Perl <https://www.perl.org/>`_ | ||
- `Python <https://www.python.org/>`_ | ||
- `R <https://www.r-project.org/>`_ | ||
- `Racket <http://www.racket-lang.org/>`_ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
Command Line Synopsis | ||
===================== | ||
|
||
Compiling effi using ``escriptize`` creates an Erlang script file ``effi`` whcih allows starting it via the command line. | ||
|
||
To display a help text enter:: | ||
|
||
./effi --help | ||
|
||
This shows the command line synopsis, which looks like the following:: | ||
|
||
._,,,, ,,_,=_ | ||
W `_@__#__ The Erlang Foreign Function Interface (Effi) allows the | ||
@P+# F @F @ execution of functions defined in different programming | ||
_W y @ # qF languages (e.g., Bash, Python, or R) by specifying the | ||
^^^^^ P qF ` function's arguments, body and output values. | ||
|
||
Copyright 2015-2018 Jorgen Brandt <joergen.brandt@onlinehome.de> | ||
|
||
Usage: effi [-v] [-h] [-d [<dir>]] [-i <input_file>] [-o <output_file>] | ||
|
||
-v, --version Show effi version. | ||
-h, --help Show command line options. | ||
-d, --dir Working directory in which to look for input data and | ||
run the request. [default: .] | ||
-i, --input_file Input file holding the effi request (must be | ||
specified). | ||
-o, --output_file Output file into which to write the effi reply (must | ||
be specified). | ||
|
||
|
||
The input_file and output_file arguments must be specified. | ||
|
||
To start Effi from the command line consuming the request file ``effi_request.json`` and let it produce the reply file ``effi_reply.json`` enter:: | ||
|
||
./effi -i effi_request.json -o effi_reply.json | ||
|
||
The format of the request and reply is described below. | ||
|
This file was deleted.
Oops, something went wrong.