diff --git a/reference/idl_language.rst b/reference/idl_language.rst new file mode 100644 index 0000000..62a32ba --- /dev/null +++ b/reference/idl_language.rst @@ -0,0 +1,45 @@ +.. _idl_language: + +IDL Language +============ + +An IDL block describes the interface of an external module +(which could be written in Python, for instance). +It is embedded in :ref:`nstack.yaml` + +The IDL block consists of two sections: +the first section declares types, +the second section declares functions. + +Types are declared using the ``type`` keyword: :: + + type PlantInfo = { petalLength : Double + , petalWidth : Double + , sepalLength : Double + , sepalWidth : Double + } + type PlantSpecies = Text + +The left-hand side of a type declaration is the new type name; +the right-hand side must be an :ref:`existing type`. +Type declarations don't need to be delimited in any way. + +Function declarations have form ``name : Type`` and +must be delimited by commas, for instance :: + + gotham : MovieRecordImage -> MovieRecordImage, + kelvin : MovieRecordImage -> MovieRecordImage, + lomo : MovieRecordImage -> MovieRecordImage + +Thus, the full IDL block might look like this: :: + + type PlantInfo = { petalLength : Double + , petalWidth : Double + , sepalLength : Double + , sepalWidth : Double + } + type PlantSpecies = Text + + gotham : MovieRecordImage -> MovieRecordImage, + kelvin : MovieRecordImage -> MovieRecordImage, + lomo : MovieRecordImage -> MovieRecordImage diff --git a/reference/index.rst b/reference/index.rst index 877ae6e..18481b6 100644 --- a/reference/index.rst +++ b/reference/index.rst @@ -8,6 +8,8 @@ Reference nstack_toolkit module_structure + supported_types workflow_language + idl_language supported_integrations diff --git a/reference/module_structure.rst b/reference/module_structure.rst index 74e3def..f9ea715 100644 --- a/reference/module_structure.rst +++ b/reference/module_structure.rst @@ -76,7 +76,7 @@ The base-image your module builds from. This is typically generated by default w *Required* -The API which defines which functions you want to expose, and their input and output schemas. Please see :ref:`Workflow Language<_workflow_langauge>` for more details. +The API which defines which functions you want to expose, and their input and output schemas. Please see :ref:`idl_language` for more details. ``files`` ^^^^^^^^^ diff --git a/reference/supported_types.rst b/reference/supported_types.rst new file mode 100644 index 0000000..c94497c --- /dev/null +++ b/reference/supported_types.rst @@ -0,0 +1,26 @@ +.. _supported_types: + +Supported Types +=============== + +Primitive types +--------------- + +NStack supports the following primitive types: + +* ``Integer`` +* ``Double`` +* ``Boolean`` +* ``Text`` + +.. ByteArray + +More complex types can be built out of primitive ones: + +* Optional types: ``type1 optional`` +* Tuples: ``(type1, type2, ...)``. A tuple must have at least two fields. +* Structs: ``{ name1: type1, name2: type2, ... }`` +* Arrays: ``[type1]`` +* `Sums `_: ``Name1 type1a ... | Name2 type2a ... | ...`` + +A user can define their own type in the :ref:`idl_language`. diff --git a/reference/workflow_language.rst b/reference/workflow_language.rst index e69de29..6e16e40 100644 --- a/reference/workflow_language.rst +++ b/reference/workflow_language.rst @@ -0,0 +1,107 @@ +.. _workflow_language: + +Workflow Language +================= + +A module consists of a module header, import statements, and definitions, +for instance: :: + + module ModuleB:0.1.0 { + import ModuleA:0.1.0 as A; + def x = A.y | A.z; + } + +An import statement includes the module to be imported (``MyModule:0.1.0``) +and its alias (``A``). +The alias is used to qualify functions imported from the module, +e.g. ``A.y``. + +Definitions bind function names (``x``) to expressions (``A.y | A.z``). +Any function ``x`` defined in a module ``ModuleB`` can be used in +any other module: :: + + import ModuleB:0.1.0 as B; + def z = filter B.x; + +If a name is not prefixed by a module alias, it refers to a function defined in +the current module. + +Expressions combine already defined functions through the following operations: + +* Pipe: ``A.y | A.z`` + + Every value produced by ``A.y`` is passed to ``A.z``. + + The output type of ``A.y`` must match the input type of ``A.z``. + +* Concat: ``concat A.y`` or ``A.y*`` + + ``A.y`` must be a function that produces lists of values, + in which case ``concat A.y`` is a function that "unpacks" the lists + and yields the same values one by one. + +* Filter: ``filter A.y`` or ``A.y?`` + + ``A.y`` must be a function that produces "optional" (potentially missing) values, + in which case ``filter A.y`` is a function that filters out missing values. + +* Type application. Some functions (notably, most sources and sinks) can be specialized + to multiple input or output types. + This is done with type application: ``Sources.http`` specializes + ``Sources.http`` to the type ``Text``. + +* Parameter application: ``A.y { one = "...", two = "..." }``. + + Parameters are analogous to UNIX environment variables in the following ways: + + 1. Parameters are inherited. E.g. in :: + + y = x; + z = y { foo = "bar" }; + + both functions ``x`` and ``y`` will have access to ``foo`` when ``z`` is + called. + + 2. Parameters can be overridden. E.g. in :: + + y = x { foo = "baz" }; + z = y { foo = "bar" }; + + ``y`` overrides the value of ``foo`` that is passed to ``x``. + Therefore, ``x`` will see the value of ``foo`` as ``baz``, not ``bar``. + + Parameters are used to configure sources and sinks — + for instance, to specify how to connect to a PostgreSQL database. + + Parameters can also be used to configure user-defined modules. + Inside a Python nstack method, the value of parameter ``foo`` can be accessed as + ``self.args["foo"]``. + +The workflow language supports line and block comments. +Line comments start with ``//`` and extend until the end of line. +Block comments are enclosed in ``/*`` and ``*/`` and cannot be nested. + +EBNF grammar +------------ + +The syntax is defined in EBNF (ISO/IEC 14977) in terms of tokens. + +.. highlight:: ebnf + +:: + + module = 'module', module name, '{', {import}, {definition}, '}'; + import = 'import', module name, 'as', module alias, ';'; + definition = 'def', name, '=', expression, ';'; + expression = expression1, {'|', expression1}; + expression1 = application, '*' + | application, '?' + | 'concat', application + | 'filter', application + ; + application = term [arguments]; + arguments = '{', argument binding, {',', argument binding}, '}'; + argument binding = name, '=', literal; + term = '(', expression, ')' + | qualified name ['<', type, '>'] + ;