docs(yaml): improve yaml docs on syntax (#1644)

docs/chapters/yaml/compound-executor.rst

@@ -0,0 +1,52 @@
+:class:`CompoundExecutor` YAML Syntax
+=====================================
+
+A compound executor is a set of executors bundled together, as defined in :mod:`jina.executors.compound`. It follows the syntax above with an additional feature: `routing`.
+
+.. highlight:: yaml
+.. code-block:: yaml
+
+    !CompoundExecutor
+    components:
+    - !NumpyIndexer
+      with:
+        num_dim: -1
+        index_key: HNSW32
+        index_filename: vec.idx
+      metas:
+        name: my_vec_indexer
+    - !BasePbIndexer
+      with:
+        index_filename: chunk.gzip
+      metas:
+        name: chunk_meta_indexer
+    with:
+      routes:
+        meta_add:
+          chunk_meta_indexer: add
+        meta_query:
+          chunk_meta_indexer: query
+        query:
+          my_vec_indexer: query
+        add:
+          my_vec_indexer: add
+    metas:
+      name: chunk_compound_indexer
+      workspace: $TEST_WORKDIR
+
+.. confval:: components
+
+    A list of executors specified. Note that ``metas.name`` must be specified if you want to later quote this executor in ``with.routes``.
+
+.. confval:: with
+
+    .. confval:: routes
+
+        .. highlight:: yaml
+        .. code-block:: yaml
+
+            A:
+                B: C
+
+        It defines a function mapping so that a new function :func:`A` is created for this compound executor and points to :func:`B.C`. Note that ``B`` must be a valid name defined in ``components.metas.name``
+

docs/chapters/yaml/driver.rst

@@ -0,0 +1,60 @@
+:class:`Driver` YAML Syntax
+============================
+
+:class:`jina.drivers.Driver` helps the :mod:`jina.executors` to handle the network traffic by interpreting the traffic data (e.g. Protobuf) into the format that the Executor can understand and handle (e.g. Numpy array). Drivers can be specified using keyword `requests` and `on`
+
+.. highlight:: yaml
+.. code-block:: yaml
+
+    !CompoundExecutor
+    components:
+      - !Splitter
+        metas:
+          name: splitter
+      - !Sentencizer
+        with:
+          min_sent_len: 3
+          max_sent_len: 128
+          punct_chars: '.,;!?:'
+        metas:
+          name: sentencizer
+    name: crafter
+    workspace: $WORKSPACE
+    metas:
+      py_modules: splitter.py
+    requests:
+      on:
+        [SearchRequest, IndexRequest]:
+          - !CraftDriver
+            with:
+              executor: splitter
+              method: craft
+          - !SegmentDriver
+            with:
+              executor: sentencizer
+        ControlRequest:
+          - !ControlReqDriver {}
+
+
+.. confval:: requests
+
+    .. confval:: on
+
+        .. confval:: request_type
+
+            Possible values are ``QueryRequest``, ``IndexRequest``, ``TrainRequest``, or a list of them.
+
+            .. confval:: !SomeDriverClass
+
+                The class of the driver, can be any class inherited from jina.drivers.BaseDriver. Note that it must starts with ! to tell the YAML parser that the section below is describing this class.
+
+            .. confval:: with
+
+                A list of arguments in the :func:`__init__` function of this driver. One can use environment variables here to expand the variables.
+
+            .. confval:: metas
+
+                A list of meta arguments defined in :mod:`jina.executors.metas`.
+
+.. note::
+    If no drivers are specified in the yaml file, default drivers defined in `executors.requests.*` files at :mod:`jina.resources` wii be used.

docs/chapters/yaml/executor.rst

@@ -0,0 +1,50 @@
+:class:`Executor` YAML Syntax
+==============================
+
+All executors defined in :mod:`jina.executors` can be loaded from a YAML config via :func:`jina.executors.BaseExecutor.load_config` or via the CLI :command:`jina pod --uses`.
+
+The executor YAML config follows the syntax below.
+
+.. highlight:: yaml
+.. code-block:: yaml
+
+    !BasePbIndexer
+    with:
+      index_filename: doc.gzip
+    metas:  # <- metas defined in :mod`jina.executors.metas`
+      name: doc_indexer  # a customized name
+      workspace: $TEST_WORKDIR
+
+
+.. confval:: !SomeExecutorClass
+
+    The class of the executor, can be any class inherited from :mod:`jina.executors.BaseExecutor`. Note that it must starts with ``!`` to tell the YAML parser that the section below is describing this class.
+
+.. confval:: with
+
+    A list of arguments in the :func:`__init__` function of this executor. One can use environment variables here to expand the variables.
+
+.. confval:: metas
+
+    A list of meta arguments defined in :mod:`jina.executors.metas`.
+
+
+If an executor has no :func:`__init__` or :func:`__init__` requires no arguments, then one do not need to write ``with`` at all.
+
+In the minimum case, if you don't want to specify any ``with`` and ``metas``, you can simply write:
+
+.. highlight:: yaml
+.. code-block:: yaml
+
+    # encoder.yml
+    !AwesomeExecutor
+
+Or even not using this YAML but simply write:
+
+.. highlight:: python
+.. code-block:: python
+
+    import jina.executors.BaseExecutor
+
+    a = BaseExecutor.load_config('AwesomeExecutor')
+

docs/chapters/yaml/flow.rst

@@ -0,0 +1,76 @@
+:class:`Flow` YAML Syntax
+==========================
+
+:class:`jina.Flow` can be loaded from a YAML config file. It follows the following syntax as the example below:
+
+.. highlight:: yaml
+.. code-block:: yaml
+
+    !Flow
+    version: '1.0'
+    with:
+        restful: true
+    pods:
+      - name: pod0  # notice the change here, name is now an attribute
+        method: add  # by default method is always add, available: add, needs, inspect
+        uses: _pass
+        needs: gateway
+      - name: pod1  # notice the change here, name is now an attribute
+        method: add  # by default method is always add, available: add, needs, inspect
+        uses: _pass
+        needs: gateway
+      - method: inspect  # add an inspect node on pod1
+      - method: needs  # let's try something new in Flow YAML v1: needs
+        needs: [pod1, pod0]
+
+A valid Flow specification starts with ``!Flow`` as the first line.
+
+
+.. confval:: version
+
+     The version number string of Flow YAML schema.
+
+     .. warning::
+        Don't forget to quote your version number, it must be a string.
+
+
+.. confval:: with
+
+     A list of arguments in the :func:`jina.Flow.__init__` function. Check :command:`jina flow --help` for details. Extra ``kwargs`` will be passed to **all** pods as the common ``kwargs``.
+
+.. confval:: pods
+
+     The list of :class:`jina.peapods.Pod` contained in the flow. The key is the name of this pod and the value is a map of arguments accepted by :command:`jina pod --help`. Besides those ``kwargs``, there are some optional fields one can set:
+
+.. confval:: pods[*].name
+
+    The user defined name of the Pod. Optional. When not given, it will named as ``pod0``, ``pod1``, etc.
+
+.. confval:: pods[*].method
+
+    The method for appending this Pod into the Flow. Optional, by default it's ``add``:
+
+        - ``add``: same as ``Flow.add(...)``
+        - ``needs``: same as ``Flow.needs(...)``
+        - ``inspect``: same as ``Flow.inspect(...)``
+
+.. confval:: pods[*].needs
+
+    Identifies any Pods that must complete successfully before this Pod will run. It can be a string or array of strings. By default, ``needs`` always contains the previous Pod, unless written in other way. ``needs`` can be used to create intra-Pod parallelization. For example, the Flow below runs ``pod2`` and ``pod3`` in parallel:
+
+    .. highlight:: yaml
+    .. code-block:: yaml
+
+        !Flow
+        version: '1.0'
+        pods:
+            - name: pod1
+            - name: pod2
+            - name: pod3
+              needs: pod1
+            - name: pod4
+              needs: [pod2, pod3]
+
+
+
+

docs/chapters/yaml/index.rst

@@ -0,0 +1,15 @@
+Jina YAML Syntax Reference
+==========================
+
+Jina configurations use YAML syntax, and must have either a ``.yml`` or ``.yaml`` file extension. If you're new to YAML and want to learn more, see `Learn YAML in five minutes. <https://www.codeproject.com/Articles/1214409/Learn-YAML-in-five-minutes>`_
+
+.. note::
+    In many Jina YAML config files, you often see ``!Tag`` such as ``!Flow``, ``!MyEncoder``. Note that ``!Tag`` is valid YAML 1.0 syntax, it represents a language-specific serialization and object, e.g. ``!ruby/symbol``, ``!python/list``. Tags such as ``!Flow`` have been registered inside the Jina YAML parser, and therefore they are readable. However, if you copy-paste this YAML file into an arbitrary online YAML validator, it will most likely report it as **invalid**. The reason is that those tags aren't registered at those online YAML validator.
+
+.. toctree::
+
+   flow
+   executor
+   compound-executor
+   driver
+   substitute

docs/chapters/yaml/substitute.rst

@@ -0,0 +1,68 @@
+Variable Substitution in YAML
+=============================
+
+In the YAML config, one can reference environment variables with ``$ENV``, or using ``{path.variable}`` to reference the variable defined inside the YAML. For example,
+
+.. highlight:: yaml
+.. code-block:: yaml
+
+    components:
+      - with:
+          index_filename: metaproto
+        metas:
+          name: test_meta
+          good_var:
+            - 1
+            - 2
+          bad_var: ${{root.metas.name}}
+      - with:
+          index_filename: npidx
+        metas:
+          name: test_numpy
+          bad_var: ${{root.components[0].metas.good_var[1]}}
+          float_var: ${{root.float.val}}
+          mixed: ${{root.float.val}}-${{root.components[0].metas.good_var[1]}}-${{root.metas.name}}
+          mixed_env: ${{root.float.val}}-${{ ENV.ENV1 }}
+          name_shortcut: ${{this.name}}
+          random_id: ${{ context_var }}
+          config_str: ${{ context_var2 }}
+    metas:
+      name: real-compound
+    rootvar: 123
+    float:
+      val: 0.232
+
+.. confval:: ${{ var }}
+
+    The variable will be evaluated when calling :meth:`Flow.load_config`, :meth:`BaseExecutor.load_config`, :meth:`BaseDriver.load_config`, :meth:`JAML.load`. For example,
+
+        .. highlight:: yaml
+        .. code-block:: yaml
+
+            !Flow
+            with:
+                name: ${{ context_var2 }}
+                timeout_ready: ${{ context_var }}
+
+
+        .. highlight:: python
+        .. code-block:: python
+
+            obj = Flow.load_config('my.yml',
+                                    context={'context_var': 50,
+                                            'context_var2': 'hello-world'})
+
+.. confval:: ${{ root.var }}
+
+    Referring to the top-level variable defined in the root named ``var``.
+
+.. confval:: ${{ this.var }}
+
+    Referring to the same-level variable named ``var``.
+
+.. confval:: ${{ ENV.var }}
+
+    Referring to the OS environment variable.
+
+.. note::
+    One must quote the string when using referenced values, i.e. ``'${{root.metas.name}}'`` but not ``{root.metas.name}``.