Merge pull request #409 from fast-aircraft-design/issue-393_plugin-co…

…mplement

plugin complement

.PyCharmRunConfs/pytest all no-cov.run.xml

@@ -1,6 +1,6 @@
 <!--
   ~ This file is part of FAST-OAD : A framework for rapid Overall Aircraft Design
-  ~ Copyright (C) 2020  ONERA & ISAE-SUPAERO
+  ~ Copyright (C) 2022 ONERA & ISAE-SUPAERO
   ~ FAST is free software: you can redistribute it and/or modify
   ~ it under the terms of the GNU General Public License as published by
   ~ the Free Software Foundation, either version 3 of the License, or
@@ -16,8 +16,8 @@
 <component name="ProjectRunConfigurationManager">
   <configuration default="false" name="pytest all no-cov" type="CompoundRunConfigurationType">
     <toRun name="pytest integration no-cov" type="tests" />
-    <toRun name="pytest units no-cov" type="tests" />
     <toRun name="pytest notebooks no-cov" type="tests" />
+    <toRun name="pytest units no-cov" type="tests" />
     <method v="2" />
   </configuration>
 </component>

.PyCharmRunConfs/pytest notebooks no-cov.run.xml

@@ -1,6 +1,6 @@
 <!--
   ~ This file is part of FAST-OAD : A framework for rapid Overall Aircraft Design
-  ~ Copyright (C) 2021 ONERA & ISAE-SUPAERO
+  ~ Copyright (C) 2022 ONERA & ISAE-SUPAERO
   ~ FAST is free software: you can redistribute it and/or modify
   ~ it under the terms of the GNU General Public License as published by
   ~ the Free Software Foundation, either version 3 of the License, or
@@ -26,7 +26,7 @@
     <option name="_new_keywords" value="&quot;&quot;" />
     <option name="_new_parameters" value="&quot;&quot;" />
     <option name="_new_additionalArguments" value="&quot;--no-cov --nbval-lax -p no:python&quot;" />
-    <option name="_new_target" value="&quot;fastoad&quot;" />
+    <option name="_new_target" value="&quot;fastoad.notebooks&quot;" />
     <option name="_new_targetType" value="&quot;PYTHON&quot;" />
     <method v="2" />
   </configuration>

.codacy.yml

@@ -2,6 +2,7 @@
 exclude_paths:
   - '**/tests/**'
   - 'tests/**'
+  - '**/conftest.py'
   - 'docs/conf.py'
   - 'src/fastoad/api.py'
   - '.github/**'

.codeclimate.yml

@@ -1,3 +1,10 @@
+version: "2"         # required to adjust maintainability checks
+checks:
+  method-complexity:
+    config:
+      threshold: 10
+exclude_patterns:
+  - "**/contest.py"
 plugins:
   pylint:
     enabled: true

.github/workflows/tests.yml

@@ -87,5 +87,5 @@ jobs:
 
       - name: Notebook tests
         if: ${{ github.event_name == 'pull_request' || contains(github.event.head_commit.message, '[test nb]') || github.ref == 'refs/heads/master' }}
-        run: poetry run pytest --no-cov --nbval-lax -p no:python src
+        run: poetry run pytest --no-cov --nbval-lax -p no:python src/fastoad/notebooks
         shell: bash

.github/workflows/watchman_tests.yml

@@ -34,7 +34,7 @@ jobs:
         shell: bash
 
       - name: Unit tests
-        run: pytest --no-cov src
+        run: pytest --no-cov src -c ./conftest.py
         shell: bash
 
       - name: Notebook tests

docs/documentation/custom_modules/add_plugin.rst

@@ -4,41 +4,134 @@
 How to add custom OpenMDAO modules to FAST-OAD as a plugin
 ##########################################################
 
-Once you have created your custom modules for FAST-OAD (see :ref:`add-modules`),
+Once you have :ref:`created your custom modules <add-modules>` for FAST-OAD,
 you may want to share them with other users, which can be done in two ways:
 
     - Providing your code so they can copy it on their computer and have them set their
       :code:`custom_modules` field accordingly in their :ref:`configuration-file`.
     - Packaging your code as a FAST-OAD plugin and have them install it through :code:`pip`
-      or equivalent.
+      or equivalent. This is the subject of current chapter.
 
-To declare your custom modules as a FAST-OAD plugin, you have to package them the usual way
-and declare them as a plugin with :code:`fastoad_model` as plugin group name.
+A FAST-OAD plugin can provide additional FAST-OAD modules, Jupyter notebooks and configuration files:
+
+    - plugin-provided FAST-OAD modules are usable in :ref:`configuration files <configuration-file>`,
+      and can be :ref:`listed<get-module-list>` and :ref:`used<configuration-file-problem-definition>`
+      in the same way as native modules.
+    - Command line can be used by users to retrieve :ref:`notebooks<python-usage>` and
+      :ref:`configuration files<generate-conf-file>`.
+
+Plugin structure
+################
+In your source folder, a typical plugin structure would be like this::
+
+    my_package/
+    ├── __init__.py
+    ├── configurations/
+    │   ├── __init__.py
+    │   ├── configuration_1.yaml
+    │   └── configuration_2.yaml
+    ├── models/
+    │   ├── __init__.py
+    │   ├── my_model.py
+    │   └── some_subpackage/
+    │       ├── __init__.py
+    │       └── some_more_code.py
+    └── notebooks/
+        ├── __init__.py
+        ├── any_data/
+        │   ├── __init__.py
+        │   └── some_data.xml
+        ├── awesome_notebook.ipynb
+        └── good_notebook.ipynb
+
+As shown above, the expected structure is composed of Python **packages**, i.e. every folder should
+contain a :code:`__init__.py` file, **even if it contains only non-Python files** (e.g. data for notebooks).
+
+The root folder can be anywhere in your project structure, since plugin declaration will point to
+its location.
+
+Expected folders in a plugin package are:
+
+    - :code:`models`: contains Python code where FAST-OAD modules are :ref:`registered<add-modules>`.
+    - :code:`configurations`: contains only configuration files in YAML format. No sub-folder is
+      allowed. These configuration files will be usable through :ref:`command line<generate-conf-file>`
+      or API method :meth:`~fastoad.cmd.api.generate_configuration_file`.
+    - :code:`notebooks`: contains any number of Jupyter notebooks and associated data, that will
+      be made available to users through :ref:`command line<python-usage>`.
+
+Any of these folders is optional. Any other folder will be ignored.
+
+
+Plugin packaging
+################
+
+To make your custom modules usable as a FAST-OAD plugin, you have to package them
+and declare your package as a plugin with :code:`fastoad.plugins` as plugin group name.
+
+Here under is a brief tutorial about these operations using `Poetry <https://python-poetry.org>`_.
+
+.. note::
+
+    If you are not familiar with Python packaging, it is recommended to look at this
+    `tutorial <https://packaging.python.org/en/latest/tutorials/packaging-projects/>`_ first.
+    It presents the important steps and notions of the packaging process, and the "classic" way
+    using `setuptools <https://setuptools.pypa.io/en/latest/>`_.
+    And if you want to stick to setuptools, check this
+    `page <https://packaging.python.org/guides/creating-and-discovering-plugins/#using-package-metadata>`_
+    for details about plugin declaration.
 
-This can be done classically with `setuptools <https://packaging.python.org/guides/creating-and-discovering-plugins/#using-package-metadata>`_.
-It can also be done with `Poetry <https://python-poetry.org>`_, which is the way described below:
 
 .. contents::
    :local:
    :depth: 1
 
-
 ******************************
 Plugin declaration
 ******************************
 
-Assuming you project contains the package :code:`start_trek.drives` that contains
+For the example, let's consider that your project contains the package :code:`star_trek.drives`, and
+that your project structure contains::
+
+    src/
+    ├── star_trek/
+    │   ├── __init__.py
+    │   ├── drives/
+    │   │   ├── __init__.py
+    │   │   ├── configurations/
+    │   │   ├── models/
+    │   │   └── notebooks/
+    │   └── ...
+    └── ...
+
+As previously stated, your folder :code:`src/star_trek/drives` does not have to contain all of the
+folders :code:`models`, :code:`configurations` nor :code:`notebooks`.
+
+Assuming you project contains the package :code:`star_trek.drives` that contains
 models you want to share, you can declare your plugin in your :code:`pyproject.toml`
 file with:
 
 .. code-block:: toml
 
     ...
 
-    [tool.poetry.plugins."fastoad_model"]
-    "internal_models" = "start_trek.drives"
+    [tool.poetry]
+    # Tells location of sources
+    packages = [
+        { include = "star_trek", from = "src" },
+    ]
+
+    ...
+
+    # Plugin declaration
+    [tool.poetry.plugins."fastoad.plugins"]
+    "ST_plugin" = "star_trek.drives"
 
     ...
+
+.. note::
+
+    It is discouraged to declare several FAST-OAD plugins for a same project.
+
 Once your :code:`pyproject.toml` is set, you can do :code:`poetry install`. Besides
 installing your project dependencies, it will make your models **locally** available (i.e.
 you could use their identifiers in your FAST-OAD configuration file without setting
@@ -50,7 +143,7 @@ Building
 ******************************
 You can build your package with the command line :code:`poetry build`.
 Let's assume your :code:`pyproject.toml` file is configured so that your project name is
-:code:`STST_drive_models`, as below:
+:code:`ST_drive_models`, as below:
 
 .. code-block:: toml
 
@@ -60,10 +153,29 @@ Let's assume your :code:`pyproject.toml` file is configured so that your project
     name = "ST_drive_models"
     version = "1.0.0"
 
+    # Tells location of sources
+    packages = [
+        { include = "star_trek", from = "src" },
+    ]
+
     ...
 
-It will create a :code:`dist` folder with two files: :code:`ST_drive_models-1.0.0.tar.gz`
-and :code:`ST_drive_models-1.0.0-py3-none-any.whl` (or something like this).
+    # Specify that Poetry is used for building the package
+    [build-system]
+    requires = ["poetry-core>=1.0.0"]
+    build-backend = "poetry.core.masonry.api"
+
+    ...
+
+    # Plugin declaration
+    [tool.poetry.plugins."fastoad.plugins"]
+    "ST_plugin" = "star_trek.drives"
+    ...
+
+The command :code:`poetry build` will create a :code:`dist` folder with two files:
+
+:code:`ST_drive_models-1.0.0.tar.gz` and :code:`ST_drive_models-1.0.0-py3-none-any.whl`
+(or something like this).
 
 You may then have sent any of those two files to another user, who may then install your models
 using :code:`pip` with:
@@ -79,6 +191,11 @@ Once you have built your package, you may publish it on a a package repository.
 :code:`poetry publish` will publish your package on `PyPI <https://pypi.org>`_,
 provided that you have correctly set your account.
 
+.. note::
+
+    Publishing on PyPI requires a valid account, and also that the chosen package name (defined by
+    `name` field in the `pyproject.toml` file) is unused, or already associated to your account.
+
 Poetry can also publish to another destination.
 
 Please see `here <https://python-poetry.org/docs/cli/#publish>`_ for detailed information.

docs/documentation/usage.rst

@@ -300,22 +300,47 @@ inline help using
 
     $ fastoad <sub-command> -h
 
+.. _plugin-info:
+
+How to get information about available plugins
+==============================================
+
+FAST-OAD is built on a plugin architecture where each plugin can provide FAST-OAD modules,
+Jupyter notebooks and sample configuration files (see :ref:`plugin addition<add-plugins>`),
+
+A list of installed plugins can be obtained with:
+
+.. code:: shell-session
+
+    $ fastoad plugin_info
 
 .. _generate-conf-file:
 
 How to generate a configuration file
 ====================================
 
-FAST-OAD can provide a ready-to use configuration file with:
+FAST-OAD can provide a ready-to use configuration.
 
 .. code:: shell-session
 
-    $ fastoad gen_conf my_conf.yml
+    $ fastoad gen_conf my_conf.yml --from_package my_plugin_package --source sample_configuration_1.yml
 
-This generates the file `my_conf.yml`
+This copies the file :code:`sample_configuration_1.yml`provided by installed package
+:code:`my_plugin_package` to file :code:`my_conf.yml`.
 
+See :ref:`how to get plugin information<plugin-info>` for listing the values you can put for
+options :code:`--from_package` and :code:`--source`.
 
-.. _get-module-list:
+If only one package is available, option :code:`--from_package` may be omitted.
+If the selected package provides only one configuration file, option :code:`--source` may be omitted.
+
+Hence with FAST-OAD installed (version below 2.0) without additional plugin, the command can be:
+
+.. code:: shell-session
+
+    $ fastoad gen_conf my_conf.yml
+
+.. _`get-module-list`:
 
 How to get list of registered modules
 =====================================
@@ -412,8 +437,8 @@ can provide a `XDSM <https://mdolab.engin.umich.edu/wiki/xdsm-overview>`_.
 
 XDSM offers a more synthetic view than N2 diagram.
 
-As it uses a web service, you need an internet access for this command, but you do not need
-to be a registered user on the WhatsOpt server.
+As it uses a web service, you need an internet access for this command, but you do not need to be
+a registered user on the WhatsOpt server.
 
 You can create a :code:`xdsm.html` file with:
 
@@ -480,4 +505,3 @@ To do so, type this command **in your terminal**:
     $ fastoad notebooks
 
 Then run the Jupyter server as indicated in the obtained message.
-