Merge pull request #153 from ChihweiLHBird/dev

Update Documents

.github/workflows/python-package.yml

@@ -0,0 +1,41 @@
+# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
+# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
+
+name: Python package
+
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+    branches: [ master ]
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: [3.5, 3.6, 3.7, 3.8]
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Super-Linter
+      uses: github/super-linter@v2.1.0
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install flake8 pytest
+        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+    - name: Lint with flake8
+      run: |
+        # stop the build if there are Python syntax errors or undefined names
+        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
+        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
+        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+    - name: Test with pytest
+      run: |
+        pytest

README.md

@@ -16,13 +16,13 @@
 [The conference paper](https://github.com/cyrus-/papers/raw/master/sciunit-icse14/sciunit-icse14.pdf)
 
 ## Documentation
-[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ChihweiLHBird/sciunit/blob/dev/docs/chapter1.ipynb) <br>
+[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/scidash/sciunit/blob/master/docs/chapter1.ipynb) <br>
 [Chapter 1](https://github.com/scidash/sciunit/blob/master/docs/chapter1.ipynb) /
 [Chapter 2](https://github.com/scidash/sciunit/blob/master/docs/chapter2.ipynb) /
 [Chapter 3](https://github.com/scidash/sciunit/blob/master/docs/chapter3.ipynb) /
-[Chapter 4](https://github.com/ChihweiLHBird/sciunit/blob/dev/docs/chapter4.ipynb) /
-[Chapter 5](https://github.com/ChihweiLHBird/sciunit/blob/dev/docs/chapter5.ipynb) /
-[Chapter 6](https://github.com/ChihweiLHBird/sciunit/blob/dev/docs/chapter6.ipynb) /
+[Chapter 4](https://github.com/scidash/sciunit/blob/dev/docs/chapter4.ipynb) /
+[Chapter 5](https://github.com/scidash/sciunit/blob/dev/docs/chapter5.ipynb) /
+[Chapter 6](https://github.com/scidash/sciunit/blob/dev/docs/chapter6.ipynb) /
 
 
 ## Basic Usage

docs/chapter1.ipynb

@@ -4,11 +4,11 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "![SciUnit Logo](https://raw.githubusercontent.com/scidash/assets/master/logos/sciunit.png)\n",
+    "![SciUnit Logo](https://raw.githubusercontent.com/scidash/assets/master/logos/SciUnit/sci-unit-tag.png)\n",
     "\n",
     "# SciUnit is a framework for validating scientific models by creating experimental-data-driven unit tests. \n",
     "\n",
-    "<a href=\"https://colab.research.google.com/github/ChihweiLHBird/sciunit/blob/dev/docs/chapter1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
+    "<a href=\"https://colab.research.google.com/github/scidash/sciunit/blob/master/docs/chapter1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
     "\n",
     "# Chapter 1. What is SciUnit?\n",
     "Everyone hopes that their model has some correspondence with reality.  Usually, checking whether this is true is done informally.\n",

docs/chapter2.ipynb

@@ -8,7 +8,7 @@
     "\n",
     "# SciUnit is a framework for validating scientific models by creating experimental-data-driven unit tests.  \n",
     "\n",
-    "<a href=\"https://colab.research.google.com/github/ChihweiLHBird/sciunit/blob/dev/docs/chapter2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
+    "<a href=\"https://colab.research.google.com/github/scidash/sciunit/blob/master/docs/chapter2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
     "\n",
     "# Chapter 2. Writing a `model` and `test` in SciUnit from scratch\n",
     "(or [back to Chapter 1](chapter1.ipynb))"

docs/chapter3.ipynb

@@ -8,7 +8,7 @@
     "\n",
     "# SciUnit is a framework for validating scientific models by creating experimental-data-driven unit tests.  \n",
     "\n",
-    "<a href=\"https://colab.research.google.com/github/ChihweiLHBird/sciunit/blob/dev/docs/chapter3.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
+    "<a href=\"https://colab.research.google.com/github/scidash/sciunit/blob/master/docs/chapter3.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
     "\n",
     "# Chapter 3. Testing with help from the SciUnit standard library\n",
     "(or [back to Chapter 2](chapter2.ipynb))"

docs/chapter4.ipynb

@@ -8,7 +8,7 @@
     "\n",
     "# SciUnit is a framework for validating scientific models by creating experimental-data-driven unit tests.  \n",
     "\n",
-    "<a href=\"https://colab.research.google.com/github/ChihweiLHBird/sciunit/blob/dev/docs/chapter4.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
+    "<a href=\"https://colab.research.google.com/github/scidash/sciunit/blob/master/docs/chapter4.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
     "\n",
     "# Chapter 4. Example of RunnableModel and Backend\n",
     "(or [back to Chapter 3](chapter3.ipynb))"

docs/chapter5.ipynb

@@ -8,7 +8,7 @@
     "\n",
     "# SciUnit is a framework for validating scientific models by creating experimental-data-driven unit tests.  \n",
     "\n",
-    "<a href=\"https://colab.research.google.com/github/ChihweiLHBird/sciunit/blob/dev/docs/chapter5.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
+    "<a href=\"https://colab.research.google.com/github/scidash/sciunit/blob/master/docs/chapter5.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
     "\n",
     "# Chapter 5. The Real Example of Using SciUnit To Test Cosmology Models\n",
     "(or [back to Chapter 4](chapter4.ipynb))\n",

docs/chapter6.ipynb

@@ -8,7 +8,7 @@
     "\n",
     "# SciUnit is a framework for validating scientific models by creating experimental-data-driven unit tests.  \n",
     "\n",
-    "<a href=\"https://colab.research.google.com/github/ChihweiLHBird/sciunit/blob/dev/docs/chapter6.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
+    "<a href=\"https://colab.research.google.com/github/scidash/sciunit/blob/master/docs/chapter6.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
     "\n",
     "# Chapter 6. Workshop Tutorial, Real Example\n",
     "(or [back to Chapter 5](chapter5.ipynb))"

docs/source/commandline.rst

@@ -0,0 +1,213 @@
+Config File And Using SciUnit In A Shell
+=============================================
+
+Create Config File And Execute Tests In A Shell
+----------------------------------------------------------------
+
+We can build a scientific computing project with a SciUnit config file. 
+Then, we will be able to run sciunit In A Shell
+
+Here is an example of well written SciUnit config file.
+This file was generated by executing ``sciunit create`` in the shell.
+A SciUnit config file is always named ``.sciunit``.
+
+.. code-block:: python
+
+    [misc]
+    config-version = 1.0
+    nb-name = scidash
+
+    [root]
+    path = .
+
+    [models]
+    module = models
+
+    [tests]
+    module = tests
+
+    [suites]
+    module = suites
+
+``config-version`` is the version of the config file.
+
+``nb-name`` is the name of the IPython Notebook file that can be create with ``sciunit make-nb``.
+
+``root`` is the root of the project. The ``path`` is the path to the project from the directory that contains this config file.
+
+``module`` in the ``models`` section is the path from the root of the project to the file that contains ``models``, which is a list of ``Model`` instances.
+
+``module`` in the ``tests`` section is the path the root of the project to the file that contains ``tests``, which is a list of ``Test`` instances.
+
+``module`` in the ``suites`` section is the path the root of the project to the file that contains ``suites``, which is a list of ``TestSuite`` instances.
+
+Let's use the config file above and create corresponding files that contain definitions models, tests, and suites.
+
+In the root directory of the project, let's create three files.
+
+**tests.py**
+
+.. code-block:: python
+
+    import sciunit
+    from sciunit.scores import BooleanScore
+    from sciunit.capabilities import ProducesNumber
+
+
+
+    class EqualsTest(sciunit.Test):
+        """Tests if the model predicts 
+        the same number as the observation."""   
+        
+        required_capabilities = (ProducesNumber,)
+        score_type = BooleanScore
+        
+        def generate_prediction(self, model):
+            return model.produce_number() # The model has this method if it inherits from the 'ProducesNumber' capability.
+        
+        def compute_score(self, observation, prediction):
+            score = self.score_type(observation['value'] == prediction)
+            score.description = 'Passing score if the prediction equals the observation'
+            return score
+
+    tests = []
+
+**suites.py**
+
+.. code-block:: python
+
+    import sciunit
+    from tests import EqualsTest
+
+
+    equals_1_test = EqualsTest({'value':1}, name='=1')
+    equals_2_test = EqualsTest({'value':2}, name='=2')
+    equals_37_test = EqualsTest({'value':37}, name='=37')
+
+    equals_suite = sciunit.TestSuite([equals_1_test, equals_2_test, equals_37_test], name="Equals test suite")
+
+    suites = [equals_suite]
+
+**models.py**
+
+.. code-block:: python
+
+    import sciunit
+    from sciunit.capabilities import ProducesNumber
+
+
+    class ConstModel(sciunit.Model, 
+                    ProducesNumber):
+        """A model that always produces a constant number as output."""
+        
+        def __init__(self, constant, name=None):
+            self.constant = constant 
+            super(ConstModel, self).__init__(name=name, constant=constant)
+
+        def produce_number(self):
+            return self.constant
+
+    const_model_1 = ConstModel(1, name='Constant Model 1')
+    const_model_2 = ConstModel(2, name='Constant Model 2')
+    const_model_37 = ConstModel(37, name="Constant Model 37")
+
+    models = [const_model_1, const_model_2, const_model_37]
+
+
+We have ``tests`` at the end of ``tests.py``, ``models`` at the end of ``models.py``,
+and ``suites`` at the end of ``suites.py``. Since we are using test suites instead of tests,
+``tests`` is an empty list in this example. They will be taken by sciunit when command 
+``sciunit run`` is being executing
+
+Execute ``sciunit run`` in the root directory, and then sciunit will run each test in the suites
+against each model and give us the result.
+
+.. code-block:: bash
+
+    $ sciunit run
+
+
+    Executing test =1 on model Constant Model 1... Score is Pass
+    Executing test =2 on model Constant Model 1... Score is Fail
+    Executing test =37 on model Constant Model 1... Score is Fail
+    Executing test =1 on model Constant Model 2... Score is Fail
+    Executing test =2 on model Constant Model 2... Score is Pass
+    Executing test =37 on model Constant Model 2... Score is Fail
+    Executing test =1 on model Constant Model 37... Score is Fail
+    Executing test =2 on model Constant Model 37... Score is Fail
+    Executing test =37 on model Constant Model 37... Score is Pass
+
+    Suite Equals test suite:
+                        =1    =2   =37
+    Constant Model 1   Pass  Fail  Fail
+    Constant Model 2   Fail  Pass  Fail
+    Constant Model 37  Fail  Fail  Pass
+
+
+Create and Run IPython Notebook File
+-------------------------------------
+
+Next, let's move to creating and executing IPython Notebook file with 
+``sciunit make-nb`` and ``sciunit run-nb`` commands. 
+
+Let's add a file, ``__init__.py``, to our project directory and import everything 
+including suites, tests, and models in the file. This is necessary because the made 
+notebook file will try to import everything in ``__init__.py`` and run each suite 
+(a collection of tests instances) against each model.
+
+**__init__.py**
+
+.. code-block:: python
+
+    from . import models
+    from . import tests
+    from . import suites
+
+Now, let's execute ``sciunit make-nb`` SciUnit will automatically generate 
+a notebook file. 
+
+.. code-block:: bash
+
+    $ sciunit make-nb
+    Created Jupyter notebook at:
+    /the_path_to_the_project.../test_sciunit.ipynb
+
+The notebook file will contains two blocks of code:
+
+.. code-block::
+
+    %matplotlib inline
+    from IPython.display import display
+    from importlib.machinery import SourceFileLoader
+    test_sciunit = SourceFileLoader('scidash', '/the_path_to_the_project.../__init__.py').load_module()
+
+
+.. code-block::
+
+    for suite in test_sciunit.suites.suites:
+        score_matrix = suite.judge(test_sciunit.models.models, stop_on_error=True)
+        display(score_matrix)
+
+
+
+Note: 
+
+1. the name of generated notebook file will be the value of ``nb-name`` attribute
+   in the config file, ``.sciunit``
+
+2. The path to the project's root can be different on different machine.
+   So, The notebook file generated usually only be valid on the machine where
+   it is generated. If you want to execute it on different machine, try to re-generate
+   it or change the path.
+
+Let's execute ``sciunit run-nb`` command.
+
+.. code-block:: bash
+
+    $ sciunit run-nb
+    Entering run function
+    /the_path_to_the_project_config_file..././test_sciunit.ipynb
+    /the_path_to_the_project_config_file.../.
+
+The result of running the notebook will be in the notebook file. 
+You can open it by many tools like VS Code and Jupyter Lab

docs/source/index.rst

@@ -25,7 +25,7 @@ Tutorials With Jupyter NoteBook
 -------------------------------
 
 .. image:: https://colab.research.google.com/assets/colab-badge.svg
-   :target: https://colab.research.google.com/github/ChihweiLHBird/sciunit/blob/dev/docs/chapter1.ipynb
+   :target: https://colab.research.google.com/github/scidash/sciunit/blob/master/docs/chapter1.ipynb
 
 * `Tutorial Chapter 1`_
 * `Tutorial Chapter 2`_
@@ -37,9 +37,9 @@ Tutorials With Jupyter NoteBook
 .. _Tutorial Chapter 1: https://github.com/scidash/sciunit/blob/master/docs/chapter1.ipynb
 .. _Tutorial Chapter 2: https://github.com/scidash/sciunit/blob/master/docs/chapter2.ipynb
 .. _Tutorial Chapter 3: https://github.com/scidash/sciunit/blob/master/docs/chapter3.ipynb
-.. _Tutorial Chapter 4: https://github.com/ChihweiLHBird/sciunit/blob/dev/docs/chapter4.ipynb
-.. _Tutorial Chapter 5: https://github.com/ChihweiLHBird/sciunit/blob/dev/docs/chapter5.ipynb
-.. _Tutorial Chapter 6: https://github.com/ChihweiLHBird/sciunit/blob/dev/docs/chapter6.ipynb
+.. _Tutorial Chapter 4: https://github.com/scidash/sciunit/blob/dev/docs/chapter4.ipynb
+.. _Tutorial Chapter 5: https://github.com/scidash/sciunit/blob/dev/docs/chapter5.ipynb
+.. _Tutorial Chapter 6: https://github.com/scidash/sciunit/blob/dev/docs/chapter6.ipynb
 
 Basic Usage
 -----------
@@ -101,3 +101,4 @@ Table of Contents
    setup
    quick_tutorial
    basics
+   commandline

docs/source/quick_tutorial.rst

@@ -361,3 +361,15 @@ print the score, and we can see the result.
 
 >>> print(score)
 Pass
+
+Real Example
+-------------
+
+For real example of using SciUnit, you can read Chapter 5 and 6 of the Jupyter notebook tutorial.
+
+`Tutorial Chapter 5`_
+
+`Tutorial Chapter 6`_
+
+.. _Tutorial Chapter 5: https://github.com/scidash/sciunit/blob/master/docs/chapter5.ipynb
+.. _Tutorial Chapter 6: https://github.com/scidash/sciunit/blob/master/docs/chapter6.ipynb