Add better output, docs and first set of check tools (#58)

* Make minor adjustments to progress bar.

* Sort imports.

* Sort requirements.txt

* Add ypy-websocket to requirements.

* Commit more files.

* Add checks.py

* Move to tox.

* Move to pyproject.toml.

* Document new tools.

* Add rtd config file.

* Add new GA file.

* Delete style check.

* Make cli test less flaky.

* Add example assignements and fix docs.

* Add rtds theme to conf file.

* Add doc extra require.

* Add correct config options.

* Correct doc -> docs

* Remove Windows from CI.

* Remove skip.

.github/workflows/tests.yml

@@ -0,0 +1,31 @@
+name: CI
+
+on: [push, pull_request]
+
+jobs:
+  build:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      max-parallel: 4
+      matrix:
+        os: [ubuntu-latest, macOS-latest]
+        python-version: [3.8, 3.9, "3.11", "3.12"]
+
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: update pip
+        run: |
+          python -m pip install --upgrade pip
+
+      - name: install tox
+        run: |
+          python -m pip install tox tox-gh-actions
+
+      - name: run tox
+        run: |
+          python -m tox

.gitignore

@@ -5,3 +5,6 @@
 *ipynb_checkpoints*
 .coverage
 dist/*
+env/*
+*.ipynb-feedback.testmd
+*DS_Store

.readthedocs.yml

@@ -0,0 +1,25 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+sphinx:
+  configuration: docs/conf.py
+
+formats:
+  - pdf
+  - epub
+
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.11"
+
+python:
+  install:
+    - method: pip
+      path: .
+      extra_requirements:
+        - docs

LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2023 Vincent Knight
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

assignment.ipynb

@@ -1 +1 @@
-{"cells": [{"cell_type": "markdown", "metadata": {}, "source": "# Class assignment\n\nWe will use this assignment to solidify our understanding of using Python to\ncarry out some numerical operations and also write functions.\n\nUse this notebook to write your answers in the cells as instructed, do your\nbest not to delete any of the cells that are already there.\n\n## Question one.\n\nUse python to obtain the remainder when dividing 21 by 5:\n\n\\\\[21 \\mod 5\\\\]"}, {"cell_type": "code", "execution_count": 1, "metadata": {"tags": ["answer:q1"]}, "outputs": [], "source": "### BEGIN SOLUTION\n\n\n### END SOLUTION"}, {"cell_type": "markdown", "metadata": {}, "source": "## Question two.\n\nWrite a python function `get_remainder(m, n)` that returns the remainder\nthe remainder when dividing \\\\(m\\\\) by \\\\(n\\\\).\n\n\\\\[m \\mod n\\\\]"}, {"cell_type": "code", "execution_count": 4, "metadata": {"tags": ["answer:q2"]}, "outputs": [], "source": "def get_remainder(m, n):\n    ### BEGIN SOLUTION\n\n\n### END SOLUTION"}], "metadata": {"celltoolbar": "Tags", "kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.1"}}, "nbformat": 4, "nbformat_minor": 4}
+{"cells": [{"cell_type": "markdown", "metadata": {"editable": true, "slideshow": {"slide_type": ""}, "tags": []}, "source": "# Class assignment\n\nWe will use this assignment to solidify our understanding of using Python to\ncarry out some numerical operations and also write functions.\n\nUse this notebook to write your answers in the cells as instructed, do your\nbest not to delete any of the cells that are already there.\n\n## Question one.\n\nUse python to obtain the remainder when dividing 21 by 5:\n\n\\\\[21 \\mod 5\\\\]"}, {"cell_type": "code", "execution_count": 3, "metadata": {"editable": true, "slideshow": {"slide_type": ""}, "tags": ["answer:q1"]}, "outputs": [], "source": "### BEGIN SOLUTION\n\n\n### END SOLUTION"}, {"cell_type": "markdown", "metadata": {"editable": true, "slideshow": {"slide_type": ""}, "tags": []}, "source": "## Question two.\n\nWrite a python function `get_remainder(m, n)` that returns the remainder\nthe remainder when dividing \\\\(m\\\\) by \\\\(n\\\\).\n\n\\\\[m \\mod n\\\\]"}, {"cell_type": "code", "execution_count": 6, "metadata": {"editable": true, "slideshow": {"slide_type": ""}, "tags": ["answer:q2"]}, "outputs": [], "source": "def get_remainder(m, n):\n    ### BEGIN SOLUTION\n\n\n### END SOLUTION"}], "metadata": {"celltoolbar": "Tags", "kernelspec": {"display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.4"}}, "nbformat": 4, "nbformat_minor": 4}

docs/conf.py

@@ -10,20 +10,9 @@
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 #
-import mock
 import os
 import sys
 
-MOCK_MODULES = [
-    "jupyter",
-    "typer",
-]
-
-
-for mod_name in MOCK_MODULES:
-    sys.modules[mod_name] = mock.Mock()
-
-
 import nbchkr
 
 # -- Project information -----------------------------------------------------
@@ -54,12 +43,10 @@
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-on_rtd = os.environ.get("READTHEDOCS", None) == "True"
-if not on_rtd:  # only import and set the theme if we're building docs locally
-    import sphinx_rtd_theme
+import sphinx_rtd_theme
 
-    html_theme = "sphinx_rtd_theme"
-    html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
+html_theme = "sphinx_rtd_theme"
+html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,

docs/howto/check_a_submission.rst

@@ -37,7 +37,7 @@ Reading in the source notebook :code:`main.ipynb` and removing relevant cells::
     >>> source_nb_path = pathlib.Path("main.ipynb")
     >>> source_nb_node = nbchkr.utils.read(nb_path=source_nb_path)
 
-Reading in the submitted notebook :code:`submitted.ipyb` and check that the tags
+Reading in the submitted notebook :code:`submitted.ipynb` and check that the tags
 match (if they do not match the checker will still work but the results should
 be confirmed manually)::
 
@@ -51,13 +51,15 @@ Now we will add the checks to the submission from :code:`main.ipynb` and run
 them::
 
     >>> nb_node = nbchkr.utils.add_checks(nb_node=nb_node, source_nb_node=source_nb_node)
-    >>> score, maximum_score, feedback_md = nbchkr.utils.check(nb_node=nb_node)
+    >>> score, maximum_score, feedback_md, passed_checks = nbchkr.utils.check(nb_node=nb_node)
     >>> score
     10
     >>> maximum_score
     11
     >>> feedback_md
-    '\n---\n\n## answer:q1\n\n1 / 1\n\n3 / 3\n\n---\n\n## answer:q2\n\nYou did not include a docstring. This is important to help document your code. \n\n\nIt is done  using triple quotation marks. For example:\n\ndef get_remainder(m, n):\n    """\n    This function returns the remainder of m when dividing by n\n    """\n    ...\n    \nUsing that it\'s possible to access the docstring, \none way to do this is to type: `get_remainder?` \n(which only works in Jupyter) or help(get_remainder).\n\nWe can also comment code using `#` but this is completely \nignored by Python so cannot be accessed in the same way.\n\n\n\n0 / 1\n\n6 / 6\n'
+    '\n---\n\n## answer:q1\n\n### Integer answer\n\n1 / 1\n\n### Correct answer\n...'
+    >>> passed_checks
+    {'Integer answer': True, 'Correct answer': True, 'Presence of docstring': False}
 
 Note that the :code:`nbrchkr.utils.check_tags_match`,
 :code:`nbchkr.utils.add_checks` and :code:`nbchkr.utils.check` functions can

docs/howto/write_an_assignment.rst

@@ -27,22 +27,38 @@ Add the :code:`answer:<uique_label>` tag to the cell.
 Write a check
 -------------
 
-In a :code:`code` cell write :code:`assert` statements to check specific
-elements of the answer::
-
-    assert <condition>, <error message>
-
-If the :code:`<condition>` is not meet the :code:`<error message>` will be
-written to the feedback on a submission.
+In a :code:`code` cell write statements to check any given property
+of a variable.
+To do this define a :code:`property_check` function that takes a variable
+:code:`variable`. Pass this :code:`property_check` function as well as a
+:code:`feedback_string` and the name of a variable to check the property of
+:code:`variable_string` to :code:`nbchkr.checks.check_variable_has_expected_property`
 
 Note that it is possible to refer to the output of a previous cell using
 :code:`_`.
 
+Here is a check for the previous output being even::
+
+    output = _
+
+    import nbchkr.checks
+    variable_string = "output"
+    feedback_string = "Your output is not even"
+
+    def check_even(variable):
+        return variable % 2 == 0
+
+    nbchkr.checks.check_variable_has_expected_property(
+        variable_string=variable_string,
+        feedback_string=feedback_string,
+        property_check=check_even,
+    )
+
 Add the :code:`score:<integer>` tag to the cell. The :code:`<integer>` is the
 value associated with this specific check. If the :code:`<condition>` is met
 then the :code:`<integer>` value will be added to the total score of a student.
 
-Optionally, you can also add the :code:`description:<string>` tag to the cell.
+Add the :code:`description:<string>` tag to the cell.
 This will add the :code:`<string>` to the feedback for that specific check. Note
 that spaces should be replaced with :code:`-` which will automatically be
 replaced in the feedback. For example: :code:`description:correct-answer` will
@@ -51,3 +67,27 @@ appear as :code:`### Correct answer` in the feedback.
 Note that it is possible to write multiple checks for a given answer. This can
 be done so as to programmatically offer varying levels of feedback for specific
 parts of the task.
+
+
+Property checks with arguments
+''''''''''''''''''''''''''''''
+
+Note that it is also possible to write property check functions that take
+keyword arguments and pass these to
+:code:`nbchkr.checks.check_variable_has_expected_property`. For example::
+
+    output = _
+
+    import nbchkr.checks
+    variable_string = "output"
+    feedback_string = "Your output is not even"
+
+    def check_divisibiliy_by_m(variable, m):
+        return variable % m == 0
+
+    nbchkr.checks.check_variable_has_expected_property(
+        variable_string=variable_string,
+        feedback_string=feedback_string,
+        property_check=check_divisibiliy_by_m,
+        m=2,
+    )

docs/tutorial/assignment/data.csv

@@ -1,4 +1,4 @@
-Submission filepath,Score,Maximum score,Tags match
-submissions/assignment_01.ipynb,2,11,True
-submissions/assignment_02.ipynb,10,11,True
-submissions/assignment_03.ipynb,4,11,False
+,Submission filepath,Score,Maximum score,Tags match,Integer answer,Correct answer,Presence of docstring,Run time
+0,submissions/assignment_01.ipynb,2,11,True,True,False,True,3.520920753479004
+1,submissions/assignment_02.ipynb,10,11,True,True,True,False,1.2404108047485352
+2,submissions/assignment_03.ipynb,4,11,False,True,False,False,1.5148279666900635