Merge pull request #156 from getindata/docs-update

Docs update and spellchecking

.github/workflows/spellcheck.yml

@@ -0,0 +1,20 @@
+# This is workflow for spell checking using PySpelling lib (https://pypi.org/project/pyspelling/)
+name: Spellcheck
+# Controls when the action will run.
+on:
+  # Triggers the workflow on pull request events only if docs files have changed
+  pull_request:
+    paths: "docs/**"
+  # Allows you to run this workflow manually from the Actions tab
+  workflow_dispatch:
+# A workflow run is made up of one or more jobs that can run sequentially or in parallel
+jobs:
+  spellcheck:
+    # The type of runner that the job will run on
+    runs-on: ubuntu-latest
+    # Steps represent a sequence of tasks that will be executed as part of the job
+    steps:
+      # Spellcheck
+      - uses: actions/checkout@v2
+      - uses: rojopolis/spellcheck-github-actions@0.25.0
+        name: Spellcheck

.gitignore

@@ -123,6 +123,9 @@ venv.bak/
 # mypy
 .mypy_cache/
 
+# pyspelling
+dictionary.dic
+
 docs/_build
 
 spaceflights/

.pre-commit-config.yaml

@@ -1,15 +1,19 @@
 repos:
 - repo: https://github.com/pycqa/isort
-  rev: 5.5.4
+  rev: 5.10.1
   hooks:
   - id: isort
     args: ["--profile", "black", "--line-length=79"]
 - repo: https://github.com/psf/black
-  rev: stable
+  rev: 22.6.0
   hooks:
   - id: black
 - repo: https://github.com/pre-commit/pre-commit-hooks
   rev: v2.3.0
   hooks:
   - id: flake8
     args: ['--ignore=E203,W503'] # see https://github.com/psf/black/issues/315 https://github.com/psf/black/issues/52
+- repo: https://github.com/getindata/py-pre-commit-hooks
+  rev: v0.1.3
+  hooks:
+  - id: pyspelling-docker

.spellcheck.yml

@@ -0,0 +1,35 @@
+spellchecker: aspell
+matrix:
+  - name: Check the english docs
+    aspell:
+      ignore-case: ""
+      lang: en
+    sources:
+    - "docs/**/*.md"
+    - "CONTRIBUTING.md"
+    - "README.md"
+    default_encoding: utf-8
+    pipeline:
+    - pyspelling.filters.context:
+        context_visible_first: true
+        escapes: \\[\\`~]
+        delimiters:
+        # Ignore multiline content between fences (fences can have 3 or more back ticks)
+        # ```
+        # content
+        # ```
+        - open: '^(?s)(?P<open>`{1,3})[^`]'
+          close: '(?P=open)'
+        # Ignore text between inline back ticks
+        - open: '(?P<open>`)[^`]'
+          close: '(?P=open)'
+        # Ignore text in brackets [] and () 
+        - open: '\['
+          close: '\]'
+        - open: '\('
+          close: '\)'
+        - open: '\{'
+          close: '\}'
+    dictionary:
+      wordlists:
+      - docs/spellcheck_exceptions.txt

CHANGELOG.md

@@ -1,6 +1,12 @@
 # Changelog
 
 ## [Unreleased]
+- Added templating capabilities to docs generator and used them in the docs for kedro versioning
+- Added pre-commit hook for pyspelling check
+- Changed sphinx markdown engine to myst_parser
+- Added CI for spellchecking the documentation with configuration for myst
+- Updated documentation quickstart to workaround known issues and make it work on local kind cluster
+- Updated documentation - added contributing guidelines and setup tips
 - Merged e2e and unittest workflows. Now e2e runs only if unittests succeed.
 - Changed default resource limits in `kubeflow.yaml` config
 - Added --wait-for-completion and --timeout for `kedro kubeflow run-once` command

CONTRIBUTING.md

@@ -1,23 +1,69 @@
+# Contributing guidelines 
 
-## PR Guidelines
+Everyone is welcome to contribute to the code of this plugin, however we have some automation and guidelines you should get familiar with first.
+
+### PR Guidelines
 1. Fork branch from `develop`.
 1. Ensure to provide unit tests for new functionality.
 1. Install dev requirements: `pip install -r requirements-dev.txt` and setup a hook: `pre-commit install`
+1. For commits we follow [angular commit messages guideline](https://github.com/angular/angular/blob/main/CONTRIBUTING.md#commit).
 1. Update documentation accordingly.
-1. Update [changelog](CHANGELOG.md) according to ["Keep a changelog"](https://keepachangelog.com/en/1.0.0/) guidelines.
+1. Update [changelog](https://github.com/getindata/kedro-kubeflow/blob/develop/CHANGELOG.md) according to ["Keep a changelog"](https://keepachangelog.com/en/1.0.0/) guidelines.
 1. Squash changes with a single commit as much as possible and ensure verbose PR name.
 1. Open a PR against `develop`
 
 *We reserve the right to take over and modify or abandon PRs that do not match the workflow or are abandoned.* 
 
-## Release workflow
+(updating-the-documentation)=
+### Updating the documentation
+
+For documentation updates we have `docs/Makefile` that runs `Sphinx` to update the [readthedocs](https://kedro-kubeflow.readthedocs.io).
+
+There is automation using github actions that checks the spelling of the docs. [Links](updating-the-documentation), `inline blocks escaped with back ticks` \`...\` and 
+```
+triple backtick blocks
+```
+are omitted. 
+
+ In order to add new spelling exceptions, append them to the `docs/spellcheck_exceptions.txt` file.
+
+For documentation syntax, we use Markdown with [myst](https://myst-parser.readthedocs.io/en/latest/syntax/syntax.html) parser.
+
+##### Templating
+
+We have `jinja` available to be used in documentation. The variables are defined in `docs/conf.py` in `myst_substitutions`. By default the following are available:
+
+ - `release` - the package version with which it was built
+ - `req_<package>` - the specification of version package requirement range in `setup.py`
+ - `req_lower_<package>` - the specification of version package requirement lower bound in `setup.py`
+ - `req_upper_<package>` - the specification of version package requirement upper bound in `setup.py`
+ - `req_build_<package>` - the specification of version package with which it was built
+
+The templates do not work inside code fences or links. You can workaround that by putting whole code fence inside `jinja` and then use the formatting inside. See the [substitution documentation](https://myst-parser.readthedocs.io/en/latest/syntax/optional.html#substitutions-with-jinja2) for details.
+
+```{warning}
+The substitutions do not seem to be fully `jinja2` capable. While expressions `{{ }}` do work, the control statements `{% %}` do not.
+```
+
+### Pre-commit and github hooks
+
+For linting and keeping code clean we use pre-commit package to join with github hooks. Use it by doing:
+
+```console
+$ pip install pre-commit
+$ pre-commit install
+```
+
+You can test github actions locally with [act](https://github.com/nektos/act).
+
+### Release workflow
 
 1. Create the release candidate:
     - Go to the [Prepare release](https://github.com/getindata/kedro-kubeflow/actions?query=workflow%3A%22Prepare+release%22) action.
     - Click "Run workflow"
     - Enter the part of the version to bump (one of `<major>.<minor>.<patch>`). Minor (x.**x**.x) is a default. 
-2. If the workflow has run sucessfully:
-    - Go to the newly openened PR named `Release candidate `<version>`
+2. If the workflow has run successfully:
+    - Go to the newly opened PR named `Release candidate <version>`
     - Check that changelog and version have been properly updated. If not pull the branch and apply manual changes if necessary.
     - Merge the PR to master
 3. Checkout the [Publish](https://github.com/getindata/kedro-kubeflow/actions?query=workflow%3APublish) workflow to see if:

docs/conf.py

@@ -14,7 +14,11 @@
 # import sys
 # sys.path.insert(0, os.path.abspath('.'))
 import re
+from pprint import pprint
 
+from pip._vendor import pkg_resources
+
+from kedro_kubeflow import __name__ as _package_name
 from kedro_kubeflow import version as release
 
 # -- Project information -----------------------------------------------------
@@ -23,9 +27,66 @@
 copyright = "2020, GetInData"
 author = "GetInData"
 
+myst_substitutions = {
+    "tested_kedro": "0.17.7",
+    "release": release,
+}
+
 # The full version, including alpha/beta/rc tags
 version = re.match(r"^([0-9]+\.[0-9]+).*", release).group(1)
+_package_name = _package_name.replace("_", "-")
+_package = pkg_resources.working_set.by_key[_package_name]
+
+
+# Extending keys for subsitutions with versions of package
+def update_templates_with_requirements(packages_set, label):
+    """Local function for updating template labels with requirements"""
+    myst_substitutions.update({label + p.name: str(p) for p in packages_set})
+
+    built_packages = {}
+    for p in packages_set:
+        try:
+            req_label = label + "build_" + p.name
+            built_packages[req_label] = pkg_resources.get_distribution(
+                p
+            ).version
+        except pkg_resources.DistributionNotFound:
+            pass
+        myst_substitutions.update(built_packages)
+
+    conditions = {
+        "upper": ["<", "<=", "~=", "==", "==="],
+        "lower": [">", ">=", "~=", "==", "==="],
+    }
+    for k, cond in conditions.items():
+        myst_substitutions.update(
+            {
+                label
+                + k
+                + "_"
+                + p.name: "".join(
+                    [
+                        "".join(i)
+                        for i in filter(lambda x: x[0] in cond, p.specs)
+                    ]
+                )
+                for p in packages_set
+            }
+        )
+
+
+base_requirements = set(_package.requires())
+extra_requires = {
+    extra: set(_package.requires(extras=(extra,))) - base_requirements
+    for extra in _package.extras
+}
+update_templates_with_requirements(base_requirements, "req_")
+for extra, reqs in extra_requires.items():
+    update_templates_with_requirements(reqs, f"req_{extra}_")
+
 
+print("Available patterns for substituion:")
+pprint(myst_substitutions)
 
 # -- General configuration ---------------------------------------------------
 
@@ -43,9 +104,14 @@
     # "sphinx.ext.ifconfig",
     # "sphinx.ext.viewcode",
     # "sphinx.ext.mathjax",
-    "recommonmark",
+    "myst_parser",
     "sphinx_rtd_theme",
 ]
+myst_enable_extensions = [
+    "replacements",
+    "strikethrough",
+    "substitution",
+]
 
 # Add any paths that contain templates here, relative to this directory.
 
@@ -74,8 +140,8 @@
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ["_static"]
+# html_static_path = ["_static"]
 
-language = None
+language = "en"
 
 pygments_style = "sphinx"

docs/index.rst

@@ -13,6 +13,7 @@ Welcome to Kedro Kubeflow Plugin's documentation!
    Introduction <source/01_introduction/01_intro.md>
    Installation <source/02_installation/index.rst>
    Getting Started <source/03_getting_started/index.rst>
+   Contributing <source/04_contributing/index.rst>
 
 Indices and tables
 ==================

docs/requirements.txt

@@ -1,3 +1,4 @@
-sphinx
+sphinx~=5.0.2
 sphinx_rtd_theme
 recommonmark
+myst-parser

docs/source/01_introduction/01_intro.md

@@ -17,5 +17,5 @@ docker images that cover not only the source itself, but also all the libraries
 entire execution environment. Portability is also one of the key Kedro qualities, as 
 the pipelines must be versionable and packageable. Kedro, with 
 [Kedro-docker](https://github.com/quantumblacklabs/kedro-docker) plugin does a fantastic 
-job to achieve this and Kubeflow looks like a nice addon to run the pipelines 
+job to achieve this and Kubeflow looks like a nice add-on to run the pipelines 
 on powerful remote Kubernetes clusters.

docs/source/02_installation/01_installation.md

@@ -2,17 +2,13 @@
 
 ## Kedro setup
 
-First, you need to install base Kedro package in ``<17.0`` version
+First, you need to install base Kedro package in {{req_kedro}} version
 
-> Kedro 17.0 is supported by kedro-kubeflow, but [not by kedro-mlflow](https://github.com/Galileo-Galilei/kedro-mlflow/issues/144) yet, so the latest version from 0.16 family is recommended.
-
-```console
-$ pip install 'kedro<0.17'
-```
+{{"```console\n$ pip install '{kedro}'\n```".format(kedro=req_kedro)}}
 
 ## Plugin installation
 
-### Install from PyPI
+### Install from `PyPI`
 
 You can install ``kedro-kubeflow`` plugin from ``PyPi`` with `pip`:
 
@@ -30,7 +26,7 @@ pip install git+https://github.com/getindata/kedro-kubeflow.git@develop
 
 ## Available commands
 
-You can check available commands by going into project directory and runnning:
+You can check available commands by going into project directory and running:
 
 ```console
 $ kedro kubeflow