Allow specifying extra volumes (#129)

Allow specifying extra volumes (#129) + Pydantic refactor

Co-authored-by: marcin.zablocki <marcin.zablocki@getindata.com>

.github/workflows/prepare-release.yml

@@ -13,7 +13,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [3.7]
+        python-version: [3.8]
     env:
       PYTHON_PACKAGE: kedro_kubeflow
     steps:

.github/workflows/publish.yml

@@ -10,7 +10,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [3.7]
+        python-version: [3.8]
     env:
       PYTHON_PACKAGE: kedro_kubeflow
     steps:

.github/workflows/python-package.yml

@@ -16,7 +16,7 @@ jobs:
     - name: Setup python
       uses: actions/setup-python@v2.2.1
       with:
-        python-version: 3.7
+        python-version: 3.8
 
     - name: Setup virtualenv
       run: |
@@ -33,7 +33,7 @@ jobs:
     - name: Test with tox
       run: |
         pip install tox-pip-version
-        tox -v -e py37
+        tox -v -e py38
 
     - name: Report coverage
       uses: paambaati/codeclimate-action@v2.7.5

CHANGELOG.md

@@ -1,6 +1,8 @@
 # Changelog
 
 ## [Unreleased]
+- Added support for extra volumes per node 
+- Refactored configuration classes to Pydantic
 
 ## [0.6.4] - 2022-06-01
 

README.md

@@ -1,6 +1,6 @@
 # Kedro Kubeflow Plugin
 
-[![Python Version](https://img.shields.io/badge/python-3.7%20%7C%203.8-blue.svg)](https://github.com/getindata/kedro-kubeflow)
+[![Python Version](https://img.shields.io/badge/python-3.8-blue.svg)](https://github.com/getindata/kedro-kubeflow)
 [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) 
 [![SemVer](https://img.shields.io/badge/semver-2.0.0-green)](https://semver.org/)
 [![PyPI version](https://badge.fury.io/py/kedro-kubeflow.svg)](https://pypi.org/project/kedro-kubeflow/)

docs/source/02_installation/02_configuration.md

@@ -104,6 +104,18 @@ run_config:
       value: "gpu_workload"
       effect: "NoSchedule"
 
+  # Optional section to allow mounting additional volumes (such as EmptyDir)
+  # to specific nodes
+  extra_volumes:
+    tensorflow_step:
+    - mount_path: /dev/shm
+      volume:
+        name: shared_memory
+        empty_dir:
+          cls: V1EmptyDirVolumeSource
+          params:
+            medium: Memory
+
   # Optional section allowing adjustment of the resources
   # reservations and limits for the nodes
   resources:
@@ -151,3 +163,32 @@ can later inject in configuration file using `${name}` syntax.
 
 There are two special variables `KEDRO_CONFIG_COMMIT_ID`, `KEDRO_CONFIG_BRANCH_NAME` with support specifying default when variable is not set, 
 e.g. `${commit_id|dirty}`   
+
+## Extra volumes
+You can mount additional volumes (such as `emptyDir`) to specific nodes by using `extra_volumes` config node.
+The syntax of the configuration allows to define k8s SDK compatible class hierarchy similar to the way you would define it in the KFP DSL, e.g:
+```python
+# KFP DSL
+volume = dsl.PipelineVolume(volume=k8s.client.V1Volume(
+    name="shared_memory",
+    empty_dir=k8s.client.V1EmptyDirVolumeSource(medium='Memory')))
+
+training_op.add_pvolumes({'/dev/shm': volume})
+```
+will translate to the following Kedro-Kubeflow config:
+```yaml
+extra_volumes:
+  training_op:
+  - mount_path: /dev/shm
+    volume:
+      name: shared_memory
+      empty_dir:
+        cls: V1EmptyDirVolumeSource
+        params:
+          medium: Memory
+```
+
+In general, the `volume` key accepts a dictionary with the keys being the named parameters for the [V1Volume](https://github.com/kubernetes-client/python/blob/be9a47e57358e3701ad079c98e223d3437ba1f46/kubernetes/docs/V1Volume.md) and values being one of:
+* dictionary with `cls` and `params` keys (to define nested objects) - see `kedro_kubeflow.config.ObjectKwargs`
+* list of values / list of dictionaries (`kedro_kubeflow.config.ObjectKwargs`) as described above
+* values (`str`, `int` etc.)