Merge pull request #22 from getindata/release-0.1.9

Release 0.1.9

.codeclimate.yml

@@ -0,0 +1,3 @@
+plugins:
+  pylint:
+    enabled: true

.github/CODEOWNERS

@@ -0,0 +1,2 @@
+# Define global code owners
+* @empe @szczeles

.github/pull_request_template.md

@@ -0,0 +1,9 @@
+#### Description
+
+`describe the purpose of the change here` 
+
+Resolves `<issue nr here>`
+
+##### PR Checklist
+- [ ] Tests added
+- [ ] [Changelog](CHANGELOG.md) updated 

.github/workflows/python-package.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches: 
       - master
+      - develop
   pull_request:
 
 jobs:
@@ -16,15 +17,28 @@ jobs:
       uses: actions/setup-python@v2.2.1
       with:
         python-version: 3.7
+
     - name: Setup virtualenv
       run: |
         python -V
         python -m pip install virtualenv
         virtualenv venv
         source venv/bin/activate
    
+    - name: Check pre-commit status
+      run: |
+        pip install -r requirements-dev.txt
+        pre-commit run --all-files
+
     - name: Test with tox
       run: |
-        python -m pip install flake8 tox
-        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
         tox -e py37
+
+    - name: Report coverage
+      uses: paambaati/codeclimate-action@v2.7.5
+      env:
+        CC_TEST_REPORTER_ID: ${{ secrets.CODE_CLIMATE }}
+      with:
+        coverageCommand: coverage xml
+        debug: true
+        coverageLocations: coverage.xml:coverage.py

.isort.cfg

@@ -0,0 +1,2 @@
+[settings]
+known_third_party = click,google,kedro,kfp,kubernetes,semver,setuptools,tabulate

.pre-commit-config.yaml

@@ -0,0 +1,13 @@
+repos:
+- repo: https://github.com/pre-commit/mirrors-isort
+  rev: v4.3.21
+  hooks:
+  - id: isort
+- repo: https://github.com/psf/black
+  rev: stable
+  hooks:
+  - id: black
+- repo: https://github.com/pre-commit/pre-commit-hooks
+  rev: v2.3.0
+  hooks:
+  - id: flake8

CHANGELOG.md

@@ -2,16 +2,24 @@
 
 ## [Unreleased]
 
+## [0.1.9] - 2021-01-08
+
+-   Support for MLFlow - if the package is installed then additional step is added with parent run init. Then all separate nodes runs register under this run.
+-   Support for inter-steps volume: setup (one volume per pipeline run), initial load (the content of `data/` directory from the image and mount to all the steps for artifacts passing.
+-   `kubeflow init` command added to generate sample configuration file.
+
 ## [0.1.8] - 2021-01-05
 
 ### Added
 
--   *Initial release* of kedro-kubeflow plugin
+-   _Initial release_ of kedro-kubeflow plugin
 -   Ability to run an anonymous pipeline once as within a specified experiment `kedro kubeflow run-once`.
--   Ability to upload pipeline `kedro kubeflow upload`.
+-   Ability to upload pipeline `kedro kubeflow upload-pipeline`.
 -   Method to schedule runs for most recent version of given pipeline `kedro kubeflow schedule` 
 -   Shortcut to open UI for pipelines using `kedro kubeflow ui` 
 
-[Unreleased]: https://github.com/getindata/kedro-kubeflow/compare/0.1.8...HEAD
+[Unreleased]: https://github.com/getindata/kedro-kubeflow/compare/0.1.9...HEAD
+
+[0.1.9]: https://github.com/getindata/kedro-kubeflow/compare/0.1.8...0.1.9
 
 [0.1.8]: https://github.com/getindata/kedro-kubeflow/compare/ea219ae5f70e726b7afc9d0864da4b6649e4deaf...0.1.8

CONTRIBUTING.md

@@ -1,15 +1,25 @@
-# Release workflow
+
+## PR Guidelines
+1. Fork branch from `develop`.
+1. Ensure to provide unit tests for new functionality.
+1. Install dev requirements: `pip install -r requirements-dev.txt` and setup a hook: `pre-commit install`
+1. Update documentation accordingly.
+1. Update [changelog](CHANGELOG.md) according to ["Keep a changelog"](https://keepachangelog.com/en/1.0.0/) guidelines.
+1. Squash changes with a single commit as much as possible and ensure verbose PR name.
+1. Open a PR against `develop`
+
+*We reserve the right to take over and modify or abandon PRs that do not match the workflow or are abandoned.* 
+
+## Release workflow
 
 1. Create the release candidate:
-    - Go to the `create-release-candidate action`
+    - Go to the [Prepare release](https://github.com/getindata/kedro-kubeflow/actions?query=workflow%3A%22Prepare+release%22) action.
     - Click "Run workflow"
-    - Enter the part of the version to bump (one of `<major>.<minor>.<patch>`)
+    - Enter the part of the version to bump (one of `<major>.<minor>.<patch>`). Minor (x.**x**.x) is a default. 
 2. If the workflow has run sucessfully:
     - Go to the newly openened PR named `Release candidate `<version>`
-    - Check that changelog and version have been properly updated.
-    - *(If everything is normal, skip this step)* Eventually pull the branch and make changes if necessary
+    - Check that changelog and version have been properly updated. If not pull the branch and apply manual changes if necessary.
     - Merge the PR to master
-3. Checkout the `publish workflow`to see if:
-    - The package has been uploaded on PyPI sucessfully
+3. Checkout the [Publish](https://github.com/getindata/kedro-kubeflow/actions?query=workflow%3APublish) workflow to see if:
+    - The package has been uploaded on PyPI successfully
     - The changes have been merged back to develop
-4. If the pipeline has failed, please raise an issue to correct the CI, and ensure merge on develop "by hand"".

README.md

@@ -1,9 +1,13 @@
 # Kedro Kubeflow Plugin
 
-[![Python Version](https://img.shields.io/badge/python-3.6%20%7C%203.7%20%7C%203.8-blue.svg)](https://github.com/getindata/kedro-kubeflow)
+[![Python Version](https://img.shields.io/badge/python-3.7%20%7C%203.8-blue.svg)](https://github.com/getindata/kedro-kubeflow)
 [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) 
 [![SemVer](https://img.shields.io/badge/semver-2.0.0-green)](https://semver.org/)
+[![PyPI version](https://badge.fury.io/py/kedro-kubeflow.svg)](https://pypi.org/project/kedro-kubeflow/)
+[![Downloads](https://pepy.tech/badge/kedro-kubeflow)](https://pepy.tech/project/kedro-kubeflow) 
 
+[![Maintainability](https://api.codeclimate.com/v1/badges/fff07cbd2e5012a045a3/maintainability)](https://codeclimate.com/github/getindata/kedro-kubeflow/maintainability) 
+[![Test Coverage](https://api.codeclimate.com/v1/badges/fff07cbd2e5012a045a3/test_coverage)](https://codeclimate.com/github/getindata/kedro-kubeflow/test_coverage)
 ## About
 
 The main purpose of this plugin is to enable running kedro pipeline on Kubeflow Pipelines. It supports translation from 
@@ -24,9 +28,29 @@ Usage: kedro kubeflow [OPTIONS] COMMAND [ARGS]...
  
  Commands:
    compile          Translates Kedro pipeline into YAML file with Kubeflow pipeline definition
+   init             Initializes configuration for the plugin
    list-pipelines   List deployed pipeline definitions
    run-once         Deploy pipeline as a single run within given experiment.
    schedule         Schedules recurring execution of latest version of the pipeline
    ui               Open Kubeflow Pipelines UI in new browser tab
    upload-pipeline  Uploads pipeline to Kubeflow server
 ```
+
+## Configuration file
+
+`kedro init` generates configuration file for the plugin, but users may want
+to adjust it to the requirements of the environment:
+
+```
+host: http://10.43.77.224
+
+run_config:
+  image: new-kedro-project
+  experiment_name: New Kedro Project
+  run_name: New Kedro Project
+  wait_for_completion: False
+  volume:
+    storageclass: # default
+    size: 1Gi
+    access_modes: [ReadWriteOnce]
+```

kedro_kubeflow/__init__.py

@@ -1,4 +1,3 @@
 """kedro_kubeflow."""
 
-version = "0.1.8"
-
+version = "0.1.9"

kedro_kubeflow/hooks.py

@@ -0,0 +1,38 @@
+import os
+from typing import Any, Dict
+
+from kedro.framework.context import load_context
+from kedro.framework.hooks import hook_impl
+from kedro.io import DataCatalog
+from kedro.pipeline import Pipeline
+
+from .utils import is_mlflow_enabled
+
+
+class MLFlowActivateParentHook:
+    @hook_impl
+    def before_pipeline_run(
+        self,
+        run_params: Dict[str, Any],
+        pipeline: Pipeline,
+        catalog: DataCatalog,
+    ) -> None:
+
+        if not is_mlflow_enabled():
+            return
+
+        import mlflow
+        from kedro_mlflow.framework.context import get_mlflow_config
+
+        context = load_context(
+            project_path=run_params["project_path"],
+            env=run_params["env"],
+            extra_params=run_params["extra_params"],
+        )
+        mlflow_conf = get_mlflow_config(context)
+        mlflow_conf.setup(context)
+
+        mlflow.start_run(run_id=os.getenv("MLFLOW_PARENT_ID"))
+
+
+mlflow_activate_parent_run = MLFlowActivateParentHook()