Skip to content
A Python library for building robust production-ready data and analytics pipelines.
Branch: develop
Clone or download
yetudada [KED-828] Add in Databricks Workflow information (#91)
* Add in Databricks Workflow information
Latest commit 95a0239 Jul 23, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci Remove java installation steps as kedro_docker image fixed (#139) Jul 15, 2019
.github Remove requirement to make users add labels on issues (#120) Jul 2, 2019
docs [KED-828] Add in Databricks Workflow information (#91) Jul 23, 2019
extras [KED-826] Use KedroContext class in project template (#111) Jul 16, 2019
features [KED-826] Use KedroContext class in project template (#111) Jul 16, 2019
img [KED-821] Updated images and README.md for Kedro-Viz release (#62) Jul 9, 2019
kedro Modularize default load and save argument handling (#15) Jul 23, 2019
tests Modularize default load and save argument handling (#15) Jul 23, 2019
tools Remove more non ASCII stuff (#113) Jun 28, 2019
.coveragerc First commit of version 0.14 May 16, 2019
.flake8 First commit of version 0.14 May 16, 2019
.gitignore Add NodeJS package script to watch and serve changes on Docs CSS (#47) Jun 4, 2019
.isort.cfg [KED-784] Create an `HDFS3Dataset` (#42) May 30, 2019
.pylintrc First commit of version 0.14 May 16, 2019
.readthedocs.yml Create .readthedocs.yml (#22) May 17, 2019
CODEOWNERS First commit of version 0.14 May 16, 2019
CODE_OF_CONDUCT.md Minor modifications to setup.py and template files (#35) Jun 3, 2019
CONTRIBUTING.md [KED-854] Modifications to plugin documentation (#109) Jul 2, 2019
LICENSE.md Remove all smart quotes (#103) Jun 26, 2019
MANIFEST.in First commit of version 0.14 May 16, 2019
Makefile Fix mypy errors and add mypy to make lint (#128) Jul 10, 2019
README.md [KED-821] Updated images and README.md for Kedro-Viz release (#62) Jul 9, 2019
RELEASE.md [KED-881] Run argument that makes it easy for users to run one node a… Jul 22, 2019
appveyor.yml First commit of version 0.14 May 16, 2019
behave.ini First commit of version 0.14 May 16, 2019
legal_header.txt Remove all smart quotes (#103) Jun 26, 2019
pyproject.toml First commit of version 0.14 May 16, 2019
requirements.txt Fix S3 datasets after s3fs update (#144) Jul 18, 2019
setup.cfg [KED-803] Kedro project loader for IPython (#61) Jun 13, 2019
setup.py Make PEP 561 compatible packages (#53) Jul 15, 2019
test_requirements.txt Fix mypy errors and add mypy to make lint (#128) Jul 10, 2019

README.md

Kedro Logo Banner

develop master
CircleCI CircleCI
Build status Build status

License Python Version PyPI version Documentation Code Style: Black Downloads

What is Kedro?

"The centre of your data pipeline."

Kedro is a workflow development tool that helps you build data pipelines that are robust, scalable, deployable, reproducible and versioned. We provide a standard approach so that you can:

  • spend more time building your data pipeline,
  • worry less about how to write production-ready code,
  • standardise the way that your team collaborates across your project,
  • work more efficiently.

Kedro was originally designed by Aris Valtazanos and Nikolaos Tsaousis to solve challenges they faced in their project work.

How do I install Kedro?

kedro is a Python package. To install it, simply run:

pip install kedro

For more detailed installation instructions, including how to setup Python virtual environments, please visit our installation guide.

What are the main features of Kedro?

1. Project template and coding standards

  • A standard and easy-to-use project template
  • Configuration for credentials, logging, data loading and Jupyter Notebooks / Lab
  • Test-driven development using pytest
  • Sphinx integration to produce well-documented code

2. Data abstraction and versioning

  • Separation of the compute layer from the data handling layer, including support for different data formats and storage options
  • Versioning for your data sets and machine learning models

3. Modularity and pipeline abstraction

  • Support for pure Python functions, nodes, to break large chunks of code into small independent sections
  • Automatic resolution of dependencies between nodes
  • Visualise your data pipeline with Kedro-Viz, a tool that shows the pipeline structure of Kedro projects

Note: Read our FAQs to learn how we differ from workflow managers like Airflow and Luigi.

Kedro-Viz Pipeline Visualisation A pipeline visualisation generated using Kedro-Viz

4. Feature extensibility

  • A plugin system that injects commands into the Kedro command line interface (CLI)
  • List of officially supported plugins:
    • (coming soon) Kedro-Airflow, making it easy to prototype your data pipeline in Kedro before deploying to Airflow, a workflow scheduler
    • Kedro-Docker, a tool for packaging and shipping Kedro projects within containers
  • Kedro can be deployed locally, on-premise and cloud (AWS, Azure and GCP) servers, or clusters (EMR, Azure HDinsight, GCP and Databricks)

How do I use Kedro?

Our documentation explains:

  • A typical Kedro workflow
  • How to set up the project configuration
  • Building your first pipeline
  • How to use the CLI offered by kedro_cli.py (kedro new, kedro run, ...)

Note: The CLI is a convenient tool for being able to run kedro commands but you can also invoke the Kedro CLI as a Python module with python -m kedro

How do I find Kedro documentation?

This CLI command will open the documentation for your current version of Kedro in a browser:

kedro docs

Documentation for the latest stable release can be found here. Check these out first:

Can I contribute?

Yes! Want to help build Kedro? Check out our guide to contributing.

How do I upgrade Kedro?

We use Semantic Versioning. The best way to safely upgrade is to check our release notes for any notable breaking changes.

Once Kedro is installed, you can check your version as follows:

kedro --version

To later upgrade Kedro to a different version, simply run:

pip install kedro -U

What licence do you use?

Kedro is licensed under the Apache 2.0 License.

You can’t perform that action at this time.