Skip to content
Collective Knowledge framework helps to convert ad-hoc code, data and scripts into portable, customizable and reusable components with a simple Python API and integrated package manager; assemble automated workflows adaptable to Linux, MacOS, Windows and Android; crowdsource complex experiments; generate interactive papers, etc:
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bin fixed strange warnings for Python 3.5+ on Windows and Linux (when ins… Apr 23, 2017
ck
man/man1
tests Run ck tests as an external process if unable to import python CK module Sep 8, 2016
.coveragerc
.gitignore
.travis.yml
AUTHORS.txt improving docs Nov 18, 2018
CHANGES.txt
CONTRIBUTIONS.txt
COPYRIGHT.txt
ChangeLog
LICENSE.txt 2018 -> 2019 Jan 17, 2019
MANIFEST.in added set-env.sh Feb 1, 2019
README.md started new development version V1.9.7.1 Feb 11, 2019
appveyor.yml Increase coverage of kernel.py (#62) Sep 23, 2016
set-env.sh
setup.py started new development version V1.9.7.1 Feb 11, 2019

README.md

Downloads DOI License

Linux/MacOS: Build Status Windows: Windows Build status Coverage: Coverage Status

Documentation: wiki

Note that we have just completed a proof-of-concept stage with the great help from our partners, and now plan to gradually add more CK tutorials or improve CK documentation, specification and APIs in 2019. Please be patient, stay tuned or help the CK community via this open CK forum!

We designed Collective Knowledge (CK) as a very small, stable and portable framework with minimal dependencies to help researchers and developers quickly create, share and reuse extensible Python APIs and JSON meta descriptions to abstract any (evolving) code, data and hardware.

Though seemingly simple, such approach helps the community implement customizable workflows to automate, crowdsource and reproduce complex experiments such as AI/SW/HW autotuning and co-design while automatically adapting to any rapidly evolving software and hardware without the need for virtualization!

Unified CK APIs and JSON meta descriptions allow easy integration of CK workflows with popular tools and services such as GitHub, AWS, Docker, Singularity, Jupyter notebooks, Jenkins, Travis, etc. This, in turn, help to enable collaborative and reproducible R&D based on agile, DevOps, FAIR and Wikipedia principles (see CK motivation, main features and RESCUE-HPC workshop). It is now even possible to automatically generate interactive and reproducible articles with reusable research components.

CK supports our long-term vision to connect academia and industry to solve the real-world challenges, and to enable true open science. Please, have a look at several related use cases from our partners:

We hope that CK will help you accelerate your own R&D - just give CK a try and do not hesitate to provide your feedback via this public CK discussion group!

CK resources

Installation

If you have any issues with installation, please do not hesitate to tell us or open a GitHub ticket.

The minimal CK installation requires:

  • Python 2.7 or 3.3+ with PIP (limitation is mainly due to unitests). CK automatically adapts to Python 2 or 3 and provides extra API to let users write workflows for any Python version;
  • Git command line client;
  • wget (Linux/MacOS).

Ubuntu

$ sudo apt-get install python3 python3-pip git wget
$ sudo pip install ck

$ ck version

If you don't have sudo (root) access, you can install CK in your user space either via PIP (note that you should have directory $HOME/.local/bin in your PATH)

$ pip install ck --user

$ ck version

or from GitHub:

$ git clone http://github.com/ctuning/ck
$ export PATH=$PWD/ck/bin:$PATH
$ export PYTHONPATH=$PWD/ck:$PYTHONPATH

$ ck version

You can also set CK environment variables and test dependencies using provided script as follows:

$ git clone http://github.com/ctuning/ck
$ . ./set-env.sh

$ ck version

MacOS

$ brew install python3
$ brew install git
$ brew install wget
$ pip install ck

$ ck version

You can also install CK via GitHub as described in the "Ubuntu" section above.

Windows

You can download a CK installer which already includes Git 2.20.1 and Python 3.7.2 from Zenodo using this link.

Just unzip it and run one of the following scripts:

  1. install-pip.bat to install CK via PIP
  2. install-github.bat to install CK from GitHub

These scripts will install Python in your dedicated directory and will ask you to add several environment variables to your system (just copy/paste them) - that's all! You can then test CK as follows:

$ ck version

Alternatively you can download and install Git and Python yourself:

You can then install CK via PIP:

$ pip install ck

$ ck version

You can also install CK from GitHub:

 $ git clone https://github.com/ctuning/ck.git ck-master
 $ set PATH={CURRENT PATH}\ck-master\bin;%PATH%
 $ set PYTHONPATH={CURRENT PATH}\ck-master;%PYTHONPATH%

Installation customization

Check this documentation about CK customization. For example, you can change directories with CK repositories and packages or change search paths during software detection (useful for HPC setups).

Basic usage example (automatically detect compilers, install packages, compile and run benchmarks)

Test ck:

$ ck version

Pull CK repositories with benchmarks, data sets, software detection plugins, packages, etc:

$ ck pull repo:ck-crowdtuning

See the list of installed CK repos:

$ ck ls repo | sort

Find where CK repository with benchmarks is installed on your machine and browse it to get familiar with the structure (consistent across all repos):

$ ck where repo:ctuning-programs

Detect your platform properties via extensible CK plugins as follows (needed to unify benchmarking across diverse platforms with Linux, Windows, MacOS and Android):

$ ck detect platform

Check JSON output

$ ck detect platform --out=json

Now detect available compilers on your machine and register virtual environments in the CK:

$ ck detect soft --tags=compiler,gcc
$ ck detect soft --tags=compiler,llvm
$ ck detect soft --tags=compiler,icc

See virtual environments in the CK:

$ ck show env

Find and explore CK env entries:

$ ck search env --tags=compiler

We recommend to setup CK to install new packages inside CK virtual env entries:

$ ck set kernel var.install_to_env=yes

Try to install LLVM binary via CK packages:

$ ck install package --tags=llvm

Check available data sets:

$ ck search dataset
$ ck search dataset --tags=jpeg

Now you can compile and run shared benchmarks with some data sets, benchmark and crowd-tune some C program.

$ ck ls program

Let's check the CK JSON meta for benchmark "cbench-automotive-susan":

$ ck load program:cbench-automotive-susan --min

Now let's compile and run it:

$ ck compile program:cbench-automotive-susan --speed
$ ck run program:cbench-automotive-susan

You can now benchmark this program (CK will execute several times while monitoring the state of the system):

$ ck benchmark program:cbench-automotive-susan

Finally, you can autotune this program using shared CK autotuning scenarios, record results and reply them:

$ ck autotune program:cbench-automotive-susan

You can also crowdtune this program, i.e. autotune it while sharig best results in the public repository:

$ ck crowdtune program:cbench-automotive-susan

You can now add (and later customize) your own program workflow using shared templates as follows:

$ ck add program:my-new-program

When CK asks you to select a template, please choose "C program "Hello world". You can then immediately compile and run your C program as follows:

$ ck compile program:my-new-program --speed
$ ck run program:my-new-program
$ ck run program:my-new-program --env.CK_VAR1=222

Advanced usage example (image classification via TensorFlow and Caffe)

Get shared ck-tensorflow repository with all dependencies:

$ ck pull repo:ck-tensorflow

Now install CPU-version of TensorFlow via CK packages:

$ ck install package --tags=lib,tensorflow,vcpu,vprebuilt,v1.11.0

Check that it's installed fine:

$ ck show env --tags=lib,tensorflow

You can find a path to a given entry (with TF installation) as follows:

$ ck find env:{env UID from above list}

Run CK virtual environment and test TF:

$ ck virtual env --tags=lib,tensorflow
$ ipython
> import tensorflow as tf

Run CK classification workflow example using installed TF:

$ ck run program:tensorflow --cmd_key=classify

Now you can try a more complex example to build Caffe with CUDA support and run classification. Note that CK should automatically detect your CUDA compilers, libraries and other deps or install missing packages:

$ ck pull repo --url=https://github.com/dividiti/ck-caffe
$ ck install package:lib-caffe-bvlc-master-cuda-universal
$ ck run program:caffe --cmd_key=classify

You can see how to install Caffe for Linux, MacOS, Windows and Android via CK here.

Adding your own repository and API:

You can add your own repository as follows:

$ ck add repo:my-repo --quiet

$ ck where repo:my-repo

$ ck ls repo:my-*

Now you add a Python module to prepare APIs:

$ ck add my-repo:module:hello

It will create an entry "module:hello" in the my-repo with a dummy module.py:

$ ls `ck find module:hello`

Now you can add "say" API to the CK python module "hello":

$ ck add_action module:hello --func=say

CK will add a dummy function "say" in the module.py in "module:hello" which you can immediately use (!):

$ ck say hello
$ ck say hello --out=json

Furthermore, you can now create a data entry for your module "hello":

$ ck add hello:world --tags=cool,api

$ ck search hello --tags=api

$ ck say hello:world

$ ck ren hello:world hello:team

$ ck say hello:team

Such approach allowed our partners to gradually abstract complex AI, ML, and quantum experiments via shared CK APIs, crowdsource experiments, and even automatically generate reproducible and interactive articles with reusable research components!

Trying CK from a Docker image

You can try CK using the following Docker image:

 $ (sudo) docker run -it ctuning/ck-ubuntu-18.04

Note that we added Docker automation to CK to help evaluate artifacts at the conferences, share interactive and reproducible articles, crowdsource experiments and so on.

For example, you can participate in GCC or LLVM crowd-tuning on your machine as follows:

 $ (sudo) docker run ck-crowdtune-gcc
 $ (sudo) docker run ck-crowdtune-llvm

Top optimization results are continuously aggregated in the live CK repository: http://cKnowledge.org/repo .

Citing CK

See CK publications.

CK authors

License

  • Permissive 3-clause BSD license. (See LICENSE.txt for more details).

Acknowledgments

CK development is coordinated by the cTuning foundation and dividiti. We would like to thank the TETRACOM 609491 Coordination Action for initial funding and all our partners for continuing support. We are also extremely grateful to all volunteers for their valuable feedback and contributions.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.