Integration of TensorFlow to Collective Knowledge workflow framework to provide unified CK JSON API for AI (customized builds across diverse libraries and hardware, unified AI API, collaborative experiments, performance optimization and model/data set tuning):
Python C++ Shell Makefile Batchfile HTML Other
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.cm
dataset changing TF dataset benchmark names ... May 16, 2017
module module:model.tensorflowapi implementation added Aug 16, 2017
package
program small fixes for tf programs Aug 2, 2018
script processing CK_BAZEL_CACHE_DIR it TF installation scripts Aug 14, 2018
soft Various new TF packages (#73) Jul 26, 2018
.ckr.json Add repo dependencies: ck-autotuning, ck-math. Mar 8, 2017
.gitignore Packages for building TensorFlow static C++ API (#61) May 22, 2018
.travis.yml testfix: ensuring the trailing space is included for uniqueness Aug 10, 2018
AUTHORS clean up Nov 8, 2016
CHANGES fixed python issue #74 Jul 31, 2018
CONTRIBUTIONS contribution info Jul 31, 2018
COPYRIGHT added TF 1.5.0 (CPU & GPU). Also added check for version ranges for C… Feb 22, 2018
LICENSE
README.md added link to ck-pytorch Jul 3, 2018
appveyor.yml
requirements.txt

README.md

Unification of AI for collaborative experimentation and optimization using Collective Knowledge workflow framework with common JSON API

logo logo License

Linux/MacOS: Travis Build Status Windows: AppVeyor Build status

Introduction

After spending most of our "research" time not on AI innovation but on dealing with numerous and ever changing AI engines, their API, and the whole software and hardware stack, we decided to take an alternative approach.

logo

We are developing CK-TensorFlow which is an open-source suite of convenient wrappers and workflows powered by Collective Knowledge with unified JSON API for simple and customized installation/recompilation, usage, evaluation and multi-objective optimisation of various TensorFlow implementations (CPU,CUDA,OpenCL) across diverse platforms from mobile devices and IoT to supercomputers and TPU cloud.

See cKnowledge.org/ai, reproducible and CK-powered AI/SW/HW co-design competitions at ACM/IEEE conferences, shared optimization statistics, reusable AI artifact in the CK format and online demo of CK AI API with self-optimizing DNN for more details.

We need to have easily customizable TensorFlow builds via JSON API to be able to plug it to our framework for collaborative benchmarking and optimization of realistic workloads and models (such as deep learning) across diverse inputs and hardware provided by volunteers (see cKnowledge.org/ai, live repo and papers 1, 2, 3 for more details).

We can now build TensorFlow library and run classification via CK for various Android and Linux platforms. You can even participate in collaborative evaluation and optimization of TF using your Android device (mobile phone, tablet, etc) via this engaging Android app. You can see and use all public results in the Collective Knowledge repository.

Coordination of development

License

  • BSD (3 clause)

Global prerequisites

Prerequisites for Ubuntu

Python

  • Python 2.x:
$ sudo apt-get install python-dev python-pip python-setuptools python-opencv git
  • Python 3.x:
$ sudo apt-get install python3-dev python3-pip python3-setuptools

Note that CK will automatically install the following dependencies into CK TF virtual space: protobuf easydict joblib image wheel numpy scipy absl-py

Customized build

You can find more details about customized TensorFlow builds via CK for Android, Linux, Windows, Raspberry Pi, odroid, etc here.

Installing minimal CK

The minimal installation requires:

  • Python 2.7 or 3.3+ (limitation is mainly due to unitests)
  • Git command line client.

Linux/MacOS

You can install CK in your local user space as follows:

$ git clone http://github.com/ctuning/ck
$ export PATH=$PWD/ck/bin:$PATH
$ export PYTHONPATH=$PWD/ck:$PYTHONPATH

You can also install CK via PIP with sudo to avoid setting up environment variables yourself:

$ sudo pip install ck

Windows

First you need to download and install a few dependencies from the following sites:

You can then install CK as follows:

 $ pip install ck

or

 $ git clone https://github.com/ctuning/ck.git ck-master
 $ set PATH={CURRENT PATH}\ck-master\bin;%PATH%
 $ set PYTHONPATH={CURRENT PATH}\ck-master;%PYTHONPATH%

Installing CK workflow for TensorFlow

You can this CK workflow as follows:

$ ck pull repo:ck-tensorflow

Example of unified TensorFlow installation on Ubuntu or Windows via CK (pre-build versions)

$ ck install package:lib-tensorflow-1.8.0-cpu
 and/or (CK enables easy co-existance of different versions of tools
$ ck install package:lib-tensorflow-1.8.0-cuda

Check that TF is installed locally and registered in the CK:

$ ck show env --tags=lib,tensorflow

Use CK virtual environment to test it (similar to Python virtual env but for any binary package installed via CK):

$ ck virtual env --tags=lib,tensorflow

Install other TF versions available in the CK:

$ ck install package --tags=lib,tensorflow

Test unified image classification workflow via CK using above TF

$ ck run program:tensorflow --cmd_key=classify

Note, that you will be asked to select a jpeg image from available CK data sets. We added standard demo images (cat.jpg, catgrey.jpg, fish-bike.jpg, computer_mouse.jpg) to the 'ctuning-datasets-min' repository. You can list them via

 $ ck pull repo:ctuning-datasets-min
 $ ck search dataset --tags=dnn

Benchmarking

 $ ck run program:tensorflow (--env.BATCH_SIZE=10) (--env.NUM_BATCHES=5)

Select one of the test_cpu and test_cuda commands; select an available version of TensorFlow, if prompted (more than one choice); select an available benchmark, if prompted (more than one choice), and select TensorFlow model.

Crowd-benchmarking

It is now possible to participate in crowd-benchmarking of TensorFlow (early prototype):

$ ck crowdbench tensorflow --user={your email or ID to acknowledge contributions} (--env.BATCH_SIZE=128 --env.NUM_BATCHES=100)

You can see continuously aggregated results in the public Collective Knowledge repository under 'crowd-benchmark TensorFlow library' scenario.

Note, that this is an on-going, heavily evolving and long-term project to enable collaborative and systematic benchmarking and tuning of realistic workloads across diverse hardware (ARM TechCon'16 talk and demo, DATE'16, CPC'15). We also plan to add crowd-benchmarking and crowd-tuning of Caffe, TensorFlow and other DNN frameworks to our Android application soon - please, stay tuned!

Unified, multi-dimensional and multi-objective autotuning

It is now possible to take advantage of our universal multi-objective CK autotuner to optimize TensorFlow. As a first simple example, we added batch size tuning via CK. You can invoke it as follows:

$ ck autotune tensorflow

All results will be recorded in the local CK repository and you will be given command lines to plot graphs or replay experiments such as:

$ ck plot graph:{experiment UID}
$ ck replay experiment:{experiment UID} --point={specific optimization point}

Collaborative and unified optimization of DNN

We are now working to extend above autotuner and crowdsource optimization of the whole SW/HW/model/data set stack (paper 1, paper 2).

We would like to thank the community for their interest and feedback about this collaborative AI optimization approach powered by CK at ARM TechCon'16 and the Embedded Vision Summit'17 - so please stay tuned ;) !

logo

Other DNN with unified CK API

CK allows us to unify AI interfaces while collaboratively optimizing underneath engines. For example, we added similar support to install, use and evaluate Caffe/Caffe2, CK-PyTorch and MXNet via CK:

$ ck pull repo:ck-caffe2
$ ck pull repo --url=https://github.com/dividiti/ck-caffe
$ ck pull repo:ck-mxnet

$ ck install package:lib-caffe-bvlc-master-cpu-universal --env.CAFFE_BUILD_PYTHON=ON
$ ck install package:lib-caffe2-master-eigen-cpu-universal --env.CAFFE_BUILD_PYTHON=ON
$ ck install package --tags=mxnet

$ ck run program:caffe --cmd_key=classify
$ ck run program:caffe2 --cmd_key=classify
$ ck run program:mxnet --cmd_key=classify

$ ck crowdbench caffe --env.BATCH_SIZE=5
$ ck crowdbench caffe2 --env.BATCH_SIZE=5 --user=i_want_to_ack_my_contribution

$ ck autotune caffe
$ ck autotune caffe2

Realistic/representative training sets

We provided an option in all our AI crowd-tuning tools to let the community report and share mispredictions (images, correct label and wrong misprediction) to gradually and collaboratively build realistic data/training sets:

Online demo of a unified CK-AI API

  • Simple demo to classify images with continuous optimization of DNN engines underneath, sharing of mispredictions and creation of a community training set; and to predict compiler optimizations based on program features.

Long term vision

CK-Caffe, CK-Caffe2, CK-Tensorflow are part of an ambitious long-term and community-driven project to enable collaborative and systematic optimization of realistic workloads across diverse hardware in terms of performance, energy usage, accuracy, reliability, hardware price and other costs (ARM TechCon'16 talk and demo, DATE'16, CPC'15).

We are working with the community to unify and crowdsource performance analysis and tuning of various DNN frameworks (or any realistic workload) using Collective Knowledge Technology:

We continue gradually exposing various design and optimization choices including full parameterization of existing models.

Open R&D challenges

We use crowd-benchmarking and crowd-tuning of such realistic workloads across diverse hardware for open academic and industrial R&D challenges - join this community effort!

Related Publications with long term vision

@inproceedings{ck-date16,
    title = {{Collective Knowledge}: towards {R\&D} sustainability},
    author = {Fursin, Grigori and Lokhmotov, Anton and Plowman, Ed},
    booktitle = {Proceedings of the Conference on Design, Automation and Test in Europe (DATE'16)},
    year = {2016},
    month = {March},
    url = {https://www.researchgate.net/publication/304010295_Collective_Knowledge_Towards_RD_Sustainability}
}

@inproceedings{cm:29db2248aba45e59:cd11e3a188574d80,
    url = {http://arxiv.org/abs/1506.06256},
    title = {{Collective Mind, Part II: Towards Performance- and Cost-Aware Software Engineering as a Natural Science.}},
    author = {Fursin, Grigori and Memon, Abdul and Guillon, Christophe and Lokhmotov, Anton},
    booktitle = {{18th International Workshop on Compilers for Parallel Computing (CPC'15)}},
    publisher = {ArXiv},
    year = {2015},
    month = January,
    pdf = {http://arxiv.org/pdf/1506.06256v1}
}

Troublesooting

  • SqueezeDet demo currently work well with Python 3.5 and package:squeezedetmodel-squeezedet, so install it first:
$ ck install package:squeezedetmodel-squeezedet
$ ck run program:squeezedet

Testimonials and awards

Feedback

Get in touch with CK-AI developers here. Also feel free to engage with our community via this mailing list: