Farms of CK machines

Grigori Fursin edited this page Oct 1, 2016 · 4 revisions
Clone this wiki locally

[ Home ]

Table of Contents

Intro

Researchers often want to validate their ideas and experiments (benchmarking, autotuning, new predictive modeling algorithm, etc) across different machines with different environment (OS, libraries, compilers, etc). However, it is often a very non-trivial task that can easily eat all available research time.

To solve this problem, we provided a module machine in the CK env repository to add, describe and control the status of any available machine as a target for experiments.

It can be any local Linux, MacOS or Windows host, Linux/MacOS machines accessed via SSH, an Android device connected via ADB or ARM's Workload Automation framework, and a Windows based device connected via CK crowdnode.

Adding new machine

First, you need to make sure that you have ck-env repository installed:

 $ ck pull repo:ck-env

You may also want to update all other repositories (we suggest to do it regularly):

 $ ck pull all --kernel

Make sure that your target machine is connected or online (unless it's your host machine).

Just for example, let's consider that you want to use an Android device connected to your host as a target for your experiments.

Now you can describe this new machine as following (let's name it my-android-machine):

 $ ck add machine:my-android-machine

You will be asked various questions about the type of this machine. In our case, you need to select 1) CK: Android machine accessed via adb. CK will attempt to automatically detect various properties of your connected device including Android version, device ID, ABI, required SDK, etc, and will eventually add a CK entry machine:my-android-machine.

Note, if you have another Android device, you can add it using the same procedure (just name your target differently).

Checking status of a farm

You can now check the status of all target machines registered in the CK from the command line simply via

 $ ck show machine

You can also view their status in your browser:

 $ ck browse machine

You can check the status of a give machine via

 $ ck check machine:my-android-machine

or in JSON (to use it during automation of experiments):

 $ ck check machine:my-android-machine --out=json
 $ ck check machine --help

Using CK widget to detect machine parameters

It is now possible to use CK widgets platform* to detect various parameters of machines in a farm by specifying --target= flag:

 $ ck detect platform.os --target=my-android-machine
 $ ck detect platform.cpu --target=my-android-machine
 $ ck detect platform.gpu --target=my-android-machine
 $ ck detect platform --target=my-android-machine

If you want to use these widgets in your own modules, you can obtain their JSON API using --help flag:

 $ ck detect platform --help

Compiling and running workloads on target machines

You can now easily compile and run any workload shared in CK format (module program) using any target machine from the farm.

Let's consider application for image corner detection (often used in automotive industry):

 $ ck list program
 $ ck list program:*susan

If you want to compile it for Android device, do not forget to download and install Android NDK and Android SDK (CK will then automatically detect it on your Linux/MacOS or Windows machine):

Now you can compile this benchmark simply as following:

 $ ck compile program:cbench-automotive-susan --speed --target=my-android-machine

In case of successful compilation, you can now run it on your device as following

 $ ck run program:cbench-automotive-susan --target=my-android-machine

Note, that you can compile and run program in one go via CK universal program pipeline (the one which is used in autotuning, crowd-tuning and crowd-benchmarking):

 $ ck pipeline program:cbench-automotive-susan --speed --target=my-android-machine

Autotuning and crowdtuning on target machines

Above organization allows us to reuse CK-based universal, multi-dimensional and multi-objective autotuning plugins (1) for any machine in your farm as following:

 $ ck autotune program:cbench-automotive-susan --target=my-android-machine

If you want to share your optimization knowledge via public repository (as described in 1 and 2), you can participate in crowdtuning via

 $ ck crowdtune program:cbench-automotive-susan --target=my-android-machine

Questions and feedback

If you have questions or feedback, do not hesitate to get in touch with the community via CK mailing list.