Enabling open science

Grigori Fursin edited this page Aug 30, 2018 · 38 revisions

[ Home ]


We are developing an open-source Collective Knowledge framework with a live repository to enable collaborative and reproducible research and experimentation with an open publication model. CK allows researchers to share all artifacts as customizable Python components with unified JSON API, and then reuse them to assemble various collaborative and reproducible experimental workflows. Since 2008, we regularly publish various research development challenges in computer engineering which have been (or not yet) solved with the help of the academic and industrial community. Note that our idea is not just to solve some problems, but also to share all artifacts as reusable components with a unified API to let the community easily build upon them while contributing to Collective Knowledge thus enabling truly open science! We also collect links to various related initiatives, repositories, tools, articles and events - feel free to help us update it or discuss this community-driven initiative via CK mailing list and our LinkedIn group on reproducibility!

Table of Contents



Since 2006 we have been trying to solve problems with reproducibility of experimental results in computer engineering as a side effect of our MILEPOST , cTuning.org, Collective Mind and Collective Knowledge projects to speed up optimization, benchmarking and co-design of computer systems and neural networks using multi-objective autotuning, big data, predictive analytics and experiment crowdsourcing (see our experience report and the cTuning foundation history).

We now focus on the following technological and social aspects to enable collaborative, systematic and reproducible research and experimentation particularly related to benchmarking, optimization and co-design of faster, smaller, cheaper, more power efficient and reliable software and hardware:

  • developing collaborative research and experimentation infrastructure that can share artifacts as reusable components together with the whole experimental setups (see P1, P2;
  • developing public and open source repositories of knowledge (see our live repository and our vision papers P1, P2);
  • evangelizing and enabling new open publication model for online workshops, conferences and journals (see our proposal arXiv / ACM DL);
  • setting up and improving procedure for sharing and evaluating experimental results and all related material for workshops, conferences and journals (see our proposal arXiv / ACM DL);
  • improving sharing, description of dependencies, and statistical reproducibility of experimental results and related material;
  • supporting and improving Artifact Evaluation for major workshops and conferences including PPoPP, CGO and PACT.



Our events

Community-driven research

Together with the community, non-profit cTuning foundation and dividiti we are working on the following topics to enable open research:

  • developing tools and methodology to capture, preserve, formalize, systematize, exchange and improve knowledge and experimental results including negative ones
  • describing and cataloging whole experimental setups with all related material including algorithms, benchmarks, codelets, datasets, tools, models and any other artifact
  • developing specification to preserve experiments including all software and hardware dependencies
  • dealing with variability and rising amount of experimental data using statistical analysis, data mining, predictive modeling and other techniques
  • developing new predictive analytics techniques to explore large design and optimization spaces
  • validating and verifying experimental results by the community
  • developing common research interfaces for existing or new tools
  • developing common experimental frameworks and repositories (enable automation, re-execution and sharing of experiments)
  • sharing rare hardware and computational resources for experimental validation
  • implementing previously published experimental scenarios (auto-tuning, run-time adaptation) using common infrastructure
  • implementing open access to publications and data (particularly discussing intellectual property IP and legal issues)
  • speeding up analysis of "big" experimental data
  • developing new (interactive) visualization techniques for "big" experimental data
  • enabling interactive articles


Follow us



We would like to thank our colleagues from the cTuning foundation, dividiti, artifact-eval.org, OCCAM project for their help, feedback, participation and support.

Questions and comments

You are welcome to get in touch with the CK community if you have questions or comments!

Clone this wiki locally
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.