The purpose of this page is to provide a resource collection about the Artifact Evaluation (AE) process that has been introduced for many publishing venues in Computer Science (CS) in order to promote and help researchers understand (and improve) the AE process.

General Resources

Process

https://www.artifact-eval.org/: General information on the AE process, origins, and packaging guidelines.
https://sysartifacts.github.io/: Information on the AE processes in system conferences, including calls, committees, and results.

Advocacy

Artifact Types

Proof Artifacts. Guidelines for Submission and Reviewing by Marianna Rapoport: Instructions on creating and reviewing proof artifacts.
Checking machine-checked proofs by Assia Mahboubi: Instructions on how to review machine-checked proofs as well as suggestions for authors and organizers.

Hosting Platforms

Zenodo
Nextjournal
Dagstuhl Artifacts Series (DARTS)
figshare
Dryad
Software Heritage Archive
ACM Digital Library (ACM DL): A collection of artifacts available in the ACM DL.

Packaging Platforms

Experimentation and Evaluation

emulab: A network testbed to develop, debug, and evaluate systems maintained by the Flux Research Group at the University of Utah.

Packaging Instructions

Making Your Code Citable: A GitHub Guide with instructions on how to archive GitHub repositories on Zenodo.

Licensing

How to license research artifacts? by Gustavo Pinto: A general explanation of the purpose of research artifacts and copyright as well as an overview of open-source licenses for various artifact types.
Choose an open source license: A "keep it simple" guide on choosing open-source licenses.

Badging

Artifact Review and Badging by ACM: A description of the badges and the requirements for awarding them for ACM conferences.

Efforts

SIGSOFT Artifact Evaluation Working Group by Jon Bell: A document describing the AE effort in the software engineering field.
The ROSE Festival, ESEC/FSE 2018: The ROSE festival is an effort to promote replication and reproducibility in the software engineering field.

Artifact Preparation

Author Experience Reports

My first artifact: An author's perspective on the POPL'19 Artifact Evaluation process by Simon Fowler: A comprehensive report on an artifact submitted to the POPL AE including an overview of the artifact and its structure, author expectations, the actual reviews as well as reflections on the packaging and the process.
"Taming the Parallel Effect Zoo" and the PLDI artifact evaluation process by Lindsey Kuper: A report from the author of a PLDI paper including a description of the process, expectations, and artifact packaging.
Provisioning My First Artifact. by Jan de Muijnck-Hughes: An experience report by an author of an ECOOP 2019 artifact who packaged his artifact as a VirtualBox image.
How Are Award-winning Systems Research Artifacts Prepared: An interview of Manuel Rigger by Tianyin Xu for the ACM SIGOPS blog on the topic of creating successful artifacts.

Artifact Reviewing

Reviewer Experience Reports

Experience with Artifact Evaluation for CAV 2015 by Marko Dimjašević: Suggestions on improving the AE process as well as research artifacts based on an experience with the CAV AE process.
Why I will never join an Artifacts Evaluation Committee Again by Moritz Beller: Description of a (less ideal) experience as a reviewer serving for the ESEC/FSE 2020 AE, including constructive criticism that is likely to serve future AE chairs.
Making Sure Artifact Reviewing Stays Anonymous by Marko Dimjašević: Thoughts and recommendations on maintaining reviewer (and author) anonymity.
My Experience of Artifact Evaluation in FPGA2020 by Jianyi Cheng: An experience report from the perspective of an author of an artifact as well as an AEC member of FPGA2020, including useful tips for future authors of artifacts.

AE Chairing

Chair Reports

Reflections on the VMCAI 2021 artifact evaluation by Troels Henriksen: A short experience report from the perspective of a co-chair of the VMCAI 2021 AE, including suggestions for future editions.
OOPSLA 2020 Artifacts Chairs' Report by Colin Gordon and Anders Møller: A report on the OOPSLA 2020 AE featuring recommendations for future artifact evaluations.
Artifact Evaluation for Software Conferences by Shriram Krishnamurthi: A report on the first AE process for a major software engineering conference from the perspective of a co-chair.
Interview with Eric Bodden and Alessandra Gorla, ISSTA 2016 Artifact Evaluation Chairs by Eric Bodden and Alessandra Gorla: An interview of the AE co-chairs of ISSTA 2016.

Detailed Resources for Chairing an AE

How to survive an artifact evaluation with HotCRP by Camil Demetrescu: Instructions for using HotCRP in the AE process from 2015.
ACM PPoPP’19 artifact evaluation report and HotCRP configuration by Grigori Fursin and Flavio Vella: The detailed settings of the HotCRP instance configured for the PPoPP 2019 AE and a short report on the process.
"Artifact Evaluation Artifact" for OOPSLA'13 by Steve Blackburn and Matthias Hauswirth: A detailed description of the first OOPSLA AE process including the timeline, forms, and emails.

Systematic Studies and Papers

Community expectations for research artifacts and evaluation processes: A study of AE calls and an analysis of a reviewer survey to study the community expectations in PL/SE conferences
Getting Research Software to Work: A Case Study on Artifact Evaluation for OOPSLA 2019 by Erin Dahlgren
Evaluating the artifacts of SIGCOMM papers by Damien Saucez, Luigi Iannone, and Olivier Bonaventure
Publish or perish, but do not forget your software artifacts by Robert Heumüller, Sebastian Nielebock, Jacob Krüger, and Frank Ortmeier
Artifact Evaluation: Is It a Real Incentive? by Bruce R. Childers and Panos K. Chrysanthis
Artifact Evaluation for Publications (Dagstuhl Perspectives Workshop 15452): A report on a Dagstuhl Perspectives Workshop about the AE process.

Broader Reproducibility and Artifact Resources

cTuning: many useful materials about reproducible science and AE.
Software Engineering Reproducibility Manifesto (SERM) by Marcel Böhme
Artifact Evaluation for Reproducible Quantitative Research
SIGPLAN Empirical Evaluation Checklist

CC0 1.0 License

https://github.com/csartifacts/resources by Manuel Rigger is marked with CC0 1.0

csartifacts/resources

ErrorLooks like something went wrong!

About

Topics

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Error
Looks like something went wrong!

Packages