The purpose of this page is to provide a resource collection about the Artifact Evaluation (AE) process that has been introduced for many publishing venues in Computer Science (CS) in order to promote and help researchers understand (and improve) the AE process.
- https://www.artifact-eval.org/: General information on the AE process, origins, and packaging guidelines.
- https://sysartifacts.github.io/: Information on the AE processes in system conferences, including calls, committees, and results.
- Artefact Review and Badging: Improving Confidence in our Experimental Results by Michel Steuwer
- The Real Software Crisis: Repeatability as a Core Value by Shriram Krishnamurthi and Jan Vitek
- Proof Artifacts. Guidelines for Submission and Reviewing by Marianna Rapoport: Instructions on creating and reviewing proof artifacts.
- Checking machine-checked proofs by Assia Mahboubi: Instructions on how to review machine-checked proofs as well as suggestions for authors and organizers.
- Zenodo
- Nextjournal
- Dagstuhl Artifacts Series (DARTS)
- figshare
- Dryad
- Software Heritage Archive
- ACM Digital Library (ACM DL): A collection of artifacts available in the ACM DL.
- emulab: A network testbed to develop, debug, and evaluate systems maintained by the Flux Research Group at the University of Utah.
- Making Your Code Citable: A GitHub Guide with instructions on how to archive GitHub repositories on Zenodo.
- How to license research artifacts? by Gustavo Pinto: A general explanation of the purpose of research artifacts and copyright as well as an overview of open-source licenses for various artifact types.
- Choose an open source license: A "keep it simple" guide on choosing open-source licenses.
- Artifact Review and Badging by ACM: A description of the badges and the requirements for awarding them for ACM conferences.
- SIGSOFT Artifact Evaluation Working Group by Jon Bell: A document describing the AE effort in the software engineering field.
- The ROSE Festival, ESEC/FSE 2018: The ROSE festival is an effort to promote replication and reproducibility in the software engineering field.
- Artifact Evaluation: Tips for Authors by Rohan Padhye: 10 actionable tips for preparing a successful artifact.
- HOWTO for AEC Submitters by Dan Barowy, Charlie Curtsinger, Emma Tosch, and John Vilk: Author recommendations for preparing artifacts as well as example artifacts.
- Creating successful artifacts by Hernán Ponce de León: Tips from an AEC member of POPL, PLDI, and OOPSLA.
- how to disclose data for double-blind review and make it archived open data upon acceptance by Daniel Graziotin: Sharing an artifact before paper acceptance in a double-blind way using figshare and Zenodo.
- Generating an Artifact From a Benchmarking Setup as Part of CI by Stefan Marr: Automatically generating VirtualBox images as part of the Continuous Integration process.
- My first artifact: An author's perspective on the POPL'19 Artifact Evaluation process by Simon Fowler: A comprehensive report on an artifact submitted to the POPL AE including an overview of the artifact and its structure, author expectations, the actual reviews as well as reflections on the packaging and the process.
- "Taming the Parallel Effect Zoo" and the PLDI artifact evaluation process by Lindsey Kuper: A report from the author of a PLDI paper including a description of the process, expectations, and artifact packaging.
- Provisioning My First Artifact. by Jan de Muijnck-Hughes: An experience report by an author of an ECOOP 2019 artifact who packaged his artifact as a VirtualBox image.
- How Are Award-winning Systems Research Artifacts Prepared: An interview of Manuel Rigger by Tianyin Xu for the ACM SIGOPS blog on the topic of creating successful artifacts.
- Experience with Artifact Evaluation for CAV 2015 by Marko Dimjašević: Suggestions on improving the AE process as well as research artifacts based on an experience with the CAV AE process.
- Why I will never join an Artifacts Evaluation Committee Again by Moritz Beller: Description of a (less ideal) experience as a reviewer serving for the ESEC/FSE 2020 AE, including constructive criticism that is likely to serve future AE chairs.
- Making Sure Artifact Reviewing Stays Anonymous by Marko Dimjašević: Thoughts and recommendations on maintaining reviewer (and author) anonymity.
- My Experience of Artifact Evaluation in FPGA2020 by Jianyi Cheng: An experience report from the perspective of an author of an artifact as well as an AEC member of FPGA2020, including useful tips for future authors of artifacts.
- Reflections on the VMCAI 2021 artifact evaluation by Troels Henriksen: A short experience report from the perspective of a co-chair of the VMCAI 2021 AE, including suggestions for future editions.
- OOPSLA 2020 Artifacts Chairs' Report by Colin Gordon and Anders Møller: A report on the OOPSLA 2020 AE featuring recommendations for future artifact evaluations.
- Artifact Evaluation for Software Conferences by Shriram Krishnamurthi: A report on the first AE process for a major software engineering conference from the perspective of a co-chair.
- Interview with Eric Bodden and Alessandra Gorla, ISSTA 2016 Artifact Evaluation Chairs by Eric Bodden and Alessandra Gorla: An interview of the AE co-chairs of ISSTA 2016.
- How to survive an artifact evaluation with HotCRP by Camil Demetrescu: Instructions for using HotCRP in the AE process from 2015.
- ACM PPoPP’19 artifact evaluation report and HotCRP configuration by Grigori Fursin and Flavio Vella: The detailed settings of the HotCRP instance configured for the PPoPP 2019 AE and a short report on the process.
- "Artifact Evaluation Artifact" for OOPSLA'13 by Steve Blackburn and Matthias Hauswirth: A detailed description of the first OOPSLA AE process including the timeline, forms, and emails.
- Community expectations for research artifacts and evaluation processes: A study of AE calls and an analysis of a reviewer survey to study the community expectations in PL/SE conferences
- Getting Research Software to Work: A Case Study on Artifact Evaluation for OOPSLA 2019 by Erin Dahlgren
- Evaluating the artifacts of SIGCOMM papers by Damien Saucez, Luigi Iannone, and Olivier Bonaventure
- Publish or perish, but do not forget your software artifacts by Robert Heumüller, Sebastian Nielebock, Jacob Krüger, and Frank Ortmeier
- Artifact Evaluation: Is It a Real Incentive? by Bruce R. Childers and Panos K. Chrysanthis
- Artifact Evaluation for Publications (Dagstuhl Perspectives Workshop 15452): A report on a Dagstuhl Perspectives Workshop about the AE process.
- cTuning: many useful materials about reproducible science and AE.
- Software Engineering Reproducibility Manifesto (SERM) by Marcel Böhme
- Artifact Evaluation for Reproducible Quantitative Research
- SIGPLAN Empirical Evaluation Checklist
https://github.com/csartifacts/resources by Manuel Rigger is marked with CC0 1.0
