Specifying and executing multiverse analysis

A Python library to determine what exactly the user ran at the command line, along with default values, and any transformations that happened after parsing arguments.

A Python package enabling the development of robust automation scripts that are subdivided into stages.

Implementation of metaheuristic optimization methods in Python for scientific, industrial, and educational scenarios. Experiments can be executed in parallel or in a distributed fashion. Experimental results can be evaluated in various ways, including diagrams, tables, and export to Excel.

Keep track of your results

Resources on the Artifact Evaluation (AE) Process

TriScale software

A Python Interface to Reproducibility Measures of System-Oriented IR Experiments

zcurve R package for assessing the reliability and trustworthiness of published literature with the z-curve method

A framework-agnostic deep learning package and experiment manager

Special Issue: The Potential of Notebooks for Scientific Publication: Reproducibility, and Dissemination

Position Paper for the J-STARS Special Issue on Reproducible Research in Remote Sensing

Code repository accompanying "Assessing data availability and research reproducibility in hydrology and water resources"

Git copy of our Overleaf project for the preprint of Nine Best Practices for Research Software Registries and Repositories: A Concise Guide

[ECIR 2022] Automation of Citation Screening for Systematic Literature Reviews Using Neural Networks: A Replicability Study

A curated list of publicly available datasets for replication studies.

Materials for the MIT PML workshop on Good coding practices for social scientists.

Wide Residual Networks in Keras and PyTorch

Repository for R package and website of replication diagnosis tools

A Python package that allows you to interact with the shell, while logging various metadata, statistics, and trace information.