Long Running Replications

In general, if there seems to be long-running code, either from the last time we saw a paper, or because there is a note in the README (specifying the time requirement),

• leave it running on CISER for a few days. You can work on other issues in the meantime (but be cognizant of CISER downtimes)

  • Post a note to JIRA comment that code is still running

• If after a few days, the code has not finished, write up the report as an "incomplete replication", and submit the report

  • Make it in the report that there is still code running.
  • Leave the code running as long as possible!
  • Try to get all other code (if independent) to run as well.
  • One of the reasons for incomplete reproduction is then ‘insufficient time’ (one of the checkboxes on Jira).

• What to do with the running code?

  • often, this happens during the first submission of a paper. We will request that authors specify if this is a bug, or if there is a LONG time requirement.
  • When the paper comes back, we may have the result of the long-running program, so keep updating the repository for these issues with log files, output, etc. and make a note in the JIRA issue if the code finishes running.