Skip to content
/ codeflaws Public
forked from codeflaws/codeflaws

This repository the benchmark with 3902 defects extracted from C programs in Codeforces (http://codeforces.com/)

Notifications You must be signed in to change notification settings

chdd/codeflaws

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

30 Commits
 
 
 
 

Repository files navigation

Codeflaws

The Codeflaws benchmark is a collection of C programs with 4085 defects. Each defect are where the dataset is crawled from Codeforces

##Download: Codeflaws is available for download at tar-link

##Structure of each subject folder All the subject programs are in the benchmark directory. Each subject folder is named using the following convention: <contestid>-<problem>-bug-<buggy-submisionid>-<accepted-submissionid> Each folder contains:

  • Buggy submission with name <contestid>-<problem>-<buggy-submisionid>.c
  • Accepted submission with name <contestid>-<problem>-<accepted-submisionid>.c
  • Two sets of test scripts:
    1. Repair Test script (test suite given to repair tools for generating repair): test-genprog.sh is for search-based repair tools (GenProg, SPR, Prophet), test-angelix.sh is for Angelix as it requires inserting special instrumentation.
    2. Test script for patch validation (held-out test suite): test-valid.sh is for validating the correctness of patches
  • Test input files: input[0-9]+ file used by Test suite (i), and heldout-input[0-9]+ file used by Test suite (ii)
  • Test output files: output[0-9]+ file used by Test suite (i), and heldout-output[0-9]+ file used by Test suite (ii)
  • Makefile for compiling the buggy submission. This contains the CFLAGS options recommended by Codeforces. To compile the accepted submission, use the command make FILENAME=10-A-13543524
  • Makefile.genprog for compiling the buggy submission using cilly. This is for GenProg experiments as GenProg works on CIL representation.
  • Test configuration for SPR that specify the name for pass/fail test: <contestid>-<problem>-<buggy-submisionid>.c.revlog

##Script for running each repair tool All the files mentioned below are stored in the all-script directory

###Angelix Use the following files for running Angelix:

  • File for running Angelix: run-version-angelix.sh

###GenProg Use the following files for running GenProg:

  • File for GenProg general configuration: configuration-default
  • File for compilation configuration: compile.pl
  • File for running GenProg: run-version-genprog.sh
  • File for validating patches generated by GenProg (This script is called from run-version-genprog.sh): validate-fix-genprog.sh

###SPR Use the following files for running SPR:

  • File for compilation configuration: code-build.py
  • File for test configuration: run-test.py
  • File for running SPR: run-version-spr.sh
  • File for validating patches generated by SPR (This script is called from run-version-spr.sh): validate-fix-spr.sh

###Prophet Use the following files for running Prophet:

  • File for compilation configuration: code-build.py
  • File for test configuration: run-test.py
  • File for running Prophet: run-version-prophet.sh
  • File for validating patches generated by Prophet (This script is called from run-version-prophet.sh): validate-fix-prophet.sh
  • Parameter file with learned model (From Prophet original experiment): para-rext-all.out

##Citing Codeflaws

If you use Codeflaws in an academic work, we would be really glad if you cite our paper using the following bibtex:

@inproceedings{Tancodeflaws,
  author    = {Tan, Shin Hwei and Yi, Jooyong and Yulis and Mechtaev, Sergey and Roychoudhury,Abhik},
  title     = {Codeflaws: A Programming Competition Benchmark for Evaluating Automated Program Repair Tools},
  booktitle = {ICSE Poster},
  year      = {2017},
  note = {To appear}
}

##Other information For more information/questions about the benchmark, refer to the following website: https://codeflaws.github.io/

About

This repository the benchmark with 3902 defects extracted from C programs in Codeforces (http://codeforces.com/)

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Shell 90.6%
  • Python 5.8%
  • Perl 3.6%