User_Building a Scoring Program for a Competition

Eric Carmichael edited this page Jul 25, 2014 · 7 revisions

Building a Scoring Program

In the current CodaLab competition model a competition is constructed with a set of reference data and a scoring program. When a competition is run, the reference data is compared with each participant's submission, and the differences in the two are scored using the scoring program.

Reference and Submission Bundles

The bundle contains the Scoring Program that compares the participants submission with the reference data (in the bundle) to score the submission. In this example the reference data contains the value of pi. The bundle computes the absolute difference of the submitted value from the reference value.

Here are the contents of the file: 
  |- answer.txt (Contains: 3.14159265359)
  |- metadata   (Contains: This is the authoritative result.)

A for this competition would look similar: 
  |- answer.txt (Contains: 3.1415359)

Here are the contents of the file:
  |- (The actual evaluation code to run.)
  |- metadata     (Syntax and information needed to run.)
  |- readme.txt (Notes about the evaluation program.)
  |- (Enables py2exe to build a windows executable of the script.)
  |- Supporting modules and libraries (if required).

The metadata file contains:

command: python $program/evaluate.exe $input $output
description: Example competition evaluation program.

Directory Structure for Submissions

  1. There is a fixed directory structure that the scoring program operates within. It looks like this:
Submission Directory
  |- input
    |- ref (This is the reference data unzipped)
    |- res (This is the user submission unzipped)
  |- program (This is the scoring program [and any included dependencies] unzipped)
  |- output (This is where the scores.txt file is written by the scoring program)
  1. The scoring program will be invoked as <program> <input directory> <output directory>.
  2. The scoring program is executed so that stderr and stdout are captured
  3. The scoring program will generate a scores.txt file (it has to be named scores.txt) that contains key value pairs of metrics. This is a score file for the example competition: Difference: 0.0057
  4. Each key in the scores.txt file is identical to a leaderboard column key in the competition.yaml. (e.g. "DIFFERENCE" in the example competition.yaml shown in Building a Competition Bundle). This is how scores are related to the competition for reporting so it is critical the leaderboard keys and the keys in the scores.txt are identical.

Metadata passed to scoring program

Metadata is passed to the scoring program input directory as metadata

Example metadata

ref: competition/2/2/data/competition/2/2/data/
res: competition/2/submission/1/
history: competition/2/8/submissions/1/2/history.txt
submitted-by: codalab
submitted-at: 2014-07-18T00:09:55+00:00
competition-submission: 2
competition-phase: 2
automatic-submission: False


The history.txt path


The username of the participant who submitted this


Time the submission was submitted


The submission number


The phase number


This will only be passed if phase.auto_migration is enabled. If this is an automatic submission (from phase-to-phase migrations) then it will be marked as true.