Skip to content

Example_Hello_world

Jump to bottom
Benjamin Bearce edited this page Dec 18, 2019 · 1 revision

Hello World competition

This is a very simple example of competition with result submission (and how to turn it into a code submission challenge by adding an ingestion program).

The competition bundle contains the following files:

reference_data.zip	    The solution to the problem
scoring_program.zip	    The program evaluating the solution
logo.jpg                    The logo
competition.yaml            The YAML configuration file
data.html	            HTML documentation pages	
evaluation.html		
overview.html	
terms_and_conditions.html

The YAML file is the configuration file. It consist in a number of attribute: value pairs. For a full list of attributes, see the Codalab competition YAML definition language.

  1. First section: general settings and HTML pages
title: Example Hello World Competition
description: An example competition where submissions should output "Hello World!"
image: logo.jpg
has_registration: True
html:
    overview: overview.html
    evaluation: evaluation.html
    terms: terms_and_conditions.html
    data: data.html
  1. Section section: phases (only 1 phase in this example)
phases:
    1:
        phasenumber: 1
        label: "First phase"
        start_date: 2013-06-30
        max_submissions: 100
        scoring_program: scoring_program.zip
        reference_data: reference_data.zip
  1. Third section: leaderboard configuration
leaderboard:
    leaderboards:
        RESULTS: &RESULTS
            label: Results
            rank: 1
    columns:
        correct:
            leaderboard: *RESULTS
            label: correct
            rank: 1
            numeric_format: 1

In this case, the scoring program writes a file scores.txt containing:

correct:1

for the sample submission.

Building a scoring program

This example uses Python 2.7.

evaluate.py - is an example that checks that the submission data matches the truth data, which is "Hello World!"

metadata - this is a file that instructs Codalab to execute evaluate.py.

Once these pieces are assembled they are packaged as program.zip which CodaLab can then use to evaluate the submissions for a competition.

Building an ingestion program

To turn this result submission competition into a code submission competition, we give an example of Ingestion Program.

WARNING: Code submission may result in a lot of computational load on our servers. Before doing that contact us or learn how to set up your own compute workers.

  • Download the example of ingestion program
  • Upload ingestion_program.zip to My Competitions>My datasets>Create dataset (choose type: ingestion program) go to the editor and uncheck "Results Scoring Only", then select in the ingestion program menu the ingestion program you just uloaded.
  • submit the sample code: submission_4_ingestion.zip

For another example of code submission without ingestion program, see the Compute Pi competition.

Clone this wiki locally