Creating Problems

Now that you've got JudgeLite up and running (if you haven't yet, see the Setup Instructions), it's time to create some problems to submit to! Let's take a look at the format used to store the problems in JudgeLite.

If you just want to get your hands dirty, and figure stuff out as you go, there is a template_problem_info directory on Github. This directory contains everything you need to start making problems in JudgeLite. It also includes a base problem that you can easily copy to create a new problem, along with a convenient upload script to send your problems to JudgeLite! Feel free to start making problems using that directory (take a look at this guide if there's something you would like more info about).

General Structure

All the problems you create will be stored in the problem_info directory (automatically created when you run JudgeLite for the first time). JudgeLite stores problems in a simple directory structure. Here is a sample structure:

.
├── problems.yml            # The file containing info about all the other problems
├── problem1                # A folder containing info on the problem with ID "problem1"
├── problem2                # A folder containing info on the problem with ID "problem2"
└── ...

So, the problem_info directory contains a problems.yml file (described below), along with a bunch of problems (each problem is contained in one folder).

problems.yml

Every problem is contained in its own folder. There is also a problems.yml file, which contains info about all the problems (in YAML format). JudgeLite will read this file in order to determine which problems to load. Here is a sample problems.yml file:

---
groups:
  - id: test_group
    name: Test Group
    status: up
    problems:
      - id: test1
        name: Test Problem #1
        blurb: This is the 1st test problem.
        difficulty: Beginner
        status: up
...

YAML files should start with 3 dashes (-), and end with 3 periods (.), although this isn't strictly required.

The problems.yml file should have a list of groups. Each group will have the following fields:

• id - The ID of the group.
• name - The name of the group.
• status - The status of the group. Should be one of "up" or "down".
• problems - A list of the problems in the group.

Each problem will have the following fields:

• id - The ID of the problem. This should match the name of the directory that contains the problem.
• name - The name of the problem.
• status - The status of the problem. Should be one of "up" or "down".
• blurb (Optional) - A brief description of the problem (or just some flavor text).
• difficulty (Optional) - The difficulty of the problem. You can decide how you'd like to use this, but we use the difficulties "Beginner", "Intermediate", and "Advanced".

Problem Structure

Every problem in JudgeLite is self-contained in a folder. Here is a sample problem structure:

.
├── info.yml                # File containing the info for the problem
├── statement.md            # (Optional) The problem statement
├── bonus.md                # (Optional) The bonus for this problem
├── hints.md                # (Optional) The hints for this problem
├── subtasks                # Directory containing the subtasks for this problem
│   ├── 01_main             # A subtask with name "01_main"
|   │   ├── 01.in           # The input file for test "01"
|   │   ├── 01.out          # The output file for test "01"
|   │   └── ...
|   └── ...
└── ...

The info.yml file (described below) contains important info about the problem, like the time and memory limits.

The statement.md, bonus.md, and hints.md files are optional (you could choose to only use JudgeLite for judging purposes, and handle the problem statements somewhere else). If included, they will be loaded by JudgeLite / returned as part of the info for this problem (see API Reference). These files can be written in Markdown.

The subtask folder contains all the subtasks for a problem (described in more detail later).

info.yml

The info.yml file contains useful information about a problem. It is loaded by JudgeLite to determine things like the time and memory limits, along with which subtasks to run. Here is a sample info.yml file:

---
problem_id: test1
problem_name: Test Problem #1
difficulty: Beginner
time_limit: 2
memory_limit: 256
scoring_method: average_stop
checker: diff
max_score: 100

subtasks:
  - name: 01_main
    score: 100
    num_samples: 1
...

An info.yml file has the following fields:

• problem_id - The ID of the problem. This should match the name of the directory that contains this problem.
• problem_name - The name of the problem.
• difficulty (Optional) - The difficulty of the problem. You can decide how you'd like to use this, but we use the difficulties "Beginner", "Intermediate", and "Advanced".
• time_limit - The time limit of the problem (in seconds).
• memory_limit - The memory limit of the problem (in megabytes).
• scoring_method - The way that submissions will be scored. This can be one of the following options:
  • minimum - The minimum score out of all test cases in a subtask is used (all or nothing). If one test case fails in a subtask, the rest are immediately skipped (there would be no reason to continue anyways).
  • average_stop - The score for each test case is averaged out to determine the points for each subtask (partial credit). The judge will stop on the first failed test in a subtask (the rest are skipped / treated with a score of 0). This keeps the judge from getting blocked up with submissions that TLE.
  • average - The score for each test case is averaged out to determine the points for each subtask (partial credit). The judge will not stop early when evaluating a subtask. Be warned: This could cause the judge to get backed up if there are too many test cases!

(This wiki page will be filled in later. For now, take a look at the sample problem in the sample_problem_info directory: It has a README.txt file explaining how to create problems, along with a self-documented info.yml file. It's recommended to use the template_problem_info as a starting point when creating problems.)