-
Notifications
You must be signed in to change notification settings - Fork 0
Creating Problems
Now that you've got JudgeLite up and running (if you haven't yet, see the Setup Instructions), it's time to create some problems to submit to! Let's take a look at the format used to store the problems in JudgeLite.
If you just want to get your hands dirty, and figure stuff out as you go, there is a template_problem_info directory on Github. This directory contains everything you need to start making problems in JudgeLite. It also includes a base problem that you can easily copy to create a new problem, along with a convenient upload script to send your problems to JudgeLite! Feel free to start making problems using that directory (take a look at this guide if there's something you would like more info about).
There is also a sample_problem_info directory on Github. This directory contains example problems that demonstrate the features mentioned in this wiki page.
All the problems you create will be stored in the problem_info directory (automatically created when you run JudgeLite for the first time). JudgeLite stores problems in a simple directory structure. Here is a sample structure:
.
├── problems.yml # The file containing info about all the other problems
├── problem1 # A folder containing info on the problem with ID "problem1"
├── problem2 # A folder containing info on the problem with ID "problem2"
└── ...
So, the problem_info directory contains a problems.yml file (described below), along with a bunch of problems (each problem is contained in one folder).
Every problem is contained in its own folder. There is also a problems.yml file, which contains info about all the problems (in YAML format). JudgeLite will read this file in order to determine which problems to load. Here is a sample problems.yml file:
---
groups:
- id: test_group
name: Test Group
status: up
problems:
- id: test1
name: Test Problem #1
blurb: This is the 1st test problem.
difficulty: Beginner
status: up
...YAML files should start with 3 dashes (-), and end with 3 periods (.), although this isn't strictly required.
The problems.yml file should have a list of groups. Each group will have the following fields:
- id - The ID of the group.
- name - The name of the group.
- status - The status of the group. Should be one of "up" or "down".
- problems - A list of the problems in the group.
Each problem will have the following fields:
- id - The ID of the problem. This should match the name of the directory that contains the problem.
- name - The name of the problem.
- status - The status of the problem. Should be one of "up" or "down".
- blurb (Optional) - A brief description of the problem (or just some flavor text).
- difficulty (Optional) - The difficulty of the problem. You can decide how you'd like to use this, but we use the difficulties "Beginner", "Intermediate", and "Advanced".
Every problem in JudgeLite is self-contained in a folder. Here is a sample problem structure:
.
├── info.yml # File containing the info for the problem
├── statement.md # (Optional) The problem statement
├── bonus.md # (Optional) The bonus for this problem
├── hints.md # (Optional) The hints for this problem
├── subtasks # Directory containing the subtasks for this problem
│ ├── 01_main # A subtask with name "01_main"
| │ ├── 01.in # The input file for test "01"
| │ ├── 01.out # The output file for test "01"
| │ └── ...
| └── ...
└── ...
The info.yml file (described below) contains important info about the problem, like the time and memory limits.
The statement.md, bonus.md, and hints.md files are optional (you could choose to only use JudgeLite for judging purposes, and handle the problem statements somewhere else). If included, they will be loaded by JudgeLite / returned as part of the info for this problem (see API Reference). These files can be written in Markdown.
The subtask folder contains all the subtasks for a problem. If you've never used subtasks before, subtasks are a nice way to organize test cases into separate groups, allowing you to control the exact amount of partial credit given for solutions. For example, you could have a small subtask worth 30 points that accepts sub-optimal solutions, and a large subtask worth 70 points.
Each subtask should be in its own folder. These subtasks should have a bunch of test cases, with a .in and a .out file for each test case. In order for JudgeLite to load and run a test case, the .in and .out files should have the same filename (for example, hello.in and hello.out). Test cases are run in alphabetical order, while subtasks are run in an order that's defined in the info.yml file (explained in detail below).
The info.yml file contains useful information about a problem. It is loaded by JudgeLite to determine things like the time and memory limits, along with which subtasks to run. Here is a sample info.yml file:
---
problem_id: test1
problem_name: Test Problem #1
difficulty: Beginner
time_limit: 2
memory_limit: 256
scoring_method: average_stop
checker: diff
max_score: 100
subtasks:
- name: 01_main
score: 100
num_samples: 1
...An info.yml file has the following fields:
- problem_id - The ID of the problem. This should match the name of the directory that contains this problem.
- problem_name - The name of the problem. This doesn't have to match the name given in the problems.yml file, but it's highly recommended to keep them the same.
- difficulty (Optional) - The difficulty of the problem. You can decide how you'd like to use this, but we use the difficulties "Beginner", "Intermediate", and "Advanced".
- time_limit - The time limit of the problem (in seconds).
- memory_limit - The memory limit of the problem (in megabytes).
-
scoring_method - The way that submissions will be scored. This can be one of the following options:
- minimum - The minimum score out of all test cases in a subtask is used (all or nothing). If one test case fails in a subtask, the rest are immediately skipped (there would be no reason to continue anyways).
- average_stop - The score for each test case is averaged out to determine the points for each subtask (partial credit). The judge will stop on the first failed test in a subtask (the rest are skipped / treated with a score of 0). This keeps the judge from getting blocked up with submissions that TLE.
- average - The score for each test case is averaged out to determine the points for each subtask (partial credit). The judge will not stop early when evaluating a subtask. Be warned: This could cause the judge to get backed up if there are too many test cases!
-
checker - The checker used to judge a submission's output. This can be one of the following options:
- diff - Compares the program output with the answer using diff, ignoring carriage returns and differences in spacing. Produces a 0 or 1 score depending on whether the outputs match.
- custom - Uses a custom checker to produce a score for the program output. This score could be binary (0 or 1), or it could be a floating point number. See the Custom Checkers wiki page for details on how to use a custom checker.
- max_score - The maximum score that can be received by solving this problem, not including bonus points. This value could be completely arbitrary if you wanted it to be.
- fill_missing_output (Optional) - Set this field to true to fill in .out files if they are missing when you submit a program. This is useful for automatically generating output files to test cases. This will not overwrite any existing output files. MAKE SURE YOUR CODE IS CORRECT BEFORE USING THIS!!!
- subtasks - A list of subtasks that will be run when a submission is judged. This is explained in detail below.
The subtasks field contains a list of subtasks that JudgeLite will run when a submission is judged. An individual subtask has the following fields:
- name - The name of the subtask. This must match the directory name (so JudgeLite can find the subtask).
- score - The number of points the subtask is worth.
- num_samples (Optional) - The number of sample cases in this subtask (only needed if the subtask has samples). If the program does not get accepted on samples, the rest of the subtask will be skipped. Using this attribute saves valuable time when people accidentally submit completely wrong programs. (This only has a noticeable effect on the "average" scoring method, but it's best to include it anyway).
- is_bonus (Optional) - Set this field to true to mark a subtask as a bonus subtask.
- depends_on (Optional) - This field allows you to specify a list of subtasks that need to be "solved" (non-zero score) in order to run this one. This allows JudgeLite to skip unneeded subtasks, saving valuable time.
This is a list of helpful things to know when designing problems.
- Keeping a solutions and a misc folder could be pretty useful when creating problems, since it allows you to stay organized. The misc folder could contain things like testcase generators and solution explanations.
- Since test cases are run in alphabetical order, it's recommended to use a numbering system for the test names. You can add a description for each test case by putting it after the number (01_desc.in).
- In general, test cases should be ordered in (roughly) increasing difficulty / size.
- The idea behind bonus subtasks is to give something for experienced programmers to strive for, while also not discouraging newcomers with overly hard subtasks. This is why the results from bonus subtasks are separated from the regular results (our client side code actually hides bonus results unless the submitter got at least one bonus test correct).
- Try not to use the "average" grading method when you have lots of test cases, as that could slow down the system. If you do use that grading method, make sure to use the num_samples and depends_on fields to lower the amount of TLEs a wrong program could receive.
If anything in this wiki page seems unclear, please file a short issue on Github so the documentation can be improved.
Have fun creating problems, and thank you for using JudgeLite! ❤️