Adding a problem

Overview

A problem is a task that a contestant must solve by sending a submission (a computer program written in any of the supported languages) for the problem to the grader system. A problem, at minimum, consists of a PDF that describes the task and a set of input/output pairs. When evaluating a submission of a contestant, each input is fed to the submission. The submission then compute an output from the given input and the grader system scores the submission by checking whether the computed output match the output of the problem.

Adding a problem can be done by preparing a problem archive file (.zip) and upload it to the system in the problem import page (/problems/import) by clicking at "Manage"->"Problems" menu and then click in "Import Problems" button. Enter the name and full name of the problem and pick the zip file to be uploaded. The system will create a new problem with the given name, or modifying the existing problem with the same name.

The naming of the content of the .zip file must follows the predefined structure. These directories and patterns and many parameters can also be configured in the config.yml file located anywhere in the .zip file.

Quick Example

The most simple form of a problem archive file consists of one PDF file and equal number of input/output files named as 1.in, 1.sol, 2.sin, 2.sol, ..., x.in, x.sol. See some archive file examples in https://github.com/nattee/cafe-grader-web/tree/master/test/problem_examples.

Dataset

A dataset is the main content of a problem, combining testcases, checkers, managers and some other files that are used in evaluation of a submission. A problem is required to have at least one dataset, and, in most cases, only one dataset is enough. However, if we would like to have multiple version of a problem, each having different set of testcases, this can be easily managed by using multiple datasets.

A problem is required to have one "live" dataset which is the dataset to be used for evaluation of submissions. When having multiple datasets, we can select which dataset is live in the problem edit page. In the page, we can also import another archive as a dataset of existing problem as well.

Some content of a problem are shared between datasets. This include the problem statements, any attachments and other problem specific info such as tags, groups, problem name, etc. See the problem edit page for details.

When importing a problem to the system, the files in the archive is grouped into one dataset.

Directory Structure and pattern

If any of these directories exists, the system will assume that only the content of the directories, or file, as the type identified by the name.

• Any .pdf files located anywhere will be assumed to be the problem statement.
• managers contains managers file.
• model_solutions contains model solutions.
• checker contains the custom checker.
• attachment contains any attachment of the problem that will be available for the users.

Testcases

Each testcase consists of a pair of input and answer file. The input file is the file given to an executable of a contestant as an input and the output of the executable is compared with the answer file. The testcase pairs can be placed anywhere in the archive but all of them should be in the same directory. The file should be name *.in and *.sol for input and answer files respectively. Files with the same filename are matched together and form a single testcase.

Basically, the easiest way to include testcases in the archive is to put files named 1.in, 1.sol, 2.sin, 2.sol, ..., x.in, x.sol in any directory (including the base directory) of the archive.

We can also change the matching wildcard,

config.yml

-- to be edited --