Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 54 additions & 54 deletions spec/2023-07-draft.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,9 +72,9 @@ File or Folder | Required? | Described In
`include/` | No | [Included Files](#included-files) | Files appended to all submitted solutions
`submissions/` | Yes | [Example Submissions](#example-submissions) | Correct and incorrect judge solutions of the problem
`input_validators/` | Yes | [Input Validators](#input-validators) | Programs that verifies correctness of the test data inputs
`input_visualizer/` | No | [Input Visualizer](#input-visualizer) | Scripts and documentation about how test case illustrations were generated
`output_validator/` | No | [Output Validator](#output-validator) | Custom program for judging solutions
`static_validator/` | No | [Static Validator](#static-validator) | Custom program for judging solutions with source files as input
`output_validator/` | No | [Output Validator](#output-validator) | Custom program for judging solutions
`input_visualizer/` | No | [Input Visualizer](#input-visualizer) | Scripts and documentation about how test case illustrations were generated
`output_visualizer/` | No | [Output Visualizer](#output-visualizer) | Program to generate images illustrating submission output

A minimal problem package must contain `problem.yaml`, a problem statement, a secret test case, an accepted judge solution, and an input validator.
Expand Down Expand Up @@ -949,10 +949,54 @@ Any other exit code means that the input file could not be confirmed as valid.
The validator **must not** read any files outside those defined in the Invocation section.
Its result **must** depend only on these files and the arguments.

## Input Visualizer
## Static Validator

If a tool was used to automate creating test case illustration annotations,
it is recommended to include the input visualizer source code in the directory `input_visualizer/` along with invocation instructions in a file such as `input_visualizer/README.txt`.
### Overview

A static validator is a program that is given the submission files as input and can analyze the contents to accept or reject the submission.
Optionally, the static validator may assign a score to the submission for each validation test case.
By default there is no static validator.
A static validator may be provided under the `static_validator` directory, similar to a custom output validator.

### Static Validation Test Cases

Each test group may define a static validation test case.
It is an error to define static validation test cases without providing a static validator.
A static validation test case is defined within a group's `testdata.yaml` file by specifying the key `static_validation`.
If a map is specified, its allowed key are:
- `args`, which maps to a string which represents the additional arguments passed to the static validator in this group's static validation test case;
- `score`, the maximum score of the static validation test case (see [Scoring Problems](#scoring-problems) for details).

The `static_validation` key can also have the value of `false` meaning there is no static validation, or `true` meaning that static validation is enabled with no additional arguments and unspecified maximum score (to be determined by [maximum score inference](#maximum-score-inference)).

It is an error to provide a static validator for `submit-answer` type problems, or to specify a `score` in a test group with `pass-fail` aggregation.

### Invocation

When invoked, the static validator will be passed at least three command line parameters.

The validator should be possible to use as follows on the command line:
```sh
<static_validator_program> language entry_point feedback_dir [additional_arguments]
```

The meaning of the parameters listed above are:

- language:
a string specifying the code of the language of the submission as shown in the [languages table](languages.md). A static validator must handle all of the programming languages specified in the `languages` key of `problem.yaml`.

- entry_point:
a string specifying the entry point, that is a filename, class name, or some other identifier, which the static validator should know how to use depending on the language of the submission.

- feedback_dir:
a string which specifies the name of a "feedback directory" in which the validator can produce "feedback files" in order to report additional information on the validation of the submission.
The feedback_dir must end with a path separator (typically '/' or '\\' depending on operating system),
so that simply appending a filename to feedback_dir gives the path to a file in the feedback directory.

- additional_arguments:
in case the static validation test case specifies additional args, these are passed as additional arguments to the validator on the command line.

The static validator follows the semantics of an output validator for [reporting a judgment](#reporting-a-judgement).

## Output Validator

Expand Down Expand Up @@ -1147,6 +1191,11 @@ Almost all test cases failed — are you even trying to solve the problem?
A validator program is allowed to write any kind of debug information to its standard error pipe.
This information may be displayed to the user upon invocation of the validator.

## Input Visualizer

If a tool was used to automate creating test case illustration annotations,
it is recommended to include the input visualizer source code in the directory `input_visualizer/` along with invocation instructions in a file such as `input_visualizer/README.txt`.

## Output Visualizer

An output visualizer is an optional [program](#programs) that is run after every invocation of the output validator in order to generate images illustrating the submission output.
Expand All @@ -1159,55 +1208,6 @@ It must not write to `score.txt`, `teammessage.txt`, or any other files in the f

Compile or run-time errors in the visualizer are not judge errors. The return value and any data written by the visualizer to standard error or standard output are ignored.

## Static Validator

### Overview

A static validator is a program that is given the submission files as input and can analyze the contents to accept or reject the submission.
Optionally, the static validator may assign a score to the submission for each validation test case.
By default there is no static validator.
A static validator may be provided under the `static_validator` directory, similar to a custom output validator.

### Static Validation Test Cases

Each test group may define a static validation test case.
It is an error to define static validation test cases without providing a static validator.
A static validation test case is defined within a group's `testdata.yaml` file by specifying the key `static_validation`.
If a map is specified, its allowed key are:
- `args`, which maps to a string which represents the additional arguments passed to the static validator in this group's static validation test case;
- `score`, the maximum score of the static validation test case (see [Scoring Problems](#scoring-problems) for details).

The `static_validation` key can also have the value of `false` meaning there is no static validation, or `true` meaning that static validation is enabled with no additional arguments and unspecified maximum score (to be determined by [maximum score inference](#maximum-score-inference)).

It is an error to provide a static validator for `submit-answer` type problems, or to specify a `score` in a test group with `pass-fail` aggregation.

### Invocation

When invoked, the static validator will be passed at least three command line parameters.

The validator should be possible to use as follows on the command line:
```sh
<static_validator_program> language entry_point feedback_dir [additional_arguments]
```

The meaning of the parameters listed above are:

- language:
a string specifying the code of the language of the submission as shown in the [languages table](languages.md). A static validator must handle all of the programming languages specified in the `languages` key of `problem.yaml`.

- entry_point:
a string specifying the entry point, that is a filename, class name, or some other identifier, which the static validator should know how to use depending on the language of the submission.

- feedback_dir:
a string which specifies the name of a "feedback directory" in which the validator can produce "feedback files" in order to report additional information on the validation of the submission.
The feedback_dir must end with a path separator (typically '/' or '\\' depending on operating system),
so that simply appending a filename to feedback_dir gives the path to a file in the feedback directory.

- additional_arguments:
in case the static validation test case specifies additional args, these are passed as additional arguments to the validator on the command line.

The static validator follows the semantics of an output validator for [reporting a judgment](#reporting-a-judgement).

## Verdict/Score Aggregation

### Pass-Fail Problems
Expand Down