Skip to content

Clean up the details of static validators #377

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Mar 12, 2025
Merged

Clean up the details of static validators #377

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
09ef2cf
Clean up the details of static validators
evouga Mar 7, 2025
a446a2b
Add the score back in for static validation test cases
evouga Mar 10, 2025
c7d9c86
Point to maximum score inference
Mar 11, 2025
3154f90
Merge branch 'master' into static-validators
Mar 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 59 additions & 59 deletions spec/2023-07-draft.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,8 @@ File or Folder | Required? | Described In
`input_validators/` | Yes | [Input Validators](#input-validators) | Programs that verifies correctness of the test data inputs
`input_visualizer/` | No | [Input Visualizer](#input-visualizer) | Scripts and documentation about how test case illustrations were generated
`output_validator/` | No | [Output Validator](#output-validator) | Custom program for judging solutions
`output_visualizer/` | No | [Output Visualizer](#output-visualizer) | Program to generate images illustrating submission output
`static_validator/` | No | [Static Validator](#static-validator) | Custom program for judging solutions with source files as input
`output_visualizer/` | No | [Output Visualizer](#output-visualizer) | Program to generate images illustrating submission output
Copy link
Member

@niemela niemela Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The order here seems strange:

input_validators
input_visualizer
output_validator
static_validator
output_visualizer

I would suggest:

static_validator
input_validators
input_visualizer
output_validator
output_visualizer

or possibly:

input_validators
input_visualizer
output_validator
output_visualizer
static_validator

(But I think the former is better)

Copy link
Collaborator Author

@evouga evouga Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The static validator is much closer in function to the output validator (which judges contestant submissions) than to the input validator (which checks the judge data only).

Copy link
Collaborator Author

@evouga evouga Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see an argument for putting the static validator right before the output validator, but I prefer the current ordering as it lets us say that the static validator shares the judgement semantics of the output validator.

Copy link
Contributor

@thorehusfeldt thorehusfeldt Mar 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here’s a categorisation by problem life cycle:

Output validation (statically or not) belongs to submission-time. It is performed when a solver submission is validated.

Input validation belongs to installation-time of a problem. It is performed when a problem is installed on a judge system.

Visualisation belongs to problem’s construction-time. It is performed when the problem is designed. (Like input generation.)

Copy link
Member

@niemela niemela Mar 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see an argument for putting the static validator right before the output validator, but I prefer the current ordering as it lets us say that the static validator shares the judgement semantics of the output validator.

It feels strange to me to group input_validators and input_visualizer together, but then put static_validator between output_validator and output_visualizer.

The most logical order of the validators would be input_validators (mainly install time), static_validator (run-time, but in some sense "early" run-time), an then output_validator ("later" run-time). If we want to group the visualizers with "their" validators we would get:

input_validators
input_visualizer
static_validator
output_validator
output_visualizer

But actually... I think the logical grouping would be to have all the validators together and all the visualizers together. visualizers belong to judge-time, which is after run-time, so:

input_validators
static_validator
output_validator
input_visualizer
output_visualizer

I think this would be the best order?

Copy link
Collaborator Author

@evouga evouga Mar 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let us discuss this in a separate issue?


A minimal problem package must contain `problem.yaml`, a problem statement, a secret test case, an accepted judge solution, and an input validator.

Expand Down Expand Up @@ -1168,6 +1168,55 @@ It must not write to `score.txt`, `teammessage.txt`, or any other files in the f

Compile or run-time errors in the visualizer are not judge errors. The return value and any data written by the visualizer to standard error or standard output are ignored.

## Static Validator

### Overview

A static validator is a program that is given the submission files as input and can analyze the contents to accept or reject the submission.
Optionally, the static validator may assign a score to the submission for each validation test case.
By default there is no static validator.
A static validator may be provided under the `static_validator` directory, similar to a custom output validator.

### Static Validation Test Cases

Each test group may define a static validation test case.
It is an error to define static validation test cases without providing a static validator.
A static validation test case is defined within a group's `testdata.yaml` file by specifying the key `static_validation`.
If a map is specified, its allowed key are:
- `args`, which maps to a string which represents the additional arguments passed to the static validator in this group's static validation test case;
- `score`, the maximum score of the static validation test case (see [Scoring Problems](#scoring-problems) for details).

The `static_validation` key can also have the value of `false` meaning there is no static validation, or `true` meaning that static validation is enabled with no additional arguments and unspecified maximum score (to be determined by [maximum score inference](#maximum-score-inference)).

It is an error to provide a static validator for `submit-answer` type problems, or to specify a `score` in a test group with `pass-fail` aggregation.

### Invocation

When invoked, the static validator will be passed at least three command line parameters.

The validator should be possible to use as follows on the command line:
```sh
<static_validator_program> language entry_point feedback_dir [additional_arguments]
```

The meaning of the parameters listed above are:

- language:
a string specifying the code of the language of the submission as shown in the [languages table](languages.md). A static validator must handle all of the programming languages specified in the `languages` key of `problem.yaml`.

- entry_point:
a string specifying the entry point, that is a filename, class name, or some other identifier, which the static validator should know how to use depending on the language of the submission.

- feedback_dir:
a string which specifies the name of a "feedback directory" in which the validator can produce "feedback files" in order to report additional information on the validation of the submission.
The feedback_dir must end with a path separator (typically '/' or '\\' depending on operating system),
so that simply appending a filename to feedback_dir gives the path to a file in the feedback directory.

- additional_arguments:
in case the static validation test case specifies additional args, these are passed as additional arguments to the validator on the command line.

The static validator follows the semantics of an output validator for [reporting a judgment](#reporting-a-judgement).

## Verdict/Score Aggregation

### Pass-Fail Problems
Expand Down Expand Up @@ -1206,7 +1255,7 @@ Parent Group Maximum Score | Aggregation Type | Default Maximum Score of Tes
bounded value `M` | `sum` | `(M - S)/(A + T)`
bounded value `M` | `min` | `M`

where the group has `T` test cases, `A` subgroups without a provided `score`, and whose other subgroups have maximum scores that sum to `S`.
where the group has `T` non-static-validation test cases, `A` subgroups and static validation test cases without a provided `score`, and whose other subgroups and static validation test cases have maximum scores that sum to `S`.
This formula evenly distributes a group's leftover maximum points to its test cases and subgroups with unspecified maximum score.
It is a judge error if `S > M` for a group with bounded maximum score and `sum` aggregation.

Expand All @@ -1216,7 +1265,7 @@ Only test cases in test case groups with `sum` or `min` aggregation receive a sc

The score of a failed test case is always 0.

A custom output validator may produce a `score.txt` or `score_multiplier.txt` file for an accepted test case:
A custom output validator or static validator may produce a `score.txt` or `score_multiplier.txt` file for an accepted test case:

- for test cases with bounded maximum score, `score_multiplier.txt`, if produced, must contain a single floating-point number in the range `[0,1]`.
The score of the test case is this number _multiplied_ by the test case maximum score.
Expand All @@ -1225,14 +1274,14 @@ A custom output validator may produce a `score.txt` or `score_multiplier.txt` fi
- for test cases with bounded maximum score, if no `score_multiplier.txt` or `score.txt` is produced, the test case score is its maximum score.
- for test cases with unbounded maximum score, `score.txt` must be produced and must contain a non-negative floating-point number.
The score of the test case is that number.

It is a judge error if:
- an output validator accepts a test case in an unbounded group and does not produce a `score.txt`;
- an output validator does not accept a test case, but does produce a `score.txt` or a `score_multiplier.txt`;
- an output validator produces a `score_multiplier.txt` for a test case with unbounded maximum score;
- an output validator produces both a `score.txt` and a `score_multiplier.txt` for a test case;
- an output validator produces a `score.txt` or `score_multiplier.txt` for a test case in a group with `pass-fail` aggregation;
- an output validator produces a `score.txt` or `score_multiplier.txt` with invalid contents.
- an output or static validator accepts a test case in an unbounded group and does not produce a `score.txt`;
- an output or static validator does not accept a test case, but does produce a `score.txt` or a `score_multiplier.txt`;
- an output or static validator produces a `score_multiplier.txt` for a test case with unbounded maximum score;
- an output or static validator produces both a `score.txt` and a `score_multiplier.txt` for a test case;
- an output or static validator produces a `score.txt` or `score_multiplier.txt` for a test case in a group with `pass-fail` aggregation;
- an output or static validator produces a `score.txt` or `score_multiplier.txt` with invalid contents.

#### Scoring Test Groups

Expand Down Expand Up @@ -1260,52 +1309,3 @@ The paths of these required test cases or groups, relative to the `data` folder,
The path of a group, relative to the `data/` folder, must come later lexicographically than the paths of all dependent test cases and groups.

Each required group must be either `sample` or a subgroup of `secret` with `pass-fail` aggregation.

## Static Validator

### Overview

A static validator is a program that is given the submission files as input and can analyze the contents to accept or reject the submission.
Optionally, the static validator may assign a score to the submission.
By default there is no static validator.
A static validator may be provided under the `static_validator` directory, similar to a custom output validator.

### Static Validation Test Cases

Each test group may define a static validation test case.
It is an error to define static validation test cases without providing a static validator.
A static validation test case is defined within a group's `test_group.yaml` file by specifying the key `static_validation`.
If a map is specified, it may have two keys `args`, and in the case of scoring test groups, `score`.
The key `args` maps to a string which represents the additional arguments passed to the static validator in this group's static validation test case.
The key `score` maps to a float which represents both the maximum score achievable for the static validation test case and the default score assigned in case the static validator accepts the submission for that test case.
The static validator can override this score by outputting a value to `score.txt` in the feedback directory, the same as an output validator.
It can have the value of `false` meaning there is no static validation, `true` meaning that static validation is enabled with no score defined and no arguments.
Aggregation is then applied to the test case in the same manner as other test cases.
It is an error to assign a score in a `pass-fail` test group.
It is an error to not assign a score in a `min` or `sum` test group.
It is also an error to provide a static validator for `submit-answer` type problems.

### Invocation

When invoked, the static validator will be passed at least three command line parameters.

The validator should be possible to use as follows on the command line:
```sh
<static_validator_program> language entry_point feedback_dir [additional_arguments]
```

The meaning of the parameters listed above are:

- language:
a string specifying the code of the language of the submission as shown in the [languages table](languages.md).

- entry_point:
a string specifying the entry point, that is a filename, class name, or some other identifier, which the static validator should know how to use depending on the language of the submission.

- feedback_dir:
a string which specifies the name of a "feedback directory" in which the validator can produce "feedback files" in order to report additional information on the validation of the submission.
The feedback_dir must end with a path separator (typically '/' or '\\' depending on operating system),
so that simply appending a filename to feedback_dir gives the path to a file in the feedback directory.

- additional_arguments:
in case the static validation test case specifies additional args, these are passed as additional arguments to the validator on the command line.