From a5e6e1741a3ebb7db68b2c069904325d2e3ff1f4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fredrik=20Niemel=C3=A4?= Date: Fri, 14 Mar 2025 08:14:25 +0100 Subject: [PATCH] Reorder validator and visualizer sections --- spec/2023-07-draft.md | 108 +++++++++++++++++++++--------------------- 1 file changed, 54 insertions(+), 54 deletions(-) diff --git a/spec/2023-07-draft.md b/spec/2023-07-draft.md index 8f67086d..9d7cfbab 100644 --- a/spec/2023-07-draft.md +++ b/spec/2023-07-draft.md @@ -72,9 +72,9 @@ File or Folder | Required? | Described In `include/` | No | [Included Files](#included-files) | Files appended to all submitted solutions `submissions/` | Yes | [Example Submissions](#example-submissions) | Correct and incorrect judge solutions of the problem `input_validators/` | Yes | [Input Validators](#input-validators) | Programs that verifies correctness of the test data inputs -`input_visualizer/` | No | [Input Visualizer](#input-visualizer) | Scripts and documentation about how test case illustrations were generated -`output_validator/` | No | [Output Validator](#output-validator) | Custom program for judging solutions `static_validator/` | No | [Static Validator](#static-validator) | Custom program for judging solutions with source files as input +`output_validator/` | No | [Output Validator](#output-validator) | Custom program for judging solutions +`input_visualizer/` | No | [Input Visualizer](#input-visualizer) | Scripts and documentation about how test case illustrations were generated `output_visualizer/` | No | [Output Visualizer](#output-visualizer) | Program to generate images illustrating submission output A minimal problem package must contain `problem.yaml`, a problem statement, a secret test case, an accepted judge solution, and an input validator. @@ -949,10 +949,54 @@ Any other exit code means that the input file could not be confirmed as valid. The validator **must not** read any files outside those defined in the Invocation section. Its result **must** depend only on these files and the arguments. -## Input Visualizer +## Static Validator -If a tool was used to automate creating test case illustration annotations, -it is recommended to include the input visualizer source code in the directory `input_visualizer/` along with invocation instructions in a file such as `input_visualizer/README.txt`. +### Overview + +A static validator is a program that is given the submission files as input and can analyze the contents to accept or reject the submission. +Optionally, the static validator may assign a score to the submission for each validation test case. +By default there is no static validator. +A static validator may be provided under the `static_validator` directory, similar to a custom output validator. + +### Static Validation Test Cases + +Each test group may define a static validation test case. +It is an error to define static validation test cases without providing a static validator. +A static validation test case is defined within a group's `testdata.yaml` file by specifying the key `static_validation`. +If a map is specified, its allowed key are: +- `args`, which maps to a string which represents the additional arguments passed to the static validator in this group's static validation test case; +- `score`, the maximum score of the static validation test case (see [Scoring Problems](#scoring-problems) for details). + +The `static_validation` key can also have the value of `false` meaning there is no static validation, or `true` meaning that static validation is enabled with no additional arguments and unspecified maximum score (to be determined by [maximum score inference](#maximum-score-inference)). + +It is an error to provide a static validator for `submit-answer` type problems, or to specify a `score` in a test group with `pass-fail` aggregation. + +### Invocation + +When invoked, the static validator will be passed at least three command line parameters. + +The validator should be possible to use as follows on the command line: +```sh + language entry_point feedback_dir [additional_arguments] +``` + +The meaning of the parameters listed above are: + +- language: + a string specifying the code of the language of the submission as shown in the [languages table](languages.md). A static validator must handle all of the programming languages specified in the `languages` key of `problem.yaml`. + +- entry_point: + a string specifying the entry point, that is a filename, class name, or some other identifier, which the static validator should know how to use depending on the language of the submission. + +- feedback_dir: + a string which specifies the name of a "feedback directory" in which the validator can produce "feedback files" in order to report additional information on the validation of the submission. + The feedback_dir must end with a path separator (typically '/' or '\\' depending on operating system), + so that simply appending a filename to feedback_dir gives the path to a file in the feedback directory. + +- additional_arguments: + in case the static validation test case specifies additional args, these are passed as additional arguments to the validator on the command line. + +The static validator follows the semantics of an output validator for [reporting a judgment](#reporting-a-judgement). ## Output Validator @@ -1147,6 +1191,11 @@ Almost all test cases failed — are you even trying to solve the problem? A validator program is allowed to write any kind of debug information to its standard error pipe. This information may be displayed to the user upon invocation of the validator. +## Input Visualizer + +If a tool was used to automate creating test case illustration annotations, +it is recommended to include the input visualizer source code in the directory `input_visualizer/` along with invocation instructions in a file such as `input_visualizer/README.txt`. + ## Output Visualizer An output visualizer is an optional [program](#programs) that is run after every invocation of the output validator in order to generate images illustrating the submission output. @@ -1159,55 +1208,6 @@ It must not write to `score.txt`, `teammessage.txt`, or any other files in the f Compile or run-time errors in the visualizer are not judge errors. The return value and any data written by the visualizer to standard error or standard output are ignored. -## Static Validator - -### Overview - -A static validator is a program that is given the submission files as input and can analyze the contents to accept or reject the submission. -Optionally, the static validator may assign a score to the submission for each validation test case. -By default there is no static validator. -A static validator may be provided under the `static_validator` directory, similar to a custom output validator. - -### Static Validation Test Cases - -Each test group may define a static validation test case. -It is an error to define static validation test cases without providing a static validator. -A static validation test case is defined within a group's `testdata.yaml` file by specifying the key `static_validation`. -If a map is specified, its allowed key are: -- `args`, which maps to a string which represents the additional arguments passed to the static validator in this group's static validation test case; -- `score`, the maximum score of the static validation test case (see [Scoring Problems](#scoring-problems) for details). - -The `static_validation` key can also have the value of `false` meaning there is no static validation, or `true` meaning that static validation is enabled with no additional arguments and unspecified maximum score (to be determined by [maximum score inference](#maximum-score-inference)). - -It is an error to provide a static validator for `submit-answer` type problems, or to specify a `score` in a test group with `pass-fail` aggregation. - -### Invocation - -When invoked, the static validator will be passed at least three command line parameters. - -The validator should be possible to use as follows on the command line: -```sh - language entry_point feedback_dir [additional_arguments] -``` - -The meaning of the parameters listed above are: - -- language: - a string specifying the code of the language of the submission as shown in the [languages table](languages.md). A static validator must handle all of the programming languages specified in the `languages` key of `problem.yaml`. - -- entry_point: - a string specifying the entry point, that is a filename, class name, or some other identifier, which the static validator should know how to use depending on the language of the submission. - -- feedback_dir: - a string which specifies the name of a "feedback directory" in which the validator can produce "feedback files" in order to report additional information on the validation of the submission. - The feedback_dir must end with a path separator (typically '/' or '\\' depending on operating system), - so that simply appending a filename to feedback_dir gives the path to a file in the feedback directory. - -- additional_arguments: - in case the static validation test case specifies additional args, these are passed as additional arguments to the validator on the command line. - -The static validator follows the semantics of an output validator for [reporting a judgment](#reporting-a-judgement). - ## Verdict/Score Aggregation ### Pass-Fail Problems