diff --git a/spec/2025-09.md b/spec/2025-09.md index 7f72a67a..f16d6afd 100644 --- a/spec/2025-09.md +++ b/spec/2025-09.md @@ -645,12 +645,14 @@ For each test case: - `output_validator_args` defines arguments passed to the output validator for the test case. - `input_visualizer_args` defines arguments passed to the input visualizer for the test case. - `output_visualizer_args` defines arguments passed to the output visualizer for the test case. -- When `full_feedback` is `true`, somebody whose submission didn't pass case should be shown: - - the given input, - - the produced output (stdout), +- When `full_feedback` is `true`, the following should be made available to a solver that didn't pass the test case: + - contents of `.in.statement` if it exists, otherwise the given `.in` file. + - the produced output (stdout), - any error messages (stderr), - the illustration created by the output visualizer (if applicable), - - the expected output. + - contents of `.ans.statement` if it exists, otherwise the `.out` if it exists, else the `.ans` file. + + The test case with full feedback should also be available for download; the rules determining which files are provided are specified in [Test Cases with Full Feedback](#test-cases-with-full-feedback). - `hint` provides feedback for solving a test case to, for example, somebody whose submission didn't pass. - `description` conveys the purpose of a test case. It is an explanation of what aspect or edge case of the solution the input file is meant to test. @@ -769,6 +771,21 @@ However, since only the `.out` files are validated it is advised to use these if Validation can be customized by specifying `input_validator_args` and `output_validator_args` in `data/sample/test_group.yaml`. +### Test Cases with Full Feedback + +If a test case has `full_feedback` set to `true` (see [Test Case Configuration](#test-case-configuration)), then by default, the `.in` and `.ans` pair for this particular testcase should be shown as feedback, if the submission fails this test case. +If a `.out` file exists the `.out` file is shown instead of the `.ans` file as feedback. +This behavior can be customized by creating files with extension `.in.statement` and `.ans.statement`. +If one of these files exists, its contents replaces that of the file with the same name -- except the `.statement` extension. + +Note that it is an error to provide both a `.out` and a `.ans.statement` file. + +By default, the `.in`, `.ans` files for the test cases with full feedback available for download, for the submission that fails this test case. +Note that the content of `.in.statement` replaces that of `.in` and that the content of `.out` or `.ans.statement` replace that of `.ans` for the download. +This behavior can be further customized by providing files with the extension `.in.download` or `.ans.download`. +If one of these files exists, its contents replaces that of the file with the same name -- except the `.download` extension -- for the problem download. + +If you want to make other files -- like testing tools -- available for download, you can use [attachments](#attachments). ## Generators