From 17c54e3a3b8ae9888df77fd4e617f1041555b657 Mon Sep 17 00:00:00 2001 From: hairez Date: Sun, 7 Sep 2025 12:33:20 +0200 Subject: [PATCH 1/6] Allow secret test cases to use .statement and .download --- spec/2025-09.md | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/spec/2025-09.md b/spec/2025-09.md index 7f72a67a..41c51602 100644 --- a/spec/2025-09.md +++ b/spec/2025-09.md @@ -645,12 +645,12 @@ For each test case: - `output_validator_args` defines arguments passed to the output validator for the test case. - `input_visualizer_args` defines arguments passed to the input visualizer for the test case. - `output_visualizer_args` defines arguments passed to the output visualizer for the test case. -- When `full_feedback` is `true`, somebody whose submission didn't pass case should be shown: - - the given input, - - the produced output (stdout), +- When `full_feedback` is `true`, somebody whose submission didn't pass the test case should be shown: + - contents of `.in.statement` if it exists, otherwise the given `.in` file. + - the produced output (stdout), - any error messages (stderr), - the illustration created by the output visualizer (if applicable), - - the expected output. + - contents of `.ans.statement` if it exists, otherwise the `.out` if it exists, else the `.ans` file. - `hint` provides feedback for solving a test case to, for example, somebody whose submission didn't pass. - `description` conveys the purpose of a test case. It is an explanation of what aspect or edge case of the solution the input file is meant to test. @@ -769,6 +769,21 @@ However, since only the `.out` files are validated it is advised to use these if Validation can be customized by specifying `input_validator_args` and `output_validator_args` in `data/sample/test_group.yaml`. +### Test Cases with Full Feedback + +If a test case has `full_feedback` set to `true` (see [Test Case Configuration](#test-case-configuration)), then by default, the `.in` and `.ans` pair for this particular testcase should be shown as feedback. +If a `.out` file exists the `.out` file is shown instead of the `.ans` file as feedback. +This behavior can be customized by creating files with extension `.in.statement` and `.ans.statement`. +If one of these files exists, its contents replaces that of the file with the same name -- except the `.statement` extension. + +Note that it is an error to provide both a `.out` and a `.ans.statement` file. + +By default, the `.in`, `.ans`, and `.files` files in `data/sample` are available for download. +Note that the content of `.in.statement` replaces that of `.in` and that the content of `.out` or `.ans.statement` replace that of `.ans` for the download. +This behavior can be further customized by providing files with the extension `in.download` or `ans.download`. +If one of these files exists, its contents replaces that of the file with the same name -- except the `.download` extension -- for the problem download. + +If you want to make other files -- like testing tools -- available for download, you can use [attachments](#attachments). ## Generators From 6add832b0522bf15373f753ef52c426b28165e98 Mon Sep 17 00:00:00 2001 From: hairez Date: Sun, 7 Sep 2025 13:29:25 +0200 Subject: [PATCH 2/6] =?UTF-8?q?Clarify=20download=20rules=20for=20full?= =?UTF-8?q?=E2=80=91feedback=20test=20cases?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- spec/2025-09.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/spec/2025-09.md b/spec/2025-09.md index 41c51602..5903e528 100644 --- a/spec/2025-09.md +++ b/spec/2025-09.md @@ -651,6 +651,8 @@ For each test case: - any error messages (stderr), - the illustration created by the output visualizer (if applicable), - contents of `.ans.statement` if it exists, otherwise the `.out` if it exists, else the `.ans` file. + + The test case with full feedback should also be available for download; the rules determining which files are provided are specified in [Test Cases with Full Feedback](#test-cases-with-full-feedback). - `hint` provides feedback for solving a test case to, for example, somebody whose submission didn't pass. - `description` conveys the purpose of a test case. It is an explanation of what aspect or edge case of the solution the input file is meant to test. From 44d35c893a556fe578af1c9eeb6ea3385cb0f868 Mon Sep 17 00:00:00 2001 From: Harry Zhang <75111093+hairez@users.noreply.github.com> Date: Mon, 8 Sep 2025 22:51:35 +0400 Subject: [PATCH 3/6] Update 2025-09.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Fredrik Niemelä --- spec/2025-09.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/2025-09.md b/spec/2025-09.md index 5903e528..9adf41ee 100644 --- a/spec/2025-09.md +++ b/spec/2025-09.md @@ -645,7 +645,7 @@ For each test case: - `output_validator_args` defines arguments passed to the output validator for the test case. - `input_visualizer_args` defines arguments passed to the input visualizer for the test case. - `output_visualizer_args` defines arguments passed to the output visualizer for the test case. -- When `full_feedback` is `true`, somebody whose submission didn't pass the test case should be shown: +- When `full_feedback` is `true`, the following should be made available to a solver that didn't pass the test case: - contents of `.in.statement` if it exists, otherwise the given `.in` file. - the produced output (stdout), - any error messages (stderr), From 6f1647fa63853633dbe46d8ba924240c8d2d778a Mon Sep 17 00:00:00 2001 From: Harry Zhang <75111093+hairez@users.noreply.github.com> Date: Mon, 8 Sep 2025 23:13:42 +0400 Subject: [PATCH 4/6] Update 2025-09.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Fredrik Niemelä --- spec/2025-09.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/2025-09.md b/spec/2025-09.md index 9adf41ee..1b1400d2 100644 --- a/spec/2025-09.md +++ b/spec/2025-09.md @@ -782,7 +782,7 @@ Note that it is an error to provide both a `.out` and a `.ans.statement` file. By default, the `.in`, `.ans`, and `.files` files in `data/sample` are available for download. Note that the content of `.in.statement` replaces that of `.in` and that the content of `.out` or `.ans.statement` replace that of `.ans` for the download. -This behavior can be further customized by providing files with the extension `in.download` or `ans.download`. +This behavior can be further customized by providing files with the extension `.in.download` or `.ans.download`. If one of these files exists, its contents replaces that of the file with the same name -- except the `.download` extension -- for the problem download. If you want to make other files -- like testing tools -- available for download, you can use [attachments](#attachments). From 667a359a4ab8a415a1b90ab1e5f59d1361566285 Mon Sep 17 00:00:00 2001 From: Harry Zhang <75111093+hairez@users.noreply.github.com> Date: Wed, 10 Sep 2025 01:22:57 +0200 Subject: [PATCH 5/6] Update spec/2025-09.md --- spec/2025-09.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/2025-09.md b/spec/2025-09.md index 1b1400d2..74f22b25 100644 --- a/spec/2025-09.md +++ b/spec/2025-09.md @@ -780,7 +780,7 @@ If one of these files exists, its contents replaces that of the file with the sa Note that it is an error to provide both a `.out` and a `.ans.statement` file. -By default, the `.in`, `.ans`, and `.files` files in `data/sample` are available for download. +By default, the `.in`, `.ans` files for the test cases with full feedback available for download, for the submission that fails this test case. Note that the content of `.in.statement` replaces that of `.in` and that the content of `.out` or `.ans.statement` replace that of `.ans` for the download. This behavior can be further customized by providing files with the extension `.in.download` or `.ans.download`. If one of these files exists, its contents replaces that of the file with the same name -- except the `.download` extension -- for the problem download. From cfd06d49ea0599fb8175155f9488b3ce6fa24f9d Mon Sep 17 00:00:00 2001 From: Harry Zhang <75111093+hairez@users.noreply.github.com> Date: Wed, 10 Sep 2025 01:23:08 +0200 Subject: [PATCH 6/6] Update spec/2025-09.md --- spec/2025-09.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/2025-09.md b/spec/2025-09.md index 74f22b25..f16d6afd 100644 --- a/spec/2025-09.md +++ b/spec/2025-09.md @@ -773,7 +773,7 @@ Validation can be customized by specifying `input_validator_args` and `output_va ### Test Cases with Full Feedback -If a test case has `full_feedback` set to `true` (see [Test Case Configuration](#test-case-configuration)), then by default, the `.in` and `.ans` pair for this particular testcase should be shown as feedback. +If a test case has `full_feedback` set to `true` (see [Test Case Configuration](#test-case-configuration)), then by default, the `.in` and `.ans` pair for this particular testcase should be shown as feedback, if the submission fails this test case. If a `.out` file exists the `.out` file is shown instead of the `.ans` file as feedback. This behavior can be customized by creating files with extension `.in.statement` and `.ans.statement`. If one of these files exists, its contents replaces that of the file with the same name -- except the `.statement` extension.