From 2af18a6fd9f84d73d61a951b163a9de6d926ab9d Mon Sep 17 00:00:00 2001 From: ee7 <45465154+ee7@users.noreply.github.com> Date: Wed, 7 Jul 2021 16:00:25 +0200 Subject: [PATCH 1/7] Clarify top-level `fail` in test runner spec Consider the situation of: - The user submits a solution in a compiled language - The user's solution compiles without error - The first test passes, but the next test "produces an error" (for example, it raises an exception). In this case, the correct top-level status is `fail`, with one test-level status of `pass`, and one test-level status of `error`. But the previous wording could imply that a top-level status of `fail` is only correct if at least one test has the status `fail`. --- building/tooling/test-runners/interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/building/tooling/test-runners/interface.md b/building/tooling/test-runners/interface.md index 48342d68..febc9126 100644 --- a/building/tooling/test-runners/interface.md +++ b/building/tooling/test-runners/interface.md @@ -40,7 +40,7 @@ The version of the spec that this file adheres to: The following overall statuses are valid: - `pass`: All tests passed -- `fail`: At least one test failed +- `fail`: At least one test has the status `fail` or `error` - `error`: To be used when the tests did not run correctly (e.g. a compile error, a syntax error) #### Message From 8b5174ad15f167f5692aeebe9a94ce174a9f55a2 Mon Sep 17 00:00:00 2001 From: ee7 <45465154+ee7@users.noreply.github.com> Date: Wed, 7 Jul 2021 16:00:26 +0200 Subject: [PATCH 2/7] Clarify per-test `message` in test runner spec The previous wording could imply that a per-test `message` property can only be present when the test `status` is `fail`. --- building/tooling/test-runners/interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/building/tooling/test-runners/interface.md b/building/tooling/test-runners/interface.md index febc9126..49254a60 100644 --- a/building/tooling/test-runners/interface.md +++ b/building/tooling/test-runners/interface.md @@ -127,7 +127,7 @@ The following per-test statuses are valid: > version: 2, 3 -The per-test `message` key is used to return the results of a failed test. It should be as human-readable as possible. Whatever is written here will be displayed to the student when their test fails. If there is no error message, either set the value to `null` or omit the key entirely. It is also permissible to output test suite output here. The `message` value is not limited in length. +The per-test `message` key is used to return the results of a test with a `status` of `fail` or `error`. It should be as human-readable as possible. Whatever is written here will be displayed to the student when their test does not pass. If there is no informational message, either set the value to `null` or omit the key entirely. It is also permissible to output test suite output here. The `message` value is not limited in length. #### Output From 920131c04a67a58297b158729463807673fd0749 Mon Sep 17 00:00:00 2001 From: ee7 <45465154+ee7@users.noreply.github.com> Date: Wed, 7 Jul 2021 16:00:27 +0200 Subject: [PATCH 3/7] Clarify top-level `error` in test runner spec To me, the previous wording implied that a top-level `status` of `error` is valid when at least one test "errors". But actually, if at least one test was "executed", the top-level status should be `pass` or `fail`. --- building/tooling/test-runners/interface.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/building/tooling/test-runners/interface.md b/building/tooling/test-runners/interface.md index 49254a60..e8a6782d 100644 --- a/building/tooling/test-runners/interface.md +++ b/building/tooling/test-runners/interface.md @@ -41,7 +41,7 @@ The following overall statuses are valid: - `pass`: All tests passed - `fail`: At least one test has the status `fail` or `error` -- `error`: To be used when the tests did not run correctly (e.g. a compile error, a syntax error) +- `error`: No test was executed correctly (this usually means e.g. a compile error, or a syntax error) #### Message @@ -49,7 +49,7 @@ The following overall statuses are valid: > version: 1, 2, 3 -Where the status is `error` (the tests fail to execute cleanly), the top level `message` key should be provided. It should provide the occurring error to the user. As it is the only piece of information a user will receive on how to debug their issue, it must be as clear as possible. For example, in Ruby, in the case of a syntax error, we provide the error and stack trace. In compiled languages, the compilation error should be provided. The top level `message` value is not limited in length. +Where the status is `error` (no test was executed cleanly), the top level `message` key should be provided. It should provide the occurring error to the user. As it is the only piece of information a user will receive on how to debug their issue, it must be as clear as possible. For example, in Ruby, in the case of a syntax error, we provide the error and stack trace. In compiled languages, the compilation error should be provided. The top level `message` value is not limited in length. When the status is not `error`, either set the value to `null` or omit the key entirely. From 718393fa20ce07d6affd9cfa8773e01b8c2e215f Mon Sep 17 00:00:00 2001 From: ee7 <45465154+ee7@users.noreply.github.com> Date: Wed, 7 Jul 2021 16:00:28 +0200 Subject: [PATCH 4/7] Try to clarify test-level `error` in test runner spec --- building/tooling/test-runners/interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/building/tooling/test-runners/interface.md b/building/tooling/test-runners/interface.md index e8a6782d..6f964668 100644 --- a/building/tooling/test-runners/interface.md +++ b/building/tooling/test-runners/interface.md @@ -119,7 +119,7 @@ The following per-test statuses are valid: - `pass`: The test passed - `fail`: The test failed -- `error`: The test errored +- `error`: The test errored - that is, it did not return a value #### Message From 5daab45333d47c0dd7fb50eb021b72def6b364bd Mon Sep 17 00:00:00 2001 From: ee7 <45465154+ee7@users.noreply.github.com> Date: Thu, 22 Jul 2021 11:09:24 +0200 Subject: [PATCH 5/7] Further clarify test-level `message` Co-authored-by: Erik Schierboom --- building/tooling/test-runners/interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/building/tooling/test-runners/interface.md b/building/tooling/test-runners/interface.md index 6f964668..92443784 100644 --- a/building/tooling/test-runners/interface.md +++ b/building/tooling/test-runners/interface.md @@ -127,7 +127,7 @@ The following per-test statuses are valid: > version: 2, 3 -The per-test `message` key is used to return the results of a test with a `status` of `fail` or `error`. It should be as human-readable as possible. Whatever is written here will be displayed to the student when their test does not pass. If there is no informational message, either set the value to `null` or omit the key entirely. It is also permissible to output test suite output here. The `message` value is not limited in length. +The per-test `message` key is used to return the results of a test with a `status` of `fail` or `error`. It should be as human-readable as possible. Whatever is written here will be displayed to the student when their test does not pass. If there is no test failure or error message, either set the value to `null` or omit the key entirely. It is also permissible to output test suite output here. The `message` value is not limited in length. #### Output From e8cb9c5520a71391875c3c541d701f94f31eb88d Mon Sep 17 00:00:00 2001 From: ee7 <45465154+ee7@users.noreply.github.com> Date: Thu, 22 Jul 2021 11:28:19 +0200 Subject: [PATCH 6/7] Describe top-level `error` more consistently Co-authored-by: Erik Schierboom --- building/tooling/test-runners/interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/building/tooling/test-runners/interface.md b/building/tooling/test-runners/interface.md index 92443784..eadfac6f 100644 --- a/building/tooling/test-runners/interface.md +++ b/building/tooling/test-runners/interface.md @@ -49,7 +49,7 @@ The following overall statuses are valid: > version: 1, 2, 3 -Where the status is `error` (no test was executed cleanly), the top level `message` key should be provided. It should provide the occurring error to the user. As it is the only piece of information a user will receive on how to debug their issue, it must be as clear as possible. For example, in Ruby, in the case of a syntax error, we provide the error and stack trace. In compiled languages, the compilation error should be provided. The top level `message` value is not limited in length. +Where the status is `error` (no test was executed correctly), the top level `message` key should be provided. It should provide the occurring error to the user. As it is the only piece of information a user will receive on how to debug their issue, it must be as clear as possible. For example, in Ruby, in the case of a syntax error, we provide the error and stack trace. In compiled languages, the compilation error should be provided. The top level `message` value is not limited in length. When the status is not `error`, either set the value to `null` or omit the key entirely. From 60e3eda6f6132ccff4c7e2452ca1c2b467be547e Mon Sep 17 00:00:00 2001 From: Erik Schierboom Date: Thu, 22 Jul 2021 12:16:14 +0200 Subject: [PATCH 7/7] Update building/tooling/test-runners/interface.md Co-authored-by: ee7 <45465154+ee7@users.noreply.github.com> --- building/tooling/test-runners/interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/building/tooling/test-runners/interface.md b/building/tooling/test-runners/interface.md index eadfac6f..36eb8880 100644 --- a/building/tooling/test-runners/interface.md +++ b/building/tooling/test-runners/interface.md @@ -127,7 +127,7 @@ The following per-test statuses are valid: > version: 2, 3 -The per-test `message` key is used to return the results of a test with a `status` of `fail` or `error`. It should be as human-readable as possible. Whatever is written here will be displayed to the student when their test does not pass. If there is no test failure or error message, either set the value to `null` or omit the key entirely. It is also permissible to output test suite output here. The `message` value is not limited in length. +The per-test `message` key is used to return the results of a test with a `status` of `fail` or `error`. It should be as human-readable as possible. Whatever is written here will be displayed to the student when their test does not pass. If there is no test failure message or error message, either set the value to `null` or omit the key entirely. It is also permissible to output test suite output here. The `message` value is not limited in length. #### Output