Merge pull request #76 from gradescope/ibrahim/document-test-status

Document test "status" field

docs/specs.md

@@ -48,6 +48,7 @@ Your autograder's output should be in the file results.json, in the following fo
         {
             "score": 2.0, // optional, but required if not on top level submission
             "max_score": 2.0, // optional
+            "status": "passed", // optional, see "Test case status" below
             "name": "Your name here", // optional
             "number": "1.1", // optional (will just be numbered in order of array if no number given)
             "output": "Giant multiline string that will be placed in a <pre> tag and collapsed by default", // optional
@@ -109,6 +110,41 @@ should be one of `hidden`, `after_due_date`, `after_published`, or `visible`,
 depending on when you want stdout to be visible to students. These options
 mirror the test case visibility options described above.
 
+
+### Test case status
+
+The `"status"` field on a test case can be used to customize the test case's
+appearance. By default, test cases where the student was awarded the
+`"max_score"` points are marked as passing, and tests where they achieved a
+score lower than the max are considered failing. To override this, you can set
+the `"status"` field to either `"passed"` or `"failed"` to mark the test as
+passing or failing respectively. You can omit score information to add test
+cases that serve as informational checks with a passing or failing status. See
+an example below:
+
+```json
+{
+  "tests": [
+    {
+      "name": "Compilation check",
+      "status": "passed"
+    },
+    {
+      "name": "Style check",
+      "status": "failed"
+    },
+    {
+      "name": "Test 3",
+      "score": 5,
+      "max_score": 10,
+      "status": "passed"
+    }
+  ]
+}
+```
+
+[<img src="../test_status.png" width="1253px" />](test_status.png)
+
 ### Leaderboards
 
 You can add a "leaderboard" section to create leaderboards for your

docs/updates.md

@@ -2,6 +2,10 @@
 
 Here are some updates we've made to our autograder platform. If you have any questions or issues with any of these changes, please email [help@gradescope.com](mailto:help@gradescope.com).
 
+## Aug 16th, 2022
+
+We have added the ability to specify whether a test case should be considered to have passed or failed, overriding the default styling. See [our documentation on the results.json format](specs#test-case-status) to learn more.
+
 ## Aug 3rd, 2022
 
 We have updated the default autograder base image to Ubuntu 22.04, the current LTS release of Ubuntu. This will make newer versions of packages available to install.

mkdocs.yml

@@ -3,7 +3,7 @@ repo_url: https://github.com/gradescope/autograder_samples
 docs_dir: docs
 theme: readthedocs
 extra_css: [gradescope.css]
-pages:
+nav:
 - 'Home': index.md
 - 'Updates': updates.md
 - 'Getting Started': getting_started.md