Skip to content

Commit

Permalink
Add catalog and validator plans for ai questions. (#9950)
Browse files Browse the repository at this point in the history
The implementation is not included here, but the docs changes are still needed to have the criteria appear in uploaded targets.

Of note: this does introduce the idea of a "system" parameter, which is an input to the validator plan which can change based on context, but which the user doesn't really need to specify in the crtitera template. In this case, that's the share id. The plan is for the teacher tool to automatically populate system parameters based on the provided key.
  • Loading branch information
thsparks committed Apr 5, 2024
1 parent fe0b6e7 commit cf4f60b
Show file tree
Hide file tree
Showing 2 changed files with 22 additions and 4 deletions.
14 changes: 10 additions & 4 deletions common-docs/teachertool/test/catalog-shared.json
Original file line number Diff line number Diff line change
Expand Up @@ -37,15 +37,21 @@
},
{
"id": "499F3572-E655-4DEE-953B-5F26BF0191D7",
"use": "block_used_n_times",
"template": "Long String: ${question}",
"description": "This is just a test for long string inputs.",
"use": "ai_question",
"template": "Ask Copilot: ${question}",
"description": "Experimental: AI outputs are inherently nondeterministic and may not be accurate. Use with caution and always review responses.",
"docPath": "/teachertool",
"params": [
{
"name": "question",
"type": "longString",
"paths": ["checks[0].blockCounts[0].blockId"]
"paths": ["checks[0].question"]
},
{
"name": "shareid",
"type": "system",
"key": "SHARE_ID",
"paths": ["checks[0].shareId"]
}
]
},
Expand Down
12 changes: 12 additions & 0 deletions common-docs/teachertool/test/validator-plans-shared.json
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,18 @@
"count": 0
}
]
},
{
".desc": "Ask Copilot a question",
"name": "ai_question",
"threshold": -1,
"checks": [
{
"validator": "aiQuestion",
"question": "",
"shareId": ""
}
]
}
]
}

0 comments on commit cf4f60b

Please sign in to comment.