Skip to content

Commit

Permalink
Add catalog and validator plans for ai questions. (#9950) (#9951)
Browse files Browse the repository at this point in the history
The implementation is not included here, but the docs changes are still needed to have the criteria appear in uploaded targets.

Of note: this does introduce the idea of a "system" parameter, which is an input to the validator plan which can change based on context, but which the user doesn't really need to specify in the crtitera template. In this case, that's the share id. The plan is for the teacher tool to automatically populate system parameters based on the provided key.
  • Loading branch information
thsparks committed Apr 8, 2024
1 parent dbb3c7d commit 777dcd9
Show file tree
Hide file tree
Showing 2 changed files with 22 additions and 4 deletions.
14 changes: 10 additions & 4 deletions common-docs/teachertool/test/catalog-shared.json
Original file line number Diff line number Diff line change
Expand Up @@ -37,15 +37,21 @@
},
{
"id": "499F3572-E655-4DEE-953B-5F26BF0191D7",
"use": "block_used_n_times",
"template": "Long String: ${question}",
"description": "This is just a test for long string inputs.",
"use": "ai_question",
"template": "Ask Copilot: ${question}",
"description": "Experimental: AI outputs are inherently nondeterministic and may not be accurate. Use with caution and always review responses.",
"docPath": "/teachertool",
"params": [
{
"name": "question",
"type": "longString",
"paths": ["checks[0].blockCounts[0].blockId"]
"paths": ["checks[0].question"]
},
{
"name": "shareid",
"type": "system",
"key": "SHARE_ID",
"paths": ["checks[0].shareId"]
}
]
},
Expand Down
12 changes: 12 additions & 0 deletions common-docs/teachertool/test/validator-plans-shared.json
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,18 @@
"count": 0
}
]
},
{
".desc": "Ask Copilot a question",
"name": "ai_question",
"threshold": -1,
"checks": [
{
"validator": "aiQuestion",
"question": "",
"shareId": ""
}
]
}
]
}

0 comments on commit 777dcd9

Please sign in to comment.