Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

question id support #9

Merged
merged 2 commits into from Oct 26, 2016
Merged

question id support #9

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
01e1359
question id support
aimichal Oct 26, 2016
1297983
better example question identifier
aimichal Oct 26, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
9 changes: 7 additions & 2 deletions common/src/main/scala/org/allenai/aristomini/model/ExamQuestion.scala
View file
Open in desktop
@@ -1,7 +1,12 @@
package org.allenai.aristomini.model

/** A question-answerkey pair */
case class ExamQuestion(question: MultipleChoiceQuestion, answerKey: String) {
/** An exam question.
*
* @param id an identifier for this question
* @param question an instance of MultipleChoiceQuestion representing this question
* @param answerKey the label of the correct answer
*/
case class ExamQuestion(id: String, question: MultipleChoiceQuestion, answerKey: String) {

/** Consider a candidate answer and decide if it's correct or not.
* @param candidate a candidate answer to a multiple choice question
Expand Down
1 change: 1 addition & 0 deletions common/src/test/scala/org/allenai/aristomini/model/ExamQuestionSpec.scala
View file
Open in desktop
Expand Up @@ -4,6 +4,7 @@ import org.scalatest.FunSuite

class ExamQuestionSpec extends FunSuite {
val dummyExamQuestion = ExamQuestion(
id = "foo bar 123",
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having spaces in identifiers can be annoying to deal with, so maybe set a good example by not having spaces in this sample ID?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I considered re-writing the "question number" logic to use the string identifiers but decided against it because of exactly the reason you arrived at :-)

question = MultipleChoiceQuestion(stem = "What is?", choices = Seq()),
answerKey = "A"
)
Expand Down
4 changes: 2 additions & 2 deletions evalui/src/main/scala/org/allenai/aristomini/evaluate/Evaluation.scala
View file
Open in desktop
Expand Up @@ -24,8 +24,8 @@ class Evaluation(exam: ExamMultipleChoice) {
// Exam questions to be answered
private val examQuestions: Seq[ExamQuestion] = exam.questions

// Answered provided by the solver, keyed on the question position in examQuestions. Initially
// empty.
// Answered provided by the solver, keyed on the question number (position in examQuestions).
// Initially empty.
private val solverAnswers: mutable.Map[Integer, SolverAnswer] =
scala.collection.mutable.Map[Integer, SolverAnswer]()

Expand Down
5 changes: 4 additions & 1 deletion evalui/src/main/scala/org/allenai/aristomini/evalui/ExamUI.scala
View file
Open in desktop
Expand Up @@ -115,7 +115,7 @@ object ExamUI {
private def questionTable(exam: ExamMultipleChoice, evaluation: Evaluation): Elem = {
<table border="1">
<tr>
<th>Question</th> <th>Key</th> <th>Answer</th> <th>Solver</th>
<th>Question ID</th> <th>Question</th> <th>Key</th> <th>Answer</th> <th>Solver</th>
</tr>{exam.questions.zipWithIndex.map {
case (examQuestion: ExamQuestion, questionNumber: Int) => {
val answerOpt = evaluation.solverAnswerForQuestion(questionNumber)
Expand All @@ -137,6 +137,9 @@ object ExamUI {
}

<tr>
<td>
{examQuestion.id}
</td>
<td>
{examQuestion.question.oneLine}
</td>
Expand Down
130 changes: 65 additions & 65 deletions evalui/src/universal/data/questions/AI2-8thGr-NDMC-Feb2016-Dev.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

586 changes: 293 additions & 293 deletions evalui/src/universal/data/questions/AI2-8thGr-NDMC-Feb2016-Train.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

168 changes: 84 additions & 84 deletions evalui/src/universal/data/questions/AI2-Elementary-NDMC-Feb2016-Dev.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

864 changes: 432 additions & 432 deletions evalui/src/universal/data/questions/AI2-Elementary-NDMC-Feb2016-Train.jsonl
View file
Open in desktop

Large diffs are not rendered by default.