Snippet Question

Each snippet can have a couple of related-questions. They could be used to test whether raters focused on their tasks or just randomly gave ratings.

###1. Snippet question types

Snippet questions can be one of four types:

• SINGLE_CHOICE: Raters can select only one option.

• MULTIPLE_CHOICE: Raters can select more than one option.

• RATING: Raters select one option for each given attribute. For example, there are 2 attributes that are "Should they use singleton pattern?" and "Should they use prototype pattern?", and 2 options are "Yes" and "No".

• INPUT: Raters are asked to fill the answer.

###2. Adding snippet question

Snippet questions can be added in the rating editor interface. By click on the "Add question" button at the bottom of the page, a pop-up window will appear and show the adding snippet question form. Different question types can have different configurations:

###2.1. INPUT type

The input snippet questions are the simplest that each question only has the content, constraint, and "Hidden at first" checkbox. Beside the required constraint that enforces raters to answer this questions, the "Input is a number" checkbox marks that this question only accepts a number as its answer.

The "Hidden at first" option ensures that this question won't be displayed at the first time. It will only be displayed after raters already gave ratings and answered all other required questions.

###2.2. Choices type

Both single choice and multiple choices questions have the same set of configurations. They have all configurations similarly to input questions except the options configurations.

###2.3. Rating type

Rating question is similar to choices questions except that there is a place to add attributes.

###3. Reordering snippet questions

Sometimes, the order of questions matters. Administrators can easily reorder questions of a snippet by dragging and dropping.