Snippet

Snippets will be displayed in the rating editor page in which administrators can monitor the ratings and import/export snippets while raters can give ratings for a list of snippets. Both raters and administrators can change from vertical to horizontal layout back and forth with a toggle button above the language selector.

Vertical layout

Horizontal layout

However, some of the functionalities will only be displayed to administrators but not to raters. On the other hand, administrators can neither give a rating nor change the ratings and answers of raters.

1. Interface for raters

The interface for raters is quite simple that raters can only choose highlighter (how snippet will be display), give ratings, answer snippet questions, and complete the survey.

If the local raters management is used, raters will be redirected to a goodbye page. Otherwise, they will be redirected to Prolific to complete their survey.

2. Interface for administrators

The interface for administrators is more complicated than for raters. There are a couple of buttons next to the Snippets text that allows administrators to add a snippet from github, add an attention check snippet, delete snippet, and import/export snippets of this dataset.

2.1. Add a snippet from Github

A dialog will be displayed to specify which part of a file in Github you would like to add as a snippet to this dataset.

2.2. Add an attention check snippet

This feature will clone the current snippet and mark it for attention check. The tool also assign the attention check snippet a random value as the correct rating. A piece of text would be displayed to raters to guide them to select the correct rating. If a rater selects an incorrect rating for a given attention check snippet, that means they did not pay attention when taking the survey and administrators should investigate and consider their submission.

2.3. Delete snippet

The feature allows administrators to delete the current snippet. All snippet questions and their solutions and the snippet rating will be deleted as well.

2.4. Import / Export

The tool supports administrators to upload a couple of snippet files or a zip file of all snippets to a dataset. Old snippets will be deleted as well as their ratings, questions, and solutions of raters.

Source code files can be imported together with a metadata files in json format to create snippet questions. Administrators only need to select both source code files and metadata files to import to our application. In this case, for each source code file, there is a metadata json file that has the same name as the source code file but different extension part. For example, the source code filename is "memory.cc", the metadata filename should be "memory.cc.json".

The structure of the metadata json file should follow this schema:

json { "path": "<source-code-file-name>", "fromLine": 1, "toLine": 100, "priority": 2, "correctRating": null, "attentionCheck": false, "questions": [ { "content": "<question-content>", "type": "SINGLE_CHOICE", "answer": { "options": [ "option 1", "option 2" ], "attributes": [], "correctChoices": [] }, "priority": 1, "hidden": false } ] }

• path: The source code filename (e.g. memory.cc)
• fromLine: From which line number the highlighter should display.
• toLine: To which line number the highlighter should display.
• priority: The order to determine which snippet should be display first. They will be sorted in ascending order.
• correctRating: If this snippet is an attention check (attentionCheck = true), correctRating will be a number that is a correct answer.
• attentionCheck: whether this snippet is an attention check.
• questions: list of snippet questions.
• questions[i].content: The content of the snippet question (mandatory).
• questions[i].type: The type of the snippet question (mandatory).
• questions[i].priority: The priority of question (not mandatory).
• questions[i].hidden: whether this question is a hidden question (not mandatory).
• questions[i].answer.options: The list of options for raters to choose from (mandatory if this is a choice-question).
• questions[i].answer.correctChoices: The list of indexes of correct options (not mandatory).
• questions[i].answer.attributes: The list of attributes of a RATING question (mandatory if this is a rating question).

Administrators can also export all snippets to local disk.