Skip to content

Commit

Permalink
Merge pull request #129 from skulonen/830-delayed-feedback
Browse files Browse the repository at this point in the history
Delayed feedback (a-plus-rst-tools updates)
  • Loading branch information
markkuriekkinen committed Oct 14, 2021
2 parents eee83a3 + edaa37b commit 3cec1fa
Show file tree
Hide file tree
Showing 7 changed files with 268 additions and 6 deletions.
61 changes: 58 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,13 @@ skip_language_inconsistencies = False # for debugging multilanguage courses
allow_assistant_viewing = True # May assistants view submissions?
allow_assistant_grading = False # May assistants grade submissions?

# Default rules for revealing submission feedback and model solutions.
# Can be overridden in module aplusmeta directive, or in questionnaire/submit
# directives.
# See instructions in chapter 'Defining reveal rules'.
reveal_submission_feedback = 'immediate'
reveal_model_solutions = 'deadline'

# List of JavaScript and CSS URLs for the A+ head URLs course setting.
# A+ adds these to every course page.
course_head_urls = [
Expand Down Expand Up @@ -265,7 +272,7 @@ max points `50` and difficulty `A`. If not set in the directive arguments, the m
the sum of the question points. Setting the difficulty is optional and it can be set
even if the max points aren't defined in the argument. The questionnaire directive accepts the following options:

* `submissions`: max submissions
* `submissions`: max submissions (set to 0 to remove submission limit)
* `points-to-pass`: points to pass
* `feedback`: If set, assumes the defaults for a feedback questionnaire
* `title`: exercise title
Expand All @@ -286,14 +293,19 @@ even if the max points aren't defined in the argument. The questionnaire directi
Note that A+ has a separate feature for showing exercise model solutions after
the deadline. Can be set to true or false. The default value can be set in
index.rst with the field `questionnaire-default-reveal-model-at-max-submissions`.
By default false.
By default false. Don't use this option together with `reveal-model-solutions`.
* `show-model`: Students may open the model solution in A+ after the module
deadline. Can be set to true or false. The default value can be set in
index.rst with the field `questionnaire-default-show-model`. By default true.
* `allow-assistant-viewing`: Allows assistants to view the submissions of the students.
Can be set to true or false. Overrides any options set in the conf.py or config.yaml files.
* `allow-assistant-grading`: Allows assistants to grade the submissions of the students.
Can be set to true or false. Overrides any options set in the conf.py or config.yaml files.
* `reveal-submission-feedback`: rule for revealing submission feedback. See [instructions](#defining-reveal-rules).
* `reveal-model-solutions`: rule for revealing model solutions. See [instructions](#defining-reveal-rules).
* `grading-mode`: which submission determines the final score for this exercise ("best" or "last"). Defaults to "best".
If delayed feedback is used (`reveal-submission-feedback` is set to something other than "immediate"), defaults to
"last".

The contents of the questionnaire directive define the questions and possible
instructions to students.
Expand Down Expand Up @@ -548,7 +560,7 @@ body of the submit directive will be prioritized.
It accepts the following options:

* `config`: path to the YAML configuration file
* `submissions`: max submissions
* `submissions`: max submissions (set to 0 to remove submission limit)
* `points-to-pass`: points to pass (default zero)
* `class`: CSS class(es)
* `title`: exercise title
Expand All @@ -562,6 +574,11 @@ It accepts the following options:
Can be set to true or false. Overrides any options set in the conf.py or config.yaml files.
* `allow-assistant-grading`: Allows assistants to grade the submissions of the students.
Can be set to true or false. Overrides any options set in the conf.py or config.yaml files.
* `reveal-submission-feedback`: rule for revealing submission feedback. See [instructions](#defining-reveal-rules).
* `reveal-model-solutions`: rule for revealing model solutions. See [instructions](#defining-reveal-rules).
* `grading-mode`: which submission determines the final score for this exercise ("best" or "last"). Defaults to "best".
If delayed feedback is used (`reveal-submission-feedback` is set to something other than "immediate"), defaults to
"last".
* `quiz`: If set, the exercise feedback will take the place of the exercise instructions.
This makes sense for questionnaires since their feedback contains the submission form.
In RST, you would usually define questionnaires with the questionnaire directive,
Expand Down Expand Up @@ -634,6 +651,8 @@ The aplusmeta directive does not have any content and it accepts the following o
* `hidden`: If set, set status hidden for the module or chapter
* `points-to-pass`: module points to pass
* `introduction`: module introduction as an HTML string
* `reveal-submission-feedback`: default rule for revealing submission feedback. Can be overridden per exercise. See [instructions](#defining-reveal-rules).
* `reveal-model-solutions`: default rule for revealing model solutions. Can be overridden per exercise. See [instructions](#defining-reveal-rules).

Example module index.rst file:

Expand Down Expand Up @@ -1186,3 +1205,39 @@ There are 6 possible statuses for exercises:
* enrollment: Questions for students when they enroll to a course.
* enrollment_ext: Same as enrollment but for external students.
* maintenance: Hides the exercise description and prevents submissions.

### Defining reveal rules

Rules for revealing submission feedback (`reveal-submission-feedback`) and model solutions (`reveal-model-solutions`)
can be defined and overridden on multiple levels:

* course level, in conf.py
* module level, in the `aplusmeta` directive
* exercise level, in the `questionnaire`/`submit` directive

The reveal rules are defined by providing the name of a reveal mode. Some of the modes also accept arguments after the
mode name. The reveal modes are:

* manual: Never revealed, unless a teacher manually reveals it in A+ exercise settings.
* immediate: Always revealed. **This is the default setting for revealing submission feedback.**
* time: Revealed at a specific time. The reveal date and time must be provided as an argument, in the format
`YYYY-MM-DD [hh[:mm[:ss]]]` or `DD.MM.YYYY [hh[:mm[:ss]]]`. Examples:
```
:reveal_submission_feedback: time 2020-01-16
:reveal_submission_feedback: time 2020-01-16 16
:reveal_submission_feedback: time 16.01.2020 16:00
:reveal_submission_feedback: time 16.01.2020 16:00:00
```
* deadline: Revealed after the exercise deadline, and the possible deadline extension granted to the student. **This is
the default setting for revealing model solutions.** An additional delay can optionally be provided as an argument, in
the format `+<number><unit>`, where `unit` is 'd' (days), 'h' (hours) or 'm'/'min' (minutes). Examples:
```
:reveal_submission_feedback: deadline
:reveal_submission_feedback: deadline +1d
:reveal_submission_feedback: deadline +2h
:reveal_submission_feedback: deadline +30m
:reveal_submission_feedback: deadline +30min
```
* deadline_all: Revealed after the exercise deadline, and all deadline extensions granted to any student on the course.
An additional delay can optionally be provided as an argument. See instructions above.
* completion: Revealed after the student has used all submissions or achieved full points from the exercise.
2 changes: 2 additions & 0 deletions aplus_setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@ def setup(app):
app.add_config_value('acos_submit_base_url', 'http://172.21.0.2:3000', 'html')
app.add_config_value('enable_doc_link_multilang_suffix_correction', False, 'html')
app.add_config_value('enable_ref_link_multilang_suffix_correction', False, 'html')
app.add_config_value('reveal_submission_feedback', None, 'html')
app.add_config_value('reveal_model_solutions', None, 'html')

# Connect configuration generation to events.
app.connect('builder-inited', toc_config.prepare)
Expand Down
12 changes: 12 additions & 0 deletions directives/meta.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
from sphinx.errors import SphinxError

from aplus_nodes import aplusmeta
from lib.revealrule import parse_reveal_rule

class AplusMeta(Directive):
''' Injects document meta data for A+ configuration. '''
Expand All @@ -21,6 +22,8 @@ class AplusMeta(Directive):
'hidden': directives.flag,
'points-to-pass': directives.nonnegative_int, # set points to pass for modules
'introduction': directives.unchanged, # module introduction HTML
'reveal-submission-feedback': directives.unchanged,
'reveal-model-solutions': directives.unchanged,
}

# Valid date formats are the same as in function parse_date() in
Expand All @@ -39,6 +42,12 @@ class AplusMeta(Directive):
'late-time',
}

# Keys in option_spec which are parsed as reveal rules
reveal_rules = {
'reveal-submission-feedback',
'reveal-model-solutions',
}

def run(self):
env = self.state.document.settings.env
substitutions = env.config.aplusmeta_substitutions
Expand All @@ -59,6 +68,9 @@ def run(self):
self.options[opt] = substitutions[value]
if opt in AplusMeta.date_format_required:
self.validate_time(opt, self.options[opt], old_value)
if opt in AplusMeta.reveal_rules:
source, line = self.state_machine.get_source_and_line(self.lineno)
self.options[opt] = parse_reveal_rule(value, source, line, opt)

return [aplusmeta(options=self.options)]

Expand Down
23 changes: 23 additions & 0 deletions directives/questionnaire.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
import lib.translations as translations
import lib.yaml_writer as yaml_writer
from directives.abstract_exercise import AbstractExercise, choice_truefalse, str_to_bool
from lib.revealrule import parse_reveal_rule


logger = logging.getLogger(__name__)
Expand Down Expand Up @@ -42,6 +43,9 @@ class Questionnaire(AbstractExercise):
'reveal-model-at-max-submissions': choice_truefalse,
'allow-assistant-viewing': choice_truefalse,
'allow-assistant-grading': choice_truefalse,
'reveal-submission-feedback': directives.unchanged,
'reveal-model-solutions': directives.unchanged,
'grading-mode': directives.unchanged,
}

def run(self):
Expand Down Expand Up @@ -195,6 +199,25 @@ def run(self):
else:
data['title|i18n'] = translations.opt('feedback') if is_feedback else translations.opt('exercise', postfix=" {}".format(key))

source, line = self.state_machine.get_source_and_line(self.lineno)
if 'reveal-submission-feedback' in self.options:
data['reveal_submission_feedback'] = parse_reveal_rule(
self.options['reveal-submission-feedback'],
source,
line,
'reveal-submission-feedback',
)
if 'reveal-model-solutions' in self.options:
data['reveal_model_solutions'] = parse_reveal_rule(
self.options['reveal-model-solutions'],
source,
line,
'reveal-model-solutions',
)

if 'grading-mode' in self.options:
data['grading_mode'] = self.options['grading-mode']

if not 'no-override' in self.options and category in override:
data.update(override[category])
if 'url' in data:
Expand Down
23 changes: 23 additions & 0 deletions directives/submit.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
import lib.translations as translations
import lib.yaml_writer as yaml_writer
from directives.abstract_exercise import AbstractExercise, choice_truefalse
from lib.revealrule import parse_reveal_rule


class SubmitForm(AbstractExercise):
Expand All @@ -36,6 +37,9 @@ class SubmitForm(AbstractExercise):
'status': directives.unchanged,
'allow-assistant-viewing': choice_truefalse,
'allow-assistant-grading': choice_truefalse,
'reveal-submission-feedback': directives.unchanged,
'reveal-model-solutions': directives.unchanged,
'grading-mode': directives.unchanged,
}

def run(self):
Expand Down Expand Up @@ -151,6 +155,25 @@ def run(self):

data.setdefault('status', self.options.get('status', 'unlisted'))

source, line = self.state_machine.get_source_and_line(self.lineno)
if 'reveal-submission-feedback' in self.options:
data['reveal_submission_feedback'] = parse_reveal_rule(
self.options['reveal-submission-feedback'],
source,
line,
'reveal-submission-feedback',
)
if 'reveal-model-solutions' in self.options:
data['reveal_model_solutions'] = parse_reveal_rule(
self.options['reveal-model-solutions'],
source,
line,
'reveal-model-solutions',
)

if 'grading-mode' in self.options:
data['grading_mode'] = self.options['grading-mode']

if category in override:
data.update(override[category])
if 'url' in data:
Expand Down
108 changes: 108 additions & 0 deletions lib/revealrule.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
import re
from typing import Any, Dict, Optional

from sphinx.errors import SphinxError


# Valid date formats are the same as in function parse_date() in
# toc_config.py:
# 1. 'YYYY-MM-DD [hh[:mm[:ss]]]'
# 2. 'DD.MM.YYYY [hh[:mm[:ss]]]'
date_format = re.compile(
r"^(\d\d\d\d-\d\d-\d\d|\d\d.\d\d.\d\d\d\d)" # YYYY-MM-DD or DD.MM.YYYY
"( \d\d(:\d\d(:\d\d)?)?)?$") # [hh[:mm[:ss]]]


# A delay string consists of 3 parts:
# 1. Plus sign
# 2. Number
# 3. Unit: [d]ay, [h]our, [m]inute/[min]ute
# With optionally spaces between the parts
delay_format = re.compile(
r"^\+ ?(?P<number>\d+) ?(?P<unit>d|h|m|min)$"
)


def parse_reveal_rule(
rule_str: Optional[str],
source: Optional[str],
line: Optional[int],
opt: Optional[str]
) -> Optional[Dict[str, Any]]:
def reveal_rule_error(msg: str):
base_msg_parts = []
if source is not None:
base_msg_parts.append(source)
if line is not None:
base_msg_parts.append(f"line {line}")
if opt is not None:
base_msg_parts.append(f"option '{opt}'")
base_msg = ", ".join(base_msg_parts) + ":\n"
raise SphinxError(base_msg + msg)

if rule_str is None:
return None

rule_str = rule_str.strip()
space_index = rule_str.find(' ')
if space_index > -1:
trigger = rule_str[:space_index]
argument = rule_str[(space_index + 1):].strip()
else:
trigger = rule_str
argument = None

result: Dict[str, Any] = {'trigger': trigger}

if trigger in ['immediate', 'manual', 'completion']:
if argument is not None:
reveal_rule_error(
"Unexpected argument in reveal rule. When using the 'manual', 'immediate' or\n"
"'completion' mode, no arguments are expected after the mode name.\n"
)

elif trigger == 'time':
if argument is None:
reveal_rule_error(
"Reveal time was not provided. When using the 'time' reveal mode, a time must be\n"
"provided after the mode name.\n"
)
if date_format.match(argument):
result['time'] = argument
else:
reveal_rule_error(
"Reveal time was formatted incorrectly. When using the 'time' reveal mode, use\n"
"one of the following formats:\n"
"1. 'YYYY-MM-DD [hh[:mm[:ss]]]', e.g., '2020-01-16 16:00'\n"
"2. 'DD.MM.YYYY [hh[:mm[:ss]]]', e.g., '16.01.2020 16:00'\n"
)

elif trigger in ['deadline', 'deadline_all']:
if argument is not None:
match = delay_format.match(argument)
if match:
number = int(match.group('number'))
unit = match.group('unit')
if unit == 'd':
minutes = 24 * 60 * number
elif unit == 'h':
minutes = 60 * number
else:
minutes = number
result['delay_minutes'] = minutes
else:
reveal_rule_error(
"Delay was formatted incorrectly. When using the 'deadline' or 'deadline_all'\n"
"reveal mode, a delay can optionally be provided after the mode name. Format the\n"
"delay like this:\n"
"'+<number><unit>', where unit is 'd' (days), 'h' (hours) or 'm'/'min' (minutes)\n"
"e.g., '+1d', '+2h', '+30m', '+30min'\n"
)

else:
reveal_rule_error(
"Unexpected reveal mode. Supported modes are 'manual', 'immediate', 'time',\n"
"'deadline', 'deadline_all' and 'completion'.\n"
)

return result

0 comments on commit 3cec1fa

Please sign in to comment.