Computer-aided assessment of mathematics works in the following phases.
- Testing and
- Deploying questions.
- Adding questions to a quiz and use by students.
- Reporting and statistical analysis.
Questions in STACK can contain randomly generated elements. A student will be given a random variant of a question generated by a pseudo-random seed.
Questions are not randomly generated on the fly. Instead the teacher must pre-generate and "deploy" variants of a question. When a student attempts the question they will be given a random selection from the deployed variants.
We have chosen to add the extra deploy step, instead of generating on the fly, for a number of reasons.
- STACK runs all the question tests on each deployed version to establish each variant of the question is working. This aids quality control. By using question tests it is unlikely a student will be given a random version which does not work correctly.
- The teacher can decide if each deployed version appears to be of equal difficulty. The teacher can easily delete variants they do not like. Experience suggests there are unanticipated consequences of randomly generating questions. Really only statistics generated by real use can establish fairness amongst variants.
- The pre-generation combined with a cache helps minimise server load during the start of a large class which aids robustness of the whole experience. This helps STACK to optimize its use of the CAS for efficiency and reliability via a dynamic cache.
- If an author does not deploy any variants (not advised!) then the student gets any random variant.
- Questions that don't use randomization cannot be deployed explicitly. STACK automatically detects randomization.
- Variants are different if and only if the evaluated question note is different. Any number of instances can be requested and deployed but only one instance of each question note can be deployed. It is possible to deploy (n) variants in one go, but the system will give up if too many duplicate question notes are generated. The teacher is responsible to ensure question variants are different if and only if the question notes are different. The deployment management also allows specific variants to be dropped. You can also return to the question preview window and try a specific deployed variant.
- Deployment is not required for authors to test questions: an instance is generated on-the-fly.
How to deploy question variants
The deployment interface can be found on the top of the Testing page. This page also contains the list of currently deployed versions, and links to un-deploy a variant.
- You can click on the seed numbers to view a particular random variant. The testing page lists values of all the variables, displays the question and the worked solution. The testing page is a very efficient way to look at your random variants.
- When you deploy new variants STACK will run all the question tests. If a test fails, the generation process will stop with an error message, showing the failing test.
There is currently no way to loop systematically over all variants which have not been deployed yet.