Description
Problem
Some practice exercises have scenarios "to allow for selectively including/excluding test cases based on a property of the test case". We currently have not defined how these are treated within the PHP track.
At the time of writing, these are the scenarios in use within exercises:
~/.cache/exercism/configlet/problem-specifications$ fgrep -H -e 'scenario' -r --include='*.json' exercises/
exercises/bank-account/canonical-data.json: "scenarios": ["concurrent"],
exercises/dnd-character/canonical-data.json: "scenarios": ["random"],
exercises/high-scores/canonical-data.json: "scenarios": ["immutable"],
exercises/protein-translation/canonical-data.json: "scenarios": ["input-validation", "runtime-validation"],
exercises/parallel-letter-frequency/canonical-data.json: "scenarios": ["unicode"],
exercises/parallel-letter-frequency/canonical-data.json: "scenarios": ["concurrent"],
exercises/reverse-string/canonical-data.json: "scenarios": ["unicode"],
exercises/gigasecond/canonical-data.json: "scenarios": ["date"],
exercises/gigasecond/canonical-data.json: "scenarios": ["datetime"],
exercises/gigasecond/canonical-data.json: "scenarios": ["immutable"],
exercises/diffie-hellman/canonical-data.json: "scenarios": ["random"],
exercises/diffie-hellman/canonical-data.json: "scenarios": ["immutable"],
exercises/forth/canonical-data.json: "scenarios": ["local-scope"],
exercises/armstrong-numbers/canonical-data.json: "scenarios": ["big-integer"],
exercises/simple-cipher/canonical-data.json: "scenarios": ["random"],
exercises/triangle/canonical-data.json: "scenarios": ["floating-point"],
exercises/anagram/canonical-data.json: "scenarios": ["unicode"],
Suggestion
Scenarios can be grouped into categories based on how we want to treat them:
- Never implemented: We do not implement those test cases at all.
- Always implemented: We always include those test cases as required tests.
- Optionally implemented: Students should be able to opt-in to implement their solution so they pass those test cases in addition to the required tests.
We decide, which scenarios we want to treat as "optional", "always" or "never". I think, this a way to use the categories:
- Never implemented:
concurrent
,library-test
, any unknown scenario - Always implemented:
date
,datetime
,floating-point
,local-scope
- Optionally implemented:
big-integer
,immutable
,input-validation
,random
,runtime-validation
,unicode
Based on these categories, test generators can simply skip over "never" scenarios and treat "always" scenarios like "no scenario".
For the "optional" ones I suggest using PHP constants in students files (define('BIG_INTEGER', 1);
) and inside the test methods if (!defined('BIG_INTEGER') || constant('BIG_INTEGER' !== 1)) { $this->markTestSkipped(); }
. Using the functions to create / read the constants avoids the problems that arise, when including the students file into a class method (as we do in the test class).
The constants provide a means to easily pre-define them in the students stub file:
[...]
// Change this constant value to 1 if you want to test your solution using Unicode strings:
define('UNICODE', 0);
class ExerciseName
{
[...]
}