Description
The create-all-tests script accepts --verbose and --validate flags, intended to help in debugging when a test plan won't build for whatever reason. Unfortunately, output only seems to be logged when:
- A part of the process completes successfully (such as parsing a CSV file); and
- explicit errors are thrown.
The script does not, however, log actions it is about to take, nor which file/row/column is being processed. As such, when an unexpected problem occurs, such as that highlighted in #1244, the test plan author is left at a complete loss about where to start looking for the issue.
Sticking with that example, there is literally no output when running node scripts\create-all-tests --verbose --validate --testplan=hang-sample:
- There are no successful actions for the script to report because it hangs almost immediately.
- There is no error thrown, because the regexp parser is technically doing its job albeit in a way that causes the script to hang.
As a result, it took me over an hour to track down exactly where the failure was occuring. Consider that if the script had output something like the following instead, it would've been a lot more obvious:
>node scripts\create-all-tests --verbose --validate --testplan=hang-sample
Validating row 0 in data/tests.csv for test plan "hang-sample".
Processing tests.csv column: testId
Processing tests.csv column: title
Processing tests.csv column: presentationNumber
Processing tests.csv column: setupScript
Processing tests.csv column: instructions
Processing tests.csv column: assertions... it would've been immediately obvious that the script froze right after trying to validate the assertions column on that specific row of that specific file.
CC @howard-e