programming errors are hard to debug

[mabraham]

I see that e.g. #2087 did work to improve things here, but I think I have a case where I'm stuck and reframe isn't helping me understand my problem.

Specifically, I get skipping test 'PerformanceTest': test has one or more undefined parameters and adding -v to the commandline doesn't help by giving me a stack trace.

With either spack version 3.8.1, git 3.8.1 or git master HEAD, I get output like

$ ~/git/reframe/bin/reframe -v -C settings.py -c run-benchmarks.py -r --keep-stage-files -s stage -o output --dont-restage --system amd-devcloud --module-path $HOME/progs/modulefiles
[ReFrame Setup]
  version:           3.9.0-dev.3+4ab88beb
  command:           '/home/mabraham/git/reframe/bin/reframe -v -C settings.py -c run-benchmarks.py -r --keep-stage-files -s stage -o output --dont-restage --system amd-devcloud --module-path /home/mabraham/progs/modulefiles'
  launched by:       mabraham@ts1-sjc2-03-ge
  working directory: '/home/mabraham/benchmarks/hipsycl-vs-hip'
  settings file:     'settings.py'
  check search path: '/home/mabraham/benchmarks/hipsycl-vs-hip/run-benchmarks.py'
  stage directory:   '/home/mabraham/benchmarks/hipsycl-vs-hip/stage'
  output directory:  '/home/mabraham/benchmarks/hipsycl-vs-hip/output'

/home/mabraham/git/reframe/bin/reframe: skipping test 'PerformanceTest': test has one or more undefined parameters
Loaded 1 test(s)
Generated 0 test case(s)
Filtering test cases(s) by name: 0 remaining
Filtering test cases(s) by tags: 0 remaining
Filtering test cases(s) by other attributes: 0 remaining
Final number of test cases: 0
[==========] Running 0 check(s)
[==========] Started on Tue Oct 26 07:22:56 2021 

[----------] waiting for spawned checks to finish
[----------] all spawned checks have finished

[  PASSED  ] Ran 0/0 test case(s) from 0 check(s) (0 failure(s), 0 skipped)
[==========] Finished on Tue Oct 26 07:22:56 2021 
Run report saved in '/home/mabraham/.reframe/reports/run-report.json'
Log file(s) saved in '/tmp/rfm-nroxowcm.log'

The log file is attached rfm-nroxowcm.log

[mabraham]

I also tried -v -v -v but no extra help!

[jjotero]

Your PerformanceTest has at least one parameter that does not define any values, as the error message says. You can print the parameter space of your test right after you define the class doing print(PerformanceTest.param_space). There you'll find which parameter(s) don't have any values.

[mabraham]

Great! Now I know which parameter to investigate further.

[vkarak]

Just to comment a bit on tracking programming errors. We're aware of this shortcoming, but it's not always straightforward for the framework to track all the errors back to their user source line. There is a lot happening at a later stage, while instantiating the test, and giving a stack trace there would be more confusing, since it wouldn't get you back to the source of the problem. Nonetheless, whenever the framework can track the programming error back to the offending user line, it will print that.