New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mark skipped SLI tests as skipped, not success #713

Merged
merged 7 commits into from May 27, 2017

Conversation

Projects
None yet
3 participants
@hakonsbm
Contributor

hakonsbm commented Apr 18, 2017

This PR fixes #359, and contains the following

  • When NEST is compiled with MPI, tests that require NEST to be compiled without MPI are skipped with exit_test_gracefully instead of calling quit_i directly.
  • Making exit_test_gracefully mark the test as skipped, not success.
  • A new exit code, 200, has been introduced to mark skipped tests.
Show outdated Hide outdated sli/slistartup.cc
Show outdated Hide outdated testsuite/do_tests.sh.in
@gtrensch

This comment has been minimized.

Show comment
Hide comment
@gtrensch

gtrensch Apr 27, 2017

Contributor

I tested the build with and without MPI. Everything works as expected. 👍 . I would approve this PR, but please see also my comments.

Contributor

gtrensch commented Apr 27, 2017

I tested the build with and without MPI. Everything works as expected. 👍 . I would approve this PR, but please see also my comments.

@hakonsbm

This comment has been minimized.

Show comment
Hide comment
@hakonsbm

hakonsbm Apr 28, 2017

Contributor

@gtrensch I have now used named constants for the exit codes. Are they in the right place? And should I also add named constants for the other exit codes (scripterror, exception, etc)?

There are quite a few new changed files now, but most of the new changes are minor changes to tests. This is because in order to tell do_tests.sh why a test is skipped, I have introduced some new exit codes and a new SLI function, exit_test_gracefully_reason[/stringtype]. This way do_tests.sh can differentiate different reasons for skipping, and give associated comments.

Contributor

hakonsbm commented Apr 28, 2017

@gtrensch I have now used named constants for the exit codes. Are they in the right place? And should I also add named constants for the other exit codes (scripterror, exception, etc)?

There are quite a few new changed files now, but most of the new changes are minor changes to tests. This is because in order to tell do_tests.sh why a test is skipped, I have introduced some new exit codes and a new SLI function, exit_test_gracefully_reason[/stringtype]. This way do_tests.sh can differentiate different reasons for skipping, and give associated comments.

@gtrensch

This comment has been minimized.

Show comment
Hide comment
@gtrensch

gtrensch Apr 29, 2017

Contributor

@hakonsbm many thanks for the work ! I think the constants are in the right place. Replacing all the other exit codes would of course be a good idea. But I'm not sure if this PR is the right place to do this refactoring task.

Contributor

gtrensch commented Apr 29, 2017

@hakonsbm many thanks for the work ! I think the constants are in the right place. Replacing all the other exit codes would of course be a good idea. But I'm not sure if this PR is the right place to do this refactoring task.

@heplesser

@hakonsbm Thanks for your effort, this is an important step forward! I added some suggestions for further polishing.

Show outdated Hide outdated testsuite/mpitests/issue-600.sli
Show outdated Hide outdated testsuite/mpitests/test_gap_junctions_mpi.sli
Show outdated Hide outdated testsuite/regressiontests/ticket-772.sli
Show outdated Hide outdated lib/sli/unittest.sli
Show outdated Hide outdated sli/slistartup.cc
Show outdated Hide outdated sli/slistartup.h
@@ -249,7 +249,8 @@ run_test ()
param_script="$1"
param_success="$2"
param_failure="$3"
param_skipped="$3"
param_failure="$4"

This comment has been minimized.

@heplesser

heplesser May 24, 2017

Contributor

Please update the documentation of run_test() above to include param_skipped.

@heplesser

heplesser May 24, 2017

Contributor

Please update the documentation of run_test() above to include param_skipped.

Show outdated Hide outdated testsuite/do_tests.sh.in
Show outdated Hide outdated testsuite/do_tests.sh.in

@heplesser heplesser merged commit 15b63f2 into nest:master May 27, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@hakonsbm hakonsbm deleted the hakonsbm:skipped_sli_tests branch May 29, 2017

@hakonsbm hakonsbm referenced this pull request Jun 13, 2017

Merged

Update testsuite readme #752

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment