New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PyNEST test result parsing confused by NEST output #762

Closed
heplesser opened this Issue Jun 21, 2017 · 3 comments

Comments

@heplesser
Contributor

heplesser commented Jun 21, 2017

The code parsing the output generated by nosetests to provide a progress indication during testing is confused by output generated by NEST, since the parser relies on lines of the form TestABC ... ok. If NEST input intervenes before ok (or any other result), no progress will be displayed, confusing the user (see also #677).

We thus should revise output parsing, but that should first happen once we have decided how to test in the future #761.

@gtrensch

This comment has been minimized.

Show comment
Hide comment
@gtrensch

gtrensch Jul 9, 2017

Contributor

@heplesser After some investigation I found that the NEST stderr output interferes with the nosetests messages. nosetests comes with several plugins, e.g to capture stdout. The capture-plugin is on per default. For stderr a similar plugin (nose-capturestderr) is available but needs to be installed manually: https://pypi.python.org/pypi/nose-capturestderr/1.2 I ran some tests and it works quite well in my local environment. If stderr is the only source of the corrupted nosetests output, it should be easily solvable.

nosetests takes the first comment line of a test for reporting. This also causes a strange output. E.g.
The models should behave as iaf_cond_* if a == 0., b == 0. and ... ok

Contributor

gtrensch commented Jul 9, 2017

@heplesser After some investigation I found that the NEST stderr output interferes with the nosetests messages. nosetests comes with several plugins, e.g to capture stdout. The capture-plugin is on per default. For stderr a similar plugin (nose-capturestderr) is available but needs to be installed manually: https://pypi.python.org/pypi/nose-capturestderr/1.2 I ran some tests and it works quite well in my local environment. If stderr is the only source of the corrupted nosetests output, it should be easily solvable.

nosetests takes the first comment line of a test for reporting. This also causes a strange output. E.g.
The models should behave as iaf_cond_* if a == 0., b == 0. and ... ok

@heplesser

This comment has been minimized.

Show comment
Hide comment
@heplesser

heplesser Jul 31, 2017

Contributor

@gtrensch Thanks for investigating! I would not want to make the correct operation of the NEST test suite dependent on the installation of an additional Pyton package. But that package is only a single file of code that appears to be quite directly based on similar code in nose. Could we integrate this capture technique directly in NEST, so that we can capture stderr without the need for installing additional third-party software?

I will schedule a discussion of #761 for the next Open Developer VC.

Contributor

heplesser commented Jul 31, 2017

@gtrensch Thanks for investigating! I would not want to make the correct operation of the NEST test suite dependent on the installation of an additional Pyton package. But that package is only a single file of code that appears to be quite directly based on similar code in nose. Could we integrate this capture technique directly in NEST, so that we can capture stderr without the need for installing additional third-party software?

I will schedule a discussion of #761 for the next Open Developer VC.

@gtrensch

This comment has been minimized.

Show comment
Hide comment
@gtrensch

gtrensch Jul 31, 2017

Contributor

@heplesser I see two aspects which I would handle seperatly:

  • the bug we want to fix for the 2.12.1 release which is due to missing nosetests functionality (stdout is captured while stderr is not)
  • the question whether we stay with nosetests and perhaps move to nose2 or swich to a different Python test framework

Addressing the question of the test framework we want to use, will very likely also solve the stderr-issue. Depending on the descision we make there might be some additional effort.

However, as you stated, we should carry out this discussion (#761) in the next Open Developer VC !

Contributor

gtrensch commented Jul 31, 2017

@heplesser I see two aspects which I would handle seperatly:

  • the bug we want to fix for the 2.12.1 release which is due to missing nosetests functionality (stdout is captured while stderr is not)
  • the question whether we stay with nosetests and perhaps move to nose2 or swich to a different Python test framework

Addressing the question of the test framework we want to use, will very likely also solve the stderr-issue. Depending on the descision we make there might be some additional effort.

However, as you stated, we should carry out this discussion (#761) in the next Open Developer VC !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment