New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests fail on viz module #1282

Closed
skoudoro opened this Issue Jun 28, 2017 · 12 comments

Comments

Projects
None yet
5 participants
@skoudoro
Member

skoudoro commented Jun 28, 2017

As you can see below, some tests do not pass. Any idea @MarcCote @ranveeraggarwal @Garyfallidis @dmreagan ?

 ======================================================================
ERROR: dipy.viz.tests.test_fvtk.test_fvtk_functions
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/koudoro/anaconda/envs/dipy_dev_3/lib/python3.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/Users/koudoro/anaconda/envs/dipy_dev_3/lib/python3.6/site-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/Users/koudoro/anaconda/envs/dipy_dev_3/lib/python3.6/site-packages/numpy/testing/decorators.py", line 147, in skipper_func
    return f(*args, **kwargs)
  File "/Users/koudoro/Software/dipy/dipy/testing/decorators.py", line 68, in test_with_xvfb
    my_test(*args, **kwargs)
  File "/Users/koudoro/Software/dipy/dipy/viz/tests/test_fvtk.py", line 51, in test_fvtk_functions
    v = fvtk.volume(vol)
  File "/Users/koudoro/Software/dipy/dipy/viz/fvtk.py", line 505, in volume
    raise ValueError("VolumeTextureMapper2D is not available in your "
ValueError: VolumeTextureMapper2D is not available in your version of VTK

======================================================================
FAIL: dipy.viz.tests.test_ui.test_ui_disk_slider_2d
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/koudoro/anaconda/envs/dipy_dev_3/lib/python3.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/Users/koudoro/anaconda/envs/dipy_dev_3/lib/python3.6/site-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/Users/koudoro/anaconda/envs/dipy_dev_3/lib/python3.6/site-packages/numpy/testing/decorators.py", line 147, in skipper_func
    return f(*args, **kwargs)
  File "/Users/koudoro/Software/dipy/dipy/testing/decorators.py", line 68, in test_with_xvfb
    my_test(*args, **kwargs)
  File "/Users/koudoro/Software/dipy/dipy/viz/tests/test_ui.py", line 314, in test_ui_disk_slider_2d
    event_counter.check_counts(expected)
  File "/Users/koudoro/Software/dipy/dipy/viz/tests/test_ui.py", line 81, in check_counts
    err_msg=msg.format(event))
  File "/Users/koudoro/anaconda/envs/dipy_dev_3/lib/python3.6/site-packages/numpy/testing/utils.py", line 425, in assert_equal
    raise AssertionError(msg)
AssertionError:
Items are not equal: Wrong count for 'LeftButtonPressEvent'.
 ACTUAL: 3
 DESIRED: 6

======================================================================
FAIL: dipy.viz.tests.test_ui.test_ui_textbox
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/koudoro/anaconda/envs/dipy_dev_3/lib/python3.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/Users/koudoro/anaconda/envs/dipy_dev_3/lib/python3.6/site-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/Users/koudoro/anaconda/envs/dipy_dev_3/lib/python3.6/site-packages/numpy/testing/decorators.py", line 147, in skipper_func
    return f(*args, **kwargs)
  File "/Users/koudoro/Software/dipy/dipy/testing/decorators.py", line 68, in test_with_xvfb
    my_test(*args, **kwargs)
  File "/Users/koudoro/Software/dipy/dipy/viz/tests/test_ui.py", line 215, in test_ui_textbox
    event_counter.check_counts(expected)
  File "/Users/koudoro/Software/dipy/dipy/viz/tests/test_ui.py", line 81, in check_counts
    err_msg=msg.format(event))
  File "/Users/koudoro/anaconda/envs/dipy_dev_3/lib/python3.6/site-packages/numpy/testing/utils.py", line 425, in assert_equal
    raise AssertionError(msg)
AssertionError:
Items are not equal: Wrong count for 'LeftButtonPressEvent'.
 ACTUAL: 8
 DESIRED: 7

======================================================================
FAIL: dipy.viz.tests.test_window.test_order_transparent
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/koudoro/anaconda/envs/dipy_dev_3/lib/python3.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/Users/koudoro/anaconda/envs/dipy_dev_3/lib/python3.6/site-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/Users/koudoro/anaconda/envs/dipy_dev_3/lib/python3.6/site-packages/numpy/testing/decorators.py", line 147, in skipper_func
    return f(*args, **kwargs)
  File "/Users/koudoro/Software/dipy/dipy/testing/decorators.py", line 68, in test_with_xvfb
    my_test(*args, **kwargs)
  File "/Users/koudoro/Software/dipy/dipy/viz/tests/test_window.py", line 224, in test_order_transparent
    npt.assert_equal(arr[150, 150][0] > arr[150, 150][1], True)
  File "/Users/koudoro/anaconda/envs/dipy_dev_3/lib/python3.6/site-packages/numpy/testing/utils.py", line 425, in assert_equal
    raise AssertionError(msg)
AssertionError:
Items are not equal:
 ACTUAL: False
 DESIRED: True 

@skoudoro skoudoro changed the title from Tests Failure on Viz module to Tests fail on viz module Jun 28, 2017

@ranveeraggarwal

This comment has been minimized.

Member

ranveeraggarwal commented Jun 28, 2017

Which OS are you on? Is it MacOS?

@skoudoro

This comment has been minimized.

Member

skoudoro commented Jun 28, 2017

Yes: environment : OS X -anaconda - python 3 - VTK 7.1 from clinicalgraphics

@Garyfallidis

This comment has been minimized.

Member

Garyfallidis commented Jun 28, 2017

@ranveeraggarwal

This comment has been minimized.

Member

ranveeraggarwal commented Jun 28, 2017

Even I had problems with OSX VTK 6.3.0 Python 3. We had suspected it might be because of the windowing manager. I will have to look at Windows though.

@dmreagan

This comment has been minimized.

Contributor

dmreagan commented Jul 21, 2017

I think I've traced the failure of test_ui_textbox to commit 2df29df. It looks like the test was re-recorded for that commit, and the new recording has a different number of events than are expected. I could fix the problem by either:

  1. Changing the test to expect the number of events in the new recording.
  2. Re-recording the test and setting the expected number to match.

It is probably best to do solution 2. The problem now is that I don't know what actions were actually performed and recorded. The TextBox2d is very simple, so I can guess that the test probably just clicks in the box and types a little bit. But the recording log is dominated by MouseEvents, making it difficult to figure out what is going on. You can play the recording and try to watch what happens, but it goes by too quickly since the playback doesn't happen at the same speed it was recorded. This problem makes it hard to trust solution 1, and it also makes it hard to do solution 2, since I can't know if I'm recording the same actions as before.

This lack of transparency in the recording strikes me as a problem that goes beyond this particular issue. Dipy relies on many tests like these, but we don't really know what they are testing. Could we require a description of each recording in the test definition so that future developers can duplicate them with confidence? Something like:

test_ui_textbox
1. Click inside the textbox
2. Type something
3. Hit backspace to erase it
4. Click outside the textbox
5. Click inside the textbox
6. Type something

Thoughts on this, @skoudoro , @ranveeraggarwal , @MarcCote , @Garyfallidis ?

@MarcCote

This comment has been minimized.

Contributor

MarcCote commented Aug 14, 2017

I agree with @dmreagan. We should add a detailed explanation of what was recorded. Regarding watching the recording, there must be a way of slowing it down in VTK but I don't know how (maybe by issuing VTK Events ourselves).

I don't have to do re-record the test at the moment. If someone wants to do it, it should pretty easy to do. Just call the test scripts directly with the name of the test and everything you do will be recorded in a file in dipy/data/files.

For instance: python dipy/viz/tests/test_ui.py test_ui_button_panel.

@dmreagan

This comment has been minimized.

Contributor

dmreagan commented Aug 15, 2017

Thanks, @MarcCote. I can record that test and take a look at the other failing tests as well.

@dmreagan

This comment has been minimized.

Contributor

dmreagan commented Aug 18, 2017

@MarcCote and @ranveeraggarwal, have you recorded tests on Windows 10 with Python 3? I can start the test and perform some actions, but when I close the window I see the following error:

Permission Error: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\dmreagan\\AppData\\Local\\Temp\\tmpm1x9w6uj\\recorded_events.log'

The recorded_events.log file gets written, but test_ui_textbox.log.gz and test_ui_textbox.pkl are not overwritten. I've tried running as both a standard user and an admin with the same results.

@ranveeraggarwal

This comment has been minimized.

Member

ranveeraggarwal commented Aug 18, 2017

@dmreagan I face this issue while running tests for 2D file menu - it needs to be fixed. My suspicion is that since I am using os.chdir, it's breaking the relative path.

In case you're not on the latest branch (with the FileSelectMenu2D), then I'm not sure why this is happening. Is it specific to Win10/Py3?

@Garyfallidis

This comment has been minimized.

Member

Garyfallidis commented Oct 27, 2017

The Viz test fails are still ongoing we need to resolve them asap. @ranveeraggarwal, @MarcCote, @dmreagan and @skoudoro
Travis is not complaining. But on personal machines we get errors.
Run nosetests in dipy/viz and see what you get.

@MarcCote

This comment has been minimized.

Contributor

MarcCote commented Oct 30, 2017

I think it has to do with Travis (Linux) using VTK5.8. So the tests were designed to pass for that particular environment. It means there is no guarantee they will pass on another VTK version and even worse there is no guarantee they will pass on a different OS.

I think the man culprit is MouseMoveEvent which behave differently from one system/VTK version to another. :-/

I don't know if it would be possible to alter the recorded files to replace every sequence of MouseMoveEvent with a kind of MouseJumpEvent (I just made that event up). That new event would be responsible to place the mouse at a giving position, i.e. where the mouse would have ended up.

The drawback with that is every drag events won't work as expected (e.g. a window that gets drag would jump from its initial position to its final without "seeing" the transition animation).

Otherwise, we could use a more recent version of VTK, I think it is easy to setup with conda, and hope the VTK behavior is more stable across different environments.

@skoudoro

This comment has been minimized.

Member

skoudoro commented Nov 2, 2017

Many tests with MouseMoveEvent are still running really well on VTK 5.8 / 6.3 / 7.1, so I do not think MouseMoveEvent behaves differently from one system/VTK version to another.

Like you said before, maybe we just need to update some record files. Let see, what happens in PR #1374.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment