Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Some unittests keep failing #50

Closed
130s opened this Issue Mar 5, 2014 · 9 comments

Comments

Projects
None yet
3 participants
Member

130s commented Mar 5, 2014

See for example the travis job in this PReq. The job fails due to the failure in running unittest. And so far as I've tried number of times, there are unit test suite in hironx_ros_bridge that fail no matter what. That is a separate issue of testcases that need to be fixed. Since this effects my work a lot I'll look into this soon.

@130s 130s added this to the Version 1.1 release milestone Mar 5, 2014

@130s 130s self-assigned this Mar 5, 2014

@130s 130s added the critical label Mar 5, 2014

Member

130s commented Apr 7, 2014

This is critical; In order to use unittests as a criteria for the regression test, I think it's essential to have testcases that don't fail under good condition. Otherwise we can't tell what causes the tests to fail.

Owner

k-okada commented Apr 7, 2014

we shold update .travas yaml to check both compile with hrpsys soucecode (for developpers) and hrpsys deb (for released code) code, see
https://github.com/start-jsk/rtmros_common/blob/master/.travis.yml
for example.

still, we'll have problem, I think [1] is the issue. and we'll use retry tag[2] to avoid the problem.

@130s 130s referenced this issue in start-jsk/rtmros_common Apr 14, 2014

Open

Pre-release test never got "SUCCESS" on Hydro #416

Member

130s commented Apr 30, 2014

we shold update .travas yaml to check both compile with hrpsys soucecode (for developpers) and hrpsys deb (for released code) code, see https://github.com/start-jsk/rtmros_common/blob/master/.travis.yml for example.

I see. When I originally opened this ticket I wasn't aware of it; I was only caring the following, but indeed there may be cases when the selection of hrpsys source becomes the issue:

  • testing with real robot
  • pre-release test and build on ROS buildfarm

(I suspect that the majority of hironx users would keep using .deb though.)

still, we'll have problem, I think [1] is the issue. and we'll use retry tag[2] to avoid the problem.

[1] https://code.google.com/p/hrpsys-base/issues/detail?id=192
[2] https://github.com/start-jsk/rtmros_common/pull/371/files

I think because we are aware other issues as follows in addition to those above, we just have to wait for them to be resolved first?

#81 (test code is for simulation ? real robot)
#82 (test code says OK even if it fail to connect RobotHardware)

Owner

k-okada commented Apr 30, 2014

we just have to wait for them to be resolved first?

do you mean that we have wait someone fixed (https://code.google.com/p/hrpsys-base/issues/detail?id=192) ?

Owner

k-okada commented Apr 30, 2014

currently hironx test code sometime did not pass as follows, maybe somewhere around acceleration test or something.

[addJointGroup] group name LARM is already installed
terminate called after throwing an instance of 'CORBA::OBJECT_NOT_EXIST'
testtest_hironx ... ok
testtest_hironx_ik ... ok

[ROSTEST]-----------------------------------------------------------------------

[hironx_ros_bridge.rosunit-test_hironx/test_fullbody_setJointAngles_Clear][passed]
[hironx_ros_bridge.rosunit-test_hironx/test_fullbody_setJointAngles_NoWait][passed]
[hironx_ros_bridge.rosunit-test_hironx/test_fullbody_setJointAngles_Wait][passed]
[hironx_ros_bridge.rosunit-test_hironx/test_fullbody_setJointAngles_minus][passed]
[hironx_ros_bridge.rosunit-test_hironx/test_goInitial][passed]
[hironx_ros_bridge.rosunit-test_hironx/test_rarm_setJointAnglesOfGroup_Override_Acceleration][passed]
[hironx_ros_bridge.rosunit-test_hironx/test_rarm_setJointAnglesOfGroup_minus][passed]
[hironx_ros_bridge.rosunit-test_hironx/test_rarm_setJointAngles_Clear][FAILURE]-
File "/usr/lib/python2.7/unittest/case.py", line 327, in run
    testMethod()
  File "/home/travis/ros/ws_rtmros_hironx/install/share/hironx_ros_bridge/test/test_hironx.py", line 427, in test_rarm_setJointAngles_Clear
    self.check_log_data(data, 6, (5 + clear_time[i]), [(-140+i*40/len(clear_time)),20], -100.0)
  File "/home/travis/ros/ws_rtmros_hironx/install/share/hironx_ros_bridge/test/test_hironx.py", line 148, in check_log_data
    self.assertTrue(flag)
  File "/home/travis/ros/ws_rtmros_hironx/install/share/hironx_ros_bridge/test/test_hironx.py", line 358, in assertTrue
    assert(a)
--------------------------------------------------------------------------------

[hironx_ros_bridge.rosunit-test_hironx/test_rarm_setJointAngles_NoWait][passed]
[hironx_ros_bridge.rosunit-test_hironx/test_rarm_setJointAngles_Wait][passed]
Contributor

emijah commented Apr 30, 2014

As a user, I'd just like to mention that for heavy users, who want to modify code, being able to compile from source is crucial.
Thank you for your work, and hope to contribute a bit more in the future.

Member

130s commented May 23, 2014

we just have to wait for them to be resolved first?

do you mean that we have wait someone fixed (fkanehiro/hrpsys-base#192) ?

No, I didn't mean that.

For https://code.google.com/p/hrpsys-base/issues/detail?id=192 in particular, because it looks like happens only once in 10 times or so, it's ignorably trivial isn't it (there are other failure situations that are due to many other random reasons that we don't necessarily investigate)?

Member

130s commented May 23, 2014

After a while unit tests are succeeding (eg. within this travis build 13 tests passed) and they seem to function as their purpose. However, some other issues that are linked from this ticket are not yet resolved. So I still keep this one open in case there might be underlining issues regarding tests.

Owner

k-okada commented May 24, 2014

(there are other failure situations that are due to many other random reasons that we don't necessarily investigate)?

I haven't checked the travis log but, test code is sometimes too strict . I'm not sure what's happens exactly but for example if travis is too crowded and there are no enough resources, then the sequencer did not respond as we expected. so sometimes we need to change test condition as
9bbd541
83886a4
ofcourse this increases the potential of passing wrong code.

Anyway we need to look more close on the travis log to identify the cause of failure

@k-okada k-okada closed this Jul 11, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment