LTP test is hanging

[bvpjenkins]

I am trying to run the LTP test. The test has been running for almost 11 hours and seems to have stopped while executing this command:

Running '/ghostcache/autotest/tmp/ltp/src/bin/runltp -q -l /ghostcache/autotest/results/default/ltp/results/ltp.log -C /ghostcache/autotest/results/default/ltp/debug/failcmdfile -d /ghostcache/autotest/tmp/ltp/src -S /ghostcache/autotest/tmp/site_tests/ltp/skipfile'

These are the relevant lines from the client0.log:

04:06:39 INFO | START ltp ltp timestamp=1396757199 localtime=Apr 06 04:06:39
04:06:39 DEBUG| Persistent state client._record_indent now set to 2
04:06:39 DEBUG| Persistent state client.unexpected_reboot now set to ('ltp', 'ltp')
04:06:39 DEBUG| Running 'mkdir -p /ghostcache/autotest/tmp/packages'
04:06:39 DEBUG| Running 'ls /ghostcache/autotest/tmp'
04:06:39 DEBUG| Running 'ls /ghostcache/autotest/tmp/packages'
04:06:39 DEBUG| Running 'ls /ghostcache/autotest/packages.checksum'
04:06:39 DEBUG| Running 'ls /ghostcache/autotest'
04:06:39 DEBUG| Running 'ls /ghostcache/autotest/packages.checksum'
04:06:39 INFO | Fetching packages.checksum from autoserv to /ghostcache/autotest/packages.checksum
04:06:40 DEBUG| packages.checksum could not be fetched from autoserv://
04:06:40 ERROR| packages.checksum could not be fetched from any of the repos ['autoserv://']
04:06:40 INFO | Fetching test-ltp.tar.bz2 from autoserv to /ghostcache/autotest/tmp/packages/test-ltp.tar.bz2
04:06:42 DEBUG| Successfully fetched test-ltp.tar.bz2 from autoserv
04:06:42 DEBUG| Running 'md5sum /ghostcache/autotest/tmp/packages/test-ltp.tar.bz2 '
04:06:42 DEBUG| Running 'ls /ghostcache/autotest/packages.checksum'
04:06:42 DEBUG| Running 'ls /ghostcache/autotest'
04:06:42 DEBUG| Running 'ls /ghostcache/autotest/packages.checksum'
04:06:42 INFO | Fetching packages.checksum from autoserv to /ghostcache/autotest/packages.checksum
04:06:42 DEBUG| packages.checksum could not be fetched from autoserv://
04:06:42 ERROR| packages.checksum could not be fetched from any of the repos ['autoserv://']
04:06:42 DEBUG| Running 'ls /ghostcache/autotest/tmp/site_tests/ltp'
04:06:42 DEBUG| Running 'cat /ghostcache/autotest/tmp/site_tests/ltp/.checksum'
04:06:42 DEBUG| Running 'md5sum /ghostcache/autotest/tmp/packages/test-ltp.tar.bz2 '
04:06:42 DEBUG| Running 'which gcc'
04:06:42 DEBUG| [stdout] /usr/bin/gcc
04:06:42 DEBUG| Running '/ghostcache/autotest/tmp/ltp/src/bin/runltp -q -l /ghostcache/autotest/results/default/ltp/results/ltp.log -C /ghostcache/autotest/results/default/ltp/debug/failcmdfile -d /ghostcache/autotest/tmp/ltp/src -S /ghostcache/autotest/tmp/site_tests/ltp/skipfile'

Following are the last log messages in the autoserv.DEBUG:

04/06 04:07:18 DEBUG| ssh_host:0114| Running (ssh) 'echo B > /ghostcache/autotest/tmp/tmp/_autotmp_TSPj3Hharness-fifo/autoserv.fifo'
04/06 04:07:19 INFO |server_job:0117| START ltp ltp timestamp=1396757199 localtime=Apr 06 04:06:39
04/06 04:07:19 DEBUG| ssh_host:0114| Running (ssh) 'echo B > /ghostcache/autotest/tmp/tmp/_autotmp_Shrrgaharness-fifo/autoserv.fifo'
04/06 04:07:20 INFO |autotest_r:1162| Bundling /usr/local/autotest/client/tests/ltp into test-ltp.tar.bz2
04/06 04:07:20 DEBUG|base_utils:0077| Running 'tar -cf /usr/local/autotest/tmp/tmp/_autotmp_Z_OsYUautoserv-packager/test-ltp.tar.bz2.tmp -C /usr/local/autotest/client/tests/ltp -j .'
04/06 04:07:21 DEBUG|base_utils:0077| Running 'rsync -L --timeout=1800 --rsh='/usr/bin/ssh -a -x -o ControlPath=/tmp/_autotmp_0ZyIxWssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmpY2oav8 -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=300 -l root -p 22' -az /usr/local/autotest/tmp/tmp/_autotmp_Z_OsYUautoserv-packager/test-ltp.tar.bz2 root@172.25.43.234:"/ghostcache/autotest/tmp/packages/test-ltp.tar.bz2"'
04/06 04:07:21 DEBUG| ssh_host:0114| Running (ssh) 'echo B > /ghostcache/autotest/tmp/tmp/_autotmp_6j3d8Charness-fifo/autoserv.fifo'
04/06 04:07:21 DEBUG| ssh_host:0114| Running (ssh) 'echo B > /ghostcache/autotest/tmp/tmp/_autotmp__K_PeQharness-fifo/autoserv.fifo'

Is there a way to work around this issue? A previous test run resulted in an Unhandled MemoryError. However, this time the test seems to just hang at this stage.

Thanks.

[lmr]

LTP is due some updating. I'll check your report out to see if there's more to it than it looks at first.

[ruda]

This issue is related to issue #630.

[lmr]

@bvpjenkins, I did run a test with the newest LTP that was added by @ruda in commit

autotest/autotest-client-tests@8bec6a7

And all looks well:

22:02:19 INFO |     END FAIL    ltp ltp timestamp=1397696539    localtime=Apr 16 22:02:19   
22:02:19 INFO | END GOOD    ----    ----    timestamp=1397696539    localtime=Apr 16 22:02:19   
22:02:19 INFO | Report successfully generated at /home/lmr/autotest/client/results/default/job_report.html

That is, only a a handful of tests fail, no hangs whatsoever. I'm closing this issue.