Make OSAL tests more autonomous

[skliper]

The OSAL tests (in the "tests" directory, NOT "unit-test") provide reasonable "black-box" style testing of the OSAL features via the public API. However they are currently implemented with the expectation of being explicitly run by a user at the console, not from an automated script:

• Most run infinitely, requiring a CTRL+C to stop the test
• They do not count errors or provide a status code to indicate whether the test worked as expected or not.

In order to integrate with a continuous integration system, these tests need to be more automated.

This ticket is to add a "script-mode" feature to the tests which:

• Checks for the expected conditions and maintains an error counter of any tests that did not satisfy expected conditions
• Limits the time of execution and exits the test automatically (no CTRL+C)
• Returns a non-zero exit code if any errors occurred (for scripting)

(Note: this does not apply to the "unit-tests" directory which is more white-box style code coverage testing -- this will possibly need other tweaks to work with a CI system and that is a separate ticket)

[skliper]

Imported from trac issue 6. Created by jphickey on 2014-12-30T11:21:19, last modified: 2015-11-20T16:22:16

[skliper]

Trac comment by glimes on 2015-01-29 15:10:42:

Click [changeset:4114552 here] to see the changeset.

[skliper]

Trac comment by glimes on 2015-01-29 16:33:10:

I have reviewed [changeset:4114552 changeset 4114552] and recommend ACCEPT.

[skliper]

Trac comment by mlogan on 2015-01-30 18:40:10:

I have reviewed and ACCEPT these changes with the following comment(s):

• I'm a little concerned about having a compile-time requirement for the new script mode.
  This may have been done to preserve the explicit run capability by users.

  Might be nice to have a -DSCRIPT_MODE commented out but visible to the user in
  the build/tests/*-test/Makefile(s) so they are aware of it.

[skliper]

Trac comment by jphickey on 2015-01-30 21:19:29:

You are correct that this change is is in an #ifdef SCRIPT_MODE in order to preserve the existing/old behavior when running a normal build. There may be some value in getting extended runtime on some of these tests, like running overnight or over a weekend, to prove out the implementation. So the idea is to preserve the current behavior of running until CTRL+C is pressed as a default.

The CMake scripts always define "SCRIPT_MODE" when building these tests so that running "make test" will run them and allow them to complete in a time-limited fashion. The target was to complete the full suite of tests in about 4-5 minutes which might be reasonable for an automated build server to do.

There would definitely be some value-add to putting these tests into a common UT framework like CFE has, this is just asking for a command line option to indicate "run forever" or "run short test", etc.

[skliper]

Trac comment by acudmore on 2015-02-11 15:12:25:

I conditionally ACCEPT
I would like to see OS_ApplicationExit for RTEMS and vxWorks ( and soon FreeRTOS ), at least stubs.

The RTEMS Tools project is working on a embedded systems test framework that may be useful for CI on simulators and embedded targets:
http://git.rtems.org/rtems-tools/tree/tester

[skliper]

Trac comment by jwilmot on 2015-02-11 15:24:36:

ACCEPT with Alan's comment

[skliper]

Trac comment by glimes on 2015-02-12 14:39:49:

Ames will check this out on VxWorks.

[skliper]

Trac comment by glimes on 2015-02-12 14:58:17:

Accepted by #41 for integration candidate.

[skliper]

Trac comment by glimes on 2015-03-04 15:06:52:

Confirmed that the fix from [changeset:c5f8cc8] is present in the
current development branch of OSAL, [changeset:5aae372].

https://babelfish.arc.nasa.gov/bamboo/browse/CFS-CFSOSAL
confirms that the Bamboo server is able to build these tests,
stage them on a 32-bit X86 Linux VM, and run them, and that
the "test time-out" safety net is not activated: the changes
successfully assure that the tests do not loop forever.