CI: functional tests using sim

[protobits]

I think it was discussed a few times but I'm not sure what is needed, but I think it would really help in ensuring the important parts of NuttX continue to work correctly if we at least ran some basic tests (such as ostest) using sim after it builds. I think it will also help for preparing releases (we could even make it so that more in-depth tests are done in CI for the release branch).

I also started using unity for some tests of custom apps and is quite easy to use. Maybe we can eventually migrate tests to that framework and manage some consistency.

[Ouss4]

Is it possible to use some external infrastructure if we want to run tests on something other than the simulator? I'm not sure what's the Apache stance on this one.

[protobits]

I would suggest aiming for sim as a first step. Testing on real hardware is much more complex to setup and maintain.

[Ouss4]

Testing on real hardware is much more complex to setup and maintain.

What I meant by "external infrastructure" is something that's already setup. But I agree that we should worry about this later.

I would suggest aiming for sim as a first step

For ostest, I think a simple (p)expect script should be enough. We also have this apache/nuttx-testing#70 which is still under review.

[btashton]

I have a PoC that I started validating the next release against that uses LabGrid (https://labgrid.readthedocs.io/en/latest/index.html) to run the sim and qemu tests. That is pytest driven for the validating. It is really basic, but since I can now grab the build artifacts from CI it is actually fairly quick to run.

One thing that I noticed doing this is it would be really nice if nsh would report the return code for the last command much like bash can with echo $?.

If there is interest I could put this up on the testing repo or something? I was going to wait until I ironed some of the issues out on this release, but I guess there is no real reason to wait.

I previously used labgrid to do a bunch of physical hardware testing.

[protobits]

For ostest, I think a simple (p)expect script should be enough. We also have this apache/incubator-nuttx-testing#70 which is still under review.

Yeah, I think simply detecting the test app exiting with non-zero status, detecting any error printed to console or sim crashing would go a long way. My suggestion about eventually moving tests to unity is to adopt a standard way to write the test apps. It is quite simply actually: http://www.throwtheswitch.org/unity

I have a PoC that I started validating the next release against that uses LabGrid (https://labgrid.readthedocs.io/en/latest/index.html) to run the sim and qemu tests. That is pytest driven for the validating. It is really basic, but since I can now grab the build artifacts from CI it is actually fairly quick to run.

Does this mean that the test job would not run here? I'm not familiar with LabGrid.
In my mind I would add a "test" step in CI, running some predefined tests after each sim build of interest.

One thing that I noticed doing this is it would be really nice if nsh would report the return code for the last command much like bash can with echo $?.

If not already, maybe we can make the sim exit with nsh's exit status, so that one can simply run nuttx sim binary and have the test run. Maybe it is also possible to pipe commands to nuttx binary, like: echo "ostest; poweroff $?" | nuttx/nuttx. This can be useful for more involved tests which require a series of commands, such as putting a net interface app, etc.

If there is interest I could put this up on the testing repo or something? I was going to wait until I ironed some of the issues out on this release, but I guess there is no real reason to wait.

Whatever brings us closer to this being enabled is welcomed IMHO

[btashton]

I have a PoC that I started validating the next release against that uses LabGrid (https://labgrid.readthedocs.io/en/latest/index.html) to run the sim and qemu tests. That is pytest driven for the validating. It is really basic, but since I can now grab the build artifacts from CI it is actually fairly quick to run.

Does this mean that the test job would not run here? I'm not familiar with LabGrid.
In my mind I would add a "test" step in CI, running some predefined tests after each sim build of interest.

For things that do not require actual hardware it can just be run as a regular pytest. What is really nice is that you can also use it to interact with real hardware including via openocd. So I could see a future where some of it happens in the normal CI run and some of it just is part of a nightly run or something that calls out to a physical test rack.

One thing that I noticed doing this is it would be really nice if nsh would report the return code for the last command much like bash can with echo $?.

If not already, maybe we can make the sim exit with nsh's exit status, so that one can simply run nuttx sim binary and have the test run. Maybe it is also possible to pipe commands to nuttx binary, like: echo "ostest; poweroff $?" | nuttx/nuttx. This can be useful for more involved tests which require a series of commands, such as putting a net interface app, etc.

I mean what I have setup is that I just start the nuttx binary and interact with it like it is a shell. What I was getting at is it would be be nice if in the shell you could query the exit code rather than having to write a special parser for every command.
nsh> ostest; echo "pass: $?"

If there is interest I could put this up on the testing repo or something? I was going to wait until I ironed some of the issues out on this release, but I guess there is no real reason to wait.

Whatever brings us closer to this being enabled is welcomed IMHO

Alright I'll put something up this week that at least does some of the basic execution.

[xiaoxiang781216]

Testing on real hardware is much more complex to setup and maintain.

What I meant by "external infrastructure" is something that's already setup. But I agree that we should worry about this later.

I would suggest aiming for sim as a first step

For ostest, I think a simple (p)expect script should be enough. We also have this apache/incubator-nuttx-testing#70 which is still under review.

LTP generate a very huge program list and then exceed the maximum command line length. We just made a workaround to split the list into small ones. @anchao please upstream the patch.

I have a PoC that I started validating the next release against that uses LabGrid (https://labgrid.readthedocs.io/en/latest/index.html) to run the sim and qemu tests. That is pytest driven for the validating. It is really basic, but since I can now grab the build artifacts from CI it is actually fairly quick to run.

Does this mean that the test job would not run here? I'm not familiar with LabGrid.
In my mind I would add a "test" step in CI, running some predefined tests after each sim build of interest.

For things that do not require actual hardware it can just be run as a regular pytest. What is really nice is that you can also use it to interact with real hardware including via openocd. So I could see a future where some of it happens in the normal CI run and some of it just is part of a nightly run or something that calls out to a physical test rack.

One thing that I noticed doing this is it would be really nice if nsh would report the return code for the last command much like bash can with echo $?.

If not already, maybe we can make the sim exit with nsh's exit status, so that one can simply run nuttx sim binary and have the test run. Maybe it is also possible to pipe commands to nuttx binary, like: echo "ostest; poweroff $?" | nuttx/nuttx. This can be useful for more involved tests which require a series of commands, such as putting a net interface app, etc.

I mean what I have setup is that I just start the nuttx binary and interact with it like it is a shell. What I was getting at is it would be be nice if in the shell you could query the exit code rather than having to write a special parser for every command.
nsh> ostest; echo "pass: $?"

nsh already support $?, we have been used this feature to check the test result more than half year. We have fixed serveral tests inside apps forget to return the error, @ttnie please upstream the related patch.

If there is interest I could put this up on the testing repo or something? I was going to wait until I ironed some of the issues out on this release, but I guess there is no real reason to wait.

Whatever brings us closer to this being enabled is welcomed IMHO

Alright I'll put something up this week that at least does some of the basic execution.

Yes, it will be great if you can setup the basic test infrastructure. Our test team can integrate more test cases into the infrasturcture.

[Ouss4]

Yeah, I think simply detecting the test app exiting with non-zero status, detecting any error printed to console or sim crashing would go a long way. My suggestion about eventually moving tests to unity is to adopt a standard way to write the test apps. It is quite simply actually: http://www.throwtheswitch.org/unity

Unity is more or less focused on unit testing, we already have a couple of examples (apps/examples/usrsocktest&dsptest).

LTP generate a very huge program list and then exceed the maximum command line length. We just made a workaround to split the list into small ones. @anchao please upstream the patch.

@xiaoxiang781216 If you have a ready to run LTP defconfig for the simulator (or any other board), please upstream it as well.

[ttnie]

I have a PoC that I started validating the next release against that uses LabGrid (https://labgrid.readthedocs.io/en/latest/index.html) to run the sim and qemu tests. That is pytest driven for the validating. It is really basic, but since I can now grab the build artifacts from CI it is actually fairly quick to run.

Does this mean that the test job would not run here? I'm not familiar with LabGrid.
In my mind I would add a "test" step in CI, running some predefined tests after each sim build of interest.

For things that do not require actual hardware it can just be run as a regular pytest. What is really nice is that you can also use it to interact with real hardware including via openocd. So I could see a future where some of it happens in the normal CI run and some of it just is part of a nightly run or something that calls out to a physical test rack.

One thing that I noticed doing this is it would be really nice if nsh would report the return code for the last command much like bash can with echo $?.

If not already, maybe we can make the sim exit with nsh's exit status, so that one can simply run nuttx sim binary and have the test run. Maybe it is also possible to pipe commands to nuttx binary, like: echo "ostest; poweroff $?" | nuttx/nuttx. This can be useful for more involved tests which require a series of commands, such as putting a net interface app, etc.

I mean what I have setup is that I just start the nuttx binary and interact with it like it is a shell. What I was getting at is it would be be nice if in the shell you could query the exit code rather than having to write a special parser for every command.
nsh> ostest; echo "pass: $?"

If there is interest I could put this up on the testing repo or something? I was going to wait until I ironed some of the issues out on this release, but I guess there is no real reason to wait.

Whatever brings us closer to this being enabled is welcomed IMHO

Alright I'll put something up this week that at least does some of the basic execution.

pytest is a good framework, but unittest is really good. It have a good scalability, and good to inherited and good usage. It is recommended that you use unttest. :)

[protobits]

I'm wondering if this will require repeating the build but with debug assertions (and other features) on, for example. Otherwise we may not catch all errors (only those recognized during runtime in release builds).

[xiaoxiang781216]

It's better to run the test with ASAN on sim too, so we can catch most the memory/undefined error: #3093

[protobits]

Very interesting, I did not know about that feature.

[Ouss4]

I'm wondering if this will require repeating the build but with debug assertions (and other features) on, for example. Otherwise we may not catch all errors (only those recognized during runtime in release builds).

We can add a bunch of kconfig-tweak to enable all these options just before building. This can also be behind an option that we pass to testbuild.sh/cibuild.sh.

[btashton]

I'm wondering if this will require repeating the build but with debug assertions (and other features) on, for example. Otherwise we may not catch all errors (only those recognized during runtime in release builds).

We can add a bunch of kconfig-tweak to enable all these options just before building. This can also be behind an option that we pass to testbuild.sh/cibuild.sh.

I had a PR for doing this a while back but it was dropped, I don't remember why.

It also might be worth just added another stage to the build that rebuilds a subset for run-time testing that does a bunch of little tweaking.

[xiaoxiang781216]

The basic test is added by #7109. Let's close this issue.