diff --git a/docs/po4a.cfg b/docs/po4a.cfg index 5c8c02d3175..9b06c9e7395 100644 --- a/docs/po4a.cfg +++ b/docs/po4a.cfg @@ -26,6 +26,7 @@ [type: AsciiDoc_def] src/code/nml-messages.adoc $lang:src/$lang/code/nml-messages.adoc [type: AsciiDoc_def] src/code/rs274.adoc $lang:src/$lang/code/rs274.adoc [type: AsciiDoc_def] src/code/style-guide.adoc $lang:src/$lang/code/style-guide.adoc +[type: AsciiDoc_def] src/code/writing-tests.adoc $lang:src/$lang/code/writing-tests.adoc [type: AsciiDoc_def] src/common/emc-history.adoc $lang:src/$lang/common/emc-history.adoc [type: AsciiDoc_def] src/common/glossary.adoc $lang:src/$lang/common/glossary.adoc [type: AsciiDoc_def] src/common/gpld-copyright.adoc $lang:src/$lang/common/gpld-copyright.adoc diff --git a/docs/src/Submakefile b/docs/src/Submakefile index 50428106a2e..e8656dbcf8b 100644 --- a/docs/src/Submakefile +++ b/docs/src/Submakefile @@ -41,6 +41,7 @@ DOC_SRCS_EN := \ code/adding-configs.adoc \ code/contributing-to-linuxcnc.adoc \ code/building-linuxcnc.adoc \ + code/writing-tests.adoc \ common/emc-history.adoc \ common/glossary.adoc \ common/gpld-copyright.adoc \ diff --git a/docs/src/code/contributing-to-linuxcnc.adoc b/docs/src/code/contributing-to-linuxcnc.adoc index a0b8d4f697c..a4d5550ed44 100644 --- a/docs/src/code/contributing-to-linuxcnc.adoc +++ b/docs/src/code/contributing-to-linuxcnc.adoc @@ -112,7 +112,7 @@ like this: * Communicate with the project developers and let us know what you're hacking on. Explain what you are doing, and why. * Clone the git repo. * Make your changes in a local branch. -* Adding documentation and tests is an important part of adding a new feature. Otherwise, others won't know how to use your feature, and if other changes break your feature it can go unnoticed without a test. +* Adding documentation and <> is an important part of adding a new feature. Otherwise, others won't know how to use your feature, and if other changes break your feature it can go unnoticed without a test. * Share your changes with the other project developers in one of these ways: ** Push your branch to github and create a github pull request to (this requires a github account), or ** Push your branch to a publicly visible git repo (such as github, or your own publicly-accessible server, etc) and share that location on the emc-developers mailing list, or diff --git a/docs/src/code/writing-tests.adoc b/docs/src/code/writing-tests.adoc new file mode 100644 index 00000000000..12060659c29 --- /dev/null +++ b/docs/src/code/writing-tests.adoc @@ -0,0 +1,124 @@ +[[cha:writing-tests]] += Writing automated LinuxCNC tests + +== The test framework + +The code base has unit and integration tests that can be executed +automatically to ensure the program work as intended. Such tests are +often written to trigger a bug and to ensure the bug is detected if it +resurface in the future, but also to validate behavior of components +and interfaces. + +The tests are collected in the `tests/` directory. The individual tests +are in subdirectories of this directory. The tests are grouped in +directories. + +== Running tests + +The tests are executed by the `scripts/runtests` script generated from +`scripts/runtests.in` during build. The runtests script will by default +locate tests to run under `tests/`, but can be limited to only run a +limited set of tests by specifying the directory of the test or tests +as argument(s). + +Here is an example running only the tests in `tests/lathe/` + +---- +$ scripts/runtests tests/lathe/ +Running test: tests/lathe +Runtest: 1 tests run, 1 successful, 0 failed + 0 expected, 0 skipped +---- + +The runtests script looks for all files named _test_, _test.sh_ and +_test.hal_ below the directories specified on the command line, or under +`tests/` if no command line argument is specified. These files +represent three different ways to run the tests. + +The _runtests_ script accept the following arguments, see the output +from `scripts/runtests -h` for the authorative list: +---- +-n do not remove temporary files for successful tests. +-s stop after any failed test. +-p print stderr and result files. +-c Remove temporary files from an earlier test run. +-u Only run tests that require normal user access. +-v Show stdout and stderr (normally it's hidden). +---- +== Writing tests + +Make sure the test can run successfully without a working X11 display, +i.e. with the DISPLAY environment variable unset. + +1. Create a folder in `tests/`. +2. Provide one test script. +3. Evaluate the output with one of the options below. + +These are the files considered in the directory with the individual +tests: + +.Test script (only one of these three) + +test:: + A program executed and its exit code and output is checked using + either checkresult or expected. + +test.sh:: + A bash script executed and its exit code and output is checked using + either checkresult or expected. + +test.hal:: + A HAL script executed using `halrun -f test.hal` and its exit code + and output is checked using either checkresult or expected. + +.Test evaluation + + expected:: + A file whos content is compared to the output from running the test + scripts. If the test output is identical to the content of the + expected file, the test succeed. + +checkresult:: + An excutable file to perform more complex validation than just comparing + the output of a test script. It gets the filename of the test program as + its command line argument. The exit code of this program controls the result + of the test. If both `expected` and `checkresult` exist, only `checkresult` + is consulted to validate the test output. + + xfail:: + If this file exist, a test failure is expected and do not cause + runtests to return an exit code signaling an error. + + skip:: + If this file exist, the test is skipped and not executed at all. + + control:: + This file can be used to flag specific needs in the test. At the + moment, the use of _sudo_ can be flagged, and tests requiring sudo + can be skipped when using `runtests -u`. To flag such requirement, + add a line with `Restrictions: sudo` to this file. + +== Some testing approaches + +There are various ways to structure a test, depending on what one want +to test. Here are a few ideas on how to do it. + +=== Non-interactive "GUI" + +If you want to test some operations in the user interface, a useful +approach is is to write a custom "GUI" simulating the operations. +This can be done by creating a normal LinuxCNC setup and pointing the +[DISPLAY] DISPLAY value to a script that do the operations needed to +test the behaviour. + +Examples of this approach can be found in `tests/halui/jogging/` and +`tests/interp/subroutine-return/`. + +=== Recording HAL pin transitions + +Using the _sampler_ and _halsampler_ HAL components, one can set up a +HAL configuration and collect pin value settings and changes and dump +the result to stdout (or a file). The end result can then be compared +with the expected output to verify if HAL behave as expected. + +Examples of this approach can be found in `tests/multiclick/` and +`tests/stepgen.2/`. diff --git a/docs/src/index.tmpl b/docs/src/index.tmpl index e34c91c0a9f..b5be1530ae9 100644 --- a/docs/src/index.tmpl +++ b/docs/src/index.tmpl @@ -316,6 +316,7 @@ function setup_page(){
  • Style Guide
  • Adding Configs for Selection
  • Contributing to LinuxCNC
    • +
  • Writing tests for LinuxCNC
  • Building LinuxCNC