Skip to content

Language tests

Seth Tisue edited this page Mar 15, 2014 · 13 revisions
Clone this wiki locally

We have a modest external DSL that makes it easy to write little self-contained tests that test the behavior of individual NetLogo language constructs, as well as whole models and extensions.

Where they are

in these directories:

  • test/commands/*.txt
  • test/reporters/*.txt
  • models/test/*.txt
  • extensions/*/tests.txt

On the headless branch, the first two are inside a resources directory.

How to run them (main build)

To run all command or reporter or model or extension tests from sbt:

tc
tr
tm
te

You can run a single group of tests with, for example:

tc InCone 
tr Numbers
tm Fire
te array

These commands also support running a single test, for example:

tr Numbers::Sqrt3

All of the language tests are run as part of the sbt commands test and slow:test. tr and tc are part of medium:test. All are run by the nightly.sh script which runs on Travis-CI (see Continuous integration).

How to run them (extension builds)

The tests for any extension can be run within the main NetLogo build just by putting the extension directory under extensions and using te from the NetLogo root directory.

Some extensions have been updated so that the tests can be run within the extension's own build, without having to build NetLogo. As of March 2014, the two extensions that have this are matrix (Makefile-based) and nw (sbt-based). It would be nice if all the extension builds were updated in this way.

If the extension build is make-based, the shell command to run the tests is make test.

If the extension build is sbt-based, the shell command to run the tests is sbt test.

Writing tests

All types of tests

When you use =>, the value on the right (the expected value of the reporter on the left), must be a constant: a number, string, list, true/false, or nobody.

Or, if you expect a runtime error to occur, use ERROR, e.g.:

O> set glob1 foo => ERROR REPORT must be immediately inside a TO-REPORT.
[color] of glob1 => ERROR That mouse is dead.

For code that shouldn't even compile, say COMPILER ERROR instead, e.g.:

 O> __ignore value-from turtle 0 [color] => COMPILER ERROR Nothing named VALUE-FROM has been defined

Tests that end with _2D will be run in a 2D environment only, tests that end with _3D will be run in a 3D environment only, all other tests will be run in both environments.

Tests that start with Generator will only be run when the code generator is enabled. Tests that start with NoGenerator will only be run when the code generator is disabled. Other tests run regardless.

When you write tests that write files you should be careful to use a unique file name so that tests may run in parallel without interfering with each other. See test/commands/AgentsetImport or test/commands/File for examples.

Reporter tests

Keep these simple. Don't use any of the special command-test features described below.

Nothing in reporter tests does anything with agents or agentsets. If you want to test agent stuff, use command tests, not reporter tests.

Nothing in reporter tests does anything involving randomness, since there's no way to set the random seed. Use command-tests for that.

Command tests

You can define a procedure as part of a test.

Global variables glob1, glob2, and glob3 are available for use, plus a turtle variable tvar and a patch variable pvar.

Breeds mice and frogs are available. Both breeds have an age variable. Mice also have fur. Frogs also have spots. Also available are two link breeds, directed-links (with variable lvar) and undirected-links (with variable weight).

Two plots are available, plot1 and plot2. Both have two permanent pens named pen1 and pen2.

Command tests are run two times:

  • once normally
  • once with each command wrapped in "run"

If the test should only be run normally, then begin the name with an asterisk, e.g. *MyTest

Always set the random seed first if you do anything involving randomness. Note that includes some primitives you may not expect, such as ask on an agentset, crt, sprout, max-one-of, et al.

Model tests

You can open a model at the start using OPEN>. see models/test for examples. By convention, we don't use OPEN> in other directories besides models/test (or extensions).

If the test uses a model that isn't used anywhere else (e.g. if it isn't part of the Models Library), it can go next to the test in the models/test directory.

Extension tests

Your extension should be in the extensions directory and have a tests.txt file in it containing command tests. see e.g. extensions/table/TableExtension.txt.

Extensions tests don't automatically run in NetLogo 3D if you just use ./sbt "te extension-name", but you can run them in 3D separately by doing env JAVA_OPTS="-Dorg.nlogo.is3d=true" ./sbt "te extension-name". Name suffixes like _2D and _3D are honored for extensions tests.

Stack trace tests

It's now possible to test NetLogo stack traces from both reporter and command tests. The easiest way to see how is by example. Here are examples of both:

*command-task-in-command-task-stack-trace
  O> run task [run task [print __boom]] => STACKTRACE boom!\
  error while observer running __BOOM\
    called by (command task from: procedure __EVALUATOR)\
    called by (command task from: procedure __EVALUATOR)\
    called by procedure __EVALUATOR

*Xor7
  __boom xor false => STACKTRACE boom!\
  error while observer running __BOOM\
    called by procedure __EVALUATOR

Notice the * at the start of the test name. this is to prevent it from running with the run command, since the resulting stack trace would be slightly different since the top level procedure is run instead of __EVALUATOR. It's OK that these tests don't run in run mode because we are only testing the stack traces, not the results.

Debugging with language tests

To print values to standard out from within a test, use __stdout. For example,

O> crt 5
O> __stdout sort turtles

would print [(turtle 0) (turtle 1) (turtle 2) (turtle 3) (turtle 4)].

Something went wrong with that request. Please try again.