Skip to content

Running OmnisTAP from the command line

Alex Clay edited this page Mar 19, 2018 · 4 revisions

OmnisTAP pairs extremely well with OmnisCLI. Using both tools enables continuous integration through tools like Jenkins or Bitbucket.

There are two commands available in OmnisTAP from the command-line.

  • gettests [regexp]
  • runtests tap_output [regexp]

gettests will list all tests that are available in the library whose startup task subclasses omnistap.kgTAPTask. runtests will run these tests, saving output to tap_output. Both commands can limit the tests they see using an Omnis-compatible regular expression passed to regexp. If you need to pass additional parameters, such as a test database or host, use . as the regexp value to run all tests.

TAP output is saved as one file per test class under [tap_output]/[test_method].tap. You can evaluate this output using any TAP-compatible consumer.

OmnisTAP will enforce tests run within a certain amount of time. These limits are 5 seconds for unit tests and 90 seconds for integration tests. You can adjust this timing from the command line by exporting the OMNIS_TEST_TIMING_MULTIPLIER variable as a floating-point number. For example, a macOS machine that runs tests twice a slowly would run:

export OMNIS_TEST_TIMING_MULTIPLIER=2.0

On Windows, you can use:

SET OMNIS_TEST_TIMING_MULTIPLIER=2.0

When running from the CLI the startup and shutdown methods are available to bootstrap a test environment. Add a test class to your library that subclasses omnistap.ogTAPStartupShutdown. Override $startup() to run code at the start of the run, and $shutdown() to run code at the end of the run.

Diagnostics

When running from the command line, OmnisTAP will write a number of csv files to a diagnostics directory in the tap_output directory. These files are:

  • coverage.csv
  • profile.csv
  • test_types.csv
  • timings.csv

coverage.csv tracks code coverage during the test run. Coverage is measured at the method, or functional, level. Method execution is tracked using Omnis' $collectperformancedata feature. Because this feature tracks method calls, not individual lines of code, there may be execution branches in your methods that are not evaluated even though the method is considered "covered". Methods from OmnisCLI and OmnisTAP are excluded from this coverage measurement.

profile.csv outputs the raw profiling performance data for all methods that were executed at least once during the test run, including those in OmnisCLI and OmnisTAP. See the Omnis Programming manual under Chapter 4 -> Method Performance -> Collecting Performance Data for details on this data. This output can be helpful for identifying performance issues in your application and optimizing them.

test_types.csv reports the number of unit and integration tests run. This output can help ensure your test suite follows Martin Fowler's Test Pyramid concept.

timings.csv reports how long each test took to run, sorted in descending order by time. Use this output to identify slow tests for optimization.

These diagnostics are reported as CSV files to allow easy parsing and analysis from run to run.

You can’t perform that action at this time.