Output during tests execution

[OndraM]

Currently output of all tests is printed in the console output right after each test is finished. This makes the output a bit messy and is not always necessary. We should use output levels (default, verbose, very verbose, debug) to make the output more helpful in different situations.

Output levels distribution

Standard level

Use-case: I just want to run everything and I care only about test results count, not about the cause of errors etc.

• Version info on startup
• Current status during run (finished, remaining and executed process count)
• Process results in execution loop output (added in Provide information about test results also in CLI output #44)
• Test results on end

Verbose level (-v)

Use-case: I want to run tests locally and see some basic startup/status/progress information during the run. If some test fails, I want to know which one. (But don't care about exact error message.).

• Basic startup configuration info (version, browser, environment,)
• Name of failed test

Very verbose level (-vv)

Use-case: I want to run tests locally and see detailed startup/status/progress information during the run. If something happens during the run, I want to know where and what exactly.

• Detailed startup configuration info (all run-time config, selenium server URL, log path, etc.)
• Test started/unqueued/finished
• Error output of failed tests
• Process timeout error

Debug level (-vvv)

Use-case: I am debugging some tests and want all available information, see what starts and when; OR the tests are run on CI server (and the output is stored for possible later investigation)

• List of found files, testcases and their dependencies and delays on startup
• Command line of started process
• All output and error output from tests
• With this level the DEBUG variable is passed to each PHPUnit process, thus tis output is added to the test output:
  • Executed WebDriver commands

Uncategorized messages

• "Testcase "%s" is prepared to be run" (currently in debug level)
• 'Testcase "%s" is queued to be run %01.1f minutes after testcase "%s" is finished' (currently debug)
• 'Running command for class ...' (currently debug)
• 'No tasks left, exiting the execution loop...'
• 'Process for class "%s" finished'
• 'Unqueing class "%s"'

Messages from the PHPUnit process

• "Initializing "%s" WebDriver for "%s::%s" (from the PHPUnit process)
• Destroying "%s" WebDriver for "%s::%s" (session %s) (from the PHPUnit process)
• "Test failed on page "%s", taking page snapshots" "Screenshot saved to file ..." "HTML snapshot saved to file" (from the PHPUnit process)
• "Starting execution of test %s" (from the PHPUnit process)
• "Finished execution of test %s" (from the PHPUnit process)

Any thoughts, ideas, comments?

[OndraM]

#46 is now merged, will be released in 1.3.0.