Some problems with using shellspec for integration tests

[AntoniMarcinek]

What I want to achieve

I am considering to use your great project to drive integration/system tests for a project I am responsible for in the NA61/SHINE experiment at CERN.

I have a directory structure like this:

IntegrationTests
├── Examples
│   ├── App1
│   └── App2
└── Standard
    └── AppA

Each App directory would contain a single test_spec.sh file defining our expectations towards the given app. There would be an order of 100 apps. These apps actually may run quite long (from seconds to several minutes each), so it is quite important to run them in parallel. Each test_spec.sh file would essentially define a separate and independent test suite with multiple example blocks.

The tests would be run with GitLab CI so both a terse output showing the progress online, as well as the junit generator would be needed.

The goal is to run shellspec in the top directory executing all tests in subdirectories. It would also be nice to be able to execute it from the subdirectory.

The demonstration of my use case

I prepared an example in the https://gitlab.com/amarcine/test-shellspec repository. The apps are scripts but in real life these would be compiled programs or sets of xml files constituting an input to the main executable of the project (available in the $PATH when shellspec is executed). If you grep for "NOTE", you get my comments regarding what I would like to have and what I dislike in the current behaviour.

The problems

working directory of the test

Unfortunately my demonstration does not work currently, because each test is run from the directory in which shellspec was executed, while the executables under test expect to be run from the directories they reside in (App1 etc.). I understand that I could make this work putting cd in each test_spec.sh file with the relative path to the App directory, but this is very ugly. Furthermore, then it is not possible to execute shellspec in the App directory and the test becomes location dependent (cannot be moved).

I understand that currently there is no such feature, but would it be possible to add an option to shellspec, e.g. -C like in tar or make, to cd to the directory of each spec file before executing it?

formatter

Right now using any formatter which prints messages about which tests are run (documentation, tap), I have 2 problems:

• I can only see the tests after they are finished, so actually I don't know which test currently runs, what is rather problematic with long tests.
• All examples are listed, so the list is actually longer than I would expect.

For unit tests suites our project uses ctest. I show a (truncated) example output below. What I like about this formatting is that I get a message when the test starts and another when it ends. I also get the progress expressed as sequential number of the current test / number of all tests. I only get an information which test fails, but not the failure report, which is fine, because that appears in the xml report. Last but not least I get the timing for each test.

I would like to get something similar with shellspec. The "tests" printed should correspond to the level of parallelization, i.e. a spec file, so instead of printing titles of all examples, I should get only the top example group title. So instead of a full report this would be rather a terse summary of all test suites run.

Looking at the documentation output I have an impression that this should be possible, but I couldn't understand how to implement a custom formatter myself. Could you give some instructions on how to do that (unless you find it a nice feature to have and you implement it yourself)?

The ctest -j 8 example output

Test project /home/antek/Install/Shine/dev/build
      Start 34: testMagneticField
      Start 68: testTPCDEDXCalculatorSG
      Start 37: testTPC
      Start 35: testTimeBinMode
      Start 22: testTime
      Start 57: testTPCCalibManagers
      Start 33: testDetector
      Start 36: testTOF
 1/69 Test #36: testTOF ..................................   Passed    1.72 sec
      Start 38: testTarget
 2/69 Test #33: testDetector .............................   Passed    1.75 sec
      Start 42: testGasBags

(....)

 6/69 Test #37: testTPC ..................................   Passed    2.39 sec
      Start 67: testBPDReconstructorSG
 7/69 Test #34: testMagneticField ........................   Passed    2.59 sec
      Start 69: testMagneticTrackFitter
 8/69 Test #68: testTPCDEDXCalculatorSG ..................***Failed    2.83 sec
      Start 15: testMath
 9/69 Test #15: testMath .................................   Passed    0.37 sec
      Start 56: testBPDManagers

(....)


66/69 Test #29: testSparseVector .........................   Passed    0.13 sec
67/69 Test #23: testTrackParameters ......................   Passed    0.13 sec
68/69 Test #14: testPolynomialInterpolation ..............   Passed    0.10 sec
69/69 Test #67: testBPDReconstructorSG ...................   Passed    3.36 sec

99% tests passed, 1 tests failed out of 69

Total Test time (real) =   5.76 sec

The following tests FAILED:
	 68 - testTPCDEDXCalculatorSG (Failed)
Errors while running CTest

[ko1nksm]

Hi,

I have read this Issue and am considering adding two features (change working directory and new formatter). However, since it may be difficult to implement by design, it may be more appropriate to run multiple instances of a ShellSpec (e.g., using Make or GNU paralells).

working directory of the test

We will need to accommodate the new directory structure. Each (App) directory may contain subdirectories, so I think we need a way to know the base directory to run the tests.

We might be better off using other tools like Make, but we're going to struggle with parallel execution and formatter output.

formatter

I think a new formatter that outputs results for each specfile name only is a good idea.

As you may have noticed, ShellSpec allows you to specify separately which formatter to output to the display (--format) and which one to output to the file (--output). This allows you to output to a file while displaying the progress on the screen. The exception is the jUnit XML format, which cannot be generated unless all the results are available due to the structure of the jUnit XML format.

Order of test results

Looking at the output of the ctest, it appears that the tests run in parallel are output at the time you start running them.

In ShellSpec, the output is displayed in the same order in parallel execution as in serial execution. This process is done with a parallel-executor, not a formatter, so if you want to output the output at the time you start executing, you can't achieve this with a custom formatter alone.

Test execution time

In ctest displays the run time of each test, but ShellSpec does not measure the run time by default. because ShellSpec is implemented in a POSIX shell-first and there are limited ways to measure time to the millisecond in the POSIX shell.

The profiler feature of ShellSpec is implemented in a tricky way. ShellSpec count numbers in the background while running the test when the profiler enabled. And each test time calculats based on the total execution time and the count taken for each test execution.

It is possible to get the time in milliseconds with bash, ksh, zsh, but it needs to be implemented additionally.

Test title

I should get only the top example group title.

ShellSpec allows you to give multiple top-level titles, so I think the file name is better.

If I were to implement it myself

I would make a formatter that just alternates between the name of the file and the results of the tests I ran (no test run time).

What do you think?

How to make custom formatter

It should be work, but I've barely tested it. And I also haven't documented it. This means that changes are likely to be introduced in the future.

This directory may be helpful. debug_formatter.sh and tap_formatter.sh are simple.

When run shellspec --format custom, spec/custom_formatter.sh will be loaded.

This is a (lack of explanation) document of the reporting protocol.
https://github.com/shellspec/shellspec/blob/master/docs/reporter.md

I'm sorry. I know there is a lot to be explain.

[AntoniMarcinek]

Thanks for a fast answer!

Reading through it, I started to think that maybe it is more reasonable to use a combination of ctest and shellspec, with ctest being responsible for running shellspec in every directory in a parallel manner, providing the terse, summary output that I want and delegating to xml all the detailed output. In this way shellspec would only be responsible for assertions and formalisation of the tests in a common language, while its features from the **** Execution **** and **** Ranges / Filters / Focus **** would be taken over by ctest. It is better than make or parallel (actually now we have this run via make, without parallelisation, without formalisation so every test is essentially completely arbitrary shell script which just exits with success or failure; it is horrible).

The problem of using mixed ctest and shellspec is that it kind of abuses cmake (there is nothing to build by cmake) and it requires additional files compared to test_spec.sh files (CMakeLists.txt files). Maybe it is still possible to just generate the ctest suite on the fly via a combination of find and sed since what I want is that every test suite be defined just with a single test_spec.sh file interpreted by shellspec, so each CMakeLists.txt would look exactly the same with an exception of the name of the test to reflect the location.

Still reiterating on a possibility to achieve this with the shellspec alone, let me comment on your comments.

working directory of the test

My idea is to cd to the dir where the current _spec.sh file is, for every _spec.sh file processed. Since shellspec discovers these files, it knows their directories. So I think it is not relevant what is the exact directory structure.

formatter

I know about -o and -f. The idea is precisely to use some terse -f and -o junit.

Order of test results

Right, but couldn't the executor make a callback to the formatter with the current _spec.sh file path and the formatter would either ignore the message or print it to screen? Obviously, as you wrote, that would require adding something to the executor code.

Note that in ctest each test comes with 2 lines - at beginning just writing which test is started and at the end the actual result of the test.

Test execution time

I am not sure what accuracy here is relevant, so maybe the issue of milliseconds is not an issue. Still I understand that with profiling on, the junit formatter prints the time of each test. If the information is only available when all tests are done (not just the given _spec.sh file is done), then that is it - there can be no printing in the way of ctest. This is not a big deal.

Test title

I agree the file name (or rather relative path) is better.

If I were to implement it myself

Looks fine.

Although I think it doesn't make much sense for such a formatter to print more than a summary - failed, passed, skipped.

How to make custom formatter

Thanks. Given that you already kind of agreed to take care of that yourself and the solution may go beyond the formatter itself, then probably at the end I will not try doing it myself.

[AntoniMarcinek]

I realised that trying to use combination of ctest and shellspec I would need a global flag to print the whole stdout/err from the commands under test. Right now I could achieve this specifying Dump in each example, but this is again quite ugly. Using only shellspec I would still need #124 .

[AntoniMarcinek]

I realised that trying to use combination of ctest and shellspec I would need a global flag to print the whole stdout/err from the commands under test. Right now I could achieve this specifying Dump in each example, but this is again quite ugly.

Actually maybe not... I can use ctest to parallelize and assure proper working directories, but gather xml from shellspec itself. Then I need only #124

[ko1nksm]

I believe that your demonstration is now working with #135 and #137 merged into the master. If you specify -C @specfile, you can remove the cd. Please feel free to comment if you have any problems or opinions. I'm going to implement #143 to improve this problem a bit more.

Formatter-related implementation will be done later than the next version 0.28.0. After addressing some options, I will release 0.28.0 (this weekend?).

[AntoniMarcinek]

Problems with -C @specfile

I updated the demonstration project https://gitlab.com/amarcine/test-shellspec to use the new feature. Unfortunately I got an error:

antek@top ~/test-shellspec > pwd
/home/antek/test-shellspec
antek@top ~/test-shellspec > shellspec 
/bin/sh: 91: cd: can't cd to /home/antek/test-shellspec/home/antek/test-shellspec/IntegrationTests/Examples/App1

Unexpected error occurred (syntax error?) occurred in '/home/antek/test-shellspec/IntegrationTests/Examples/App1/test_spec.sh'
Running: /bin/sh [sh]

Finished in 0.22 seconds (user 0.13 seconds, sys 0.03 seconds)
0 examples, 0 failures, aborted by an unexpected error

Aborted with status code [executor: 2] [reporter: 1] [error handler: 102]
Fatal error occurred, terminated with exit status 102.

Do you have integration tests for shellspec? I've seen the unit tests for the new functionality but looks like they either don't cover my situation or the problem appears on the later stage so beyond the scope of these tests.

Problems with #135 and the associated documentation

I have also another problem with the changes of the #135: now the project explicitly requires the .shellspec file even if empty (or it changes the directory up until it finds any .shellspec file - in my case in my home directory). This is not a problem, but was a surprise to me when I was working on something that was to become one of subdirectories in my integration tests, but was not yet in the project. Before the changes, the workflow was possible, but with the new changes it no longer works.

This is not well documented: the https://github.com/shellspec/shellspec#project-directory-structure labels .shellspec and spec/spec_helper.sh as "Required" but until now none seemed to be actually required. After #135 the first is really required. With the #137 we do get some idea what is the working directory for the tests, but unfortunately it is not clear what "project root directory" means - the information has to include that it is where the .shellspec file is located. Otherwise I was expecting that @specfile and @basedir may cause a cd while the default simply runs in the current directory of the call to shellspec. I propose the following change:

[@project]       Project root directory [default]
[@basedir]       Where the file .shellspec-basedir is located

to

[@project]       Where the .shellspec file is located (project root directory) [default]
[@basedir]       Where the .shellspec-basedir file is located

Also if the "project root directory" has broader significance than to define the working directory (e.g. associated with how shellspec localizes helpers #130 (comment)) then the definition and the significance of this term needs to be discussed somewhere in a visible place of the documentation (unfortunately here I don't have a proposal how to do it).

Finally with the changes also the content of the jUnit xml report changed - now I see absolute paths to _spec.sh files instead of relative paths. This is again not a big deal, but definitely the report will have to be passed through sed to erase the part of the path which is machine-dependent as opposed to project-only-dependent, before submitting the report to the CI system. I do not know if this change of the report was intentional or a side effect, but my opinion on the matter is that it is best if in the report the paths are relative to the project root directory.

Remaining issues

To make an order in my plea, here I list what I think remains to be implemented to use shellspec without a ctest wrapper (which for now is actually an acceptable solution and seems to work quite fine for me) after the bugs in the working directory treatment are fixed:

• ctest-like formatter suppressing actual test report in favour of a terse information which test is currently running and whether it fails or passes to be used in conjunction with a generator which stores a detailed report in a file
• I realised that while the ctest-like formatter may skip any timing information, the jUnit xml report should contain it, but currently one cannot run the profiler with --jobs > 1 Enable profiler for --jobs > 1 #145
• there should be a possibility to suppress certain kinds of warnings Add flag to suppress warnings #122 or reduce their verbosity Add flag to suppress warnings #122 (comment)
• it would be useful to be able to assert equality of output files (and possibly the stdout/stderr) with a reference file in such a way that DSL for this is concise and the diff in case of a failure goes to the report Add a matcher for file contents equality/difference #138

[ko1nksm]

Problems with -C @specfile

Fixed in 6813789.

The problem was --default-path .. In the middle of responding to this issue, I have found that various bugs occur when specified the specfile path with absolute or containing "./", "../".

So I changed it to normalize to a relative path and then process it, but I needed to make an exception if it was the same path as the project root directory (i.e. .).

Do you have integration tests for shellspec?

Because the integration tests are so slow, I mainly test with unit tests. The equivalent of integration testing is done here in a separate form from the main tests for shellspec, but only slightly.
(BTW, testing all the shells using docker would take more than two hours.)

Currently, I am expanding the scope of the unit test a bit. Eventually, I will test the parts that cannot be tested in the unit test with the integration test.

Problems with #135 and the associated documentation

Remaining issues

I'll comment later. For now I have fixed the bug.

[AntoniMarcinek]

Thank you very much! I confirm the bug is fixed. I see that also the paths in the jUnit report are fine and that you improved the description of --execdir option.

[ko1nksm]

@AntoniMarcinek

I have also another problem with the changes of the #135: now the project explicitly requires the .shellspec

For compatibility, I changed it to a warning rather than an error in the next release. 7d6c666.
The documentation is still missing, but will be added in #147.

.shellspec file - in my case in my home directory

Renamed the global options file from $HOME/.shellspec to $HOME/.shellspec-options to avoid confusion with the project options file (#152). For compatibility purposes, $HOME/.shellspec will still be loaded, but it will be obsolete in future releases.

[AntoniMarcinek]

@ko1nksm Thanks, looks fine!