Behat broken logs with --list-scenarios

[kristijanhusak]

Hi,

I am using your behat extension for testing, and later building logs because i'm using Jenkins CI.
Command is this:

./bin/behat -c app/config/behat.yml "@AppWebBundle" --list-scenarios |
    ./bin/fastest --verbose --preserve-order
    "./bin/behat --format=junit --out=build/logs/behat -c app/config/behat.yml -v {};

Tests pass, but when logs are built they don't have proper content. Examples:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite errors="0" failures="0" skipped="0" name="User can reset his password in case he forgets it." tests="1" time="5.978331">
    <testcase classname="User can reset his password in case he forgets it." name="User exists in the system" time="3.853432" assertions="8">
    </testcase>
</testsuite>
suite>

<?xml version="1.0" encoding="UTF-8"?>
<testsuite errors="0" failures="0" skipped="0" name="Filter users" tests="1" time="13.394022">
    <testcase classname="Filter users" name="Show all users" time="11.175531" assertions="7">
    </testcase>
</testsuite>
uite>
testcase>
</testsuite>

From 76 features, i get this bug for around 30 features, and this breaks my build on Jenkins. When i use
--list-features everything works fine.

Do you know what can be a problem maybe? Thanks.

[liuggio]

Hi thanks for reporting,
@diegosainz please could you help us?

[diegosainz]

Sure, I'm out of the city but as soon as I get back (next week) will look
into it.
On Nov 27, 2014 2:24 AM, "Giulio De Donato" notifications@github.com
wrote:

Hi thanks for reporting,
@diegosainz https://github.com/diegosainz please could you help us?

—
Reply to this email directly or view it on GitHub
#20 (comment).

[wcluijt]

I think I may have come across this same issue with JUnit log files, but it has to do more with how the Behat JUnitFormatter generates the log file path in its beforeFeature() method, not necessarily what fastest is doing (other than parallel processing). Essentially, you need to provide a unique directory path to the --out parameter. Otherwise, multiple parallel tests will output to the exact same file path for an individual feature file and overwrite each others content (in the examples you pasted and the ones that I've seen that are similar, the content was not overwritten completely).

The exact command I use is this:

#!/bin/sh -x

mkdir -p app/build/behat_report;
mkdir -p app/build/logs/junit/parallel;
mkdir -p app/build/logs/coverage;

bin/behat --list-scenarios -p continuous_integration src | \
    bin/fastest \
        -vvv \
        --preserve-order \
        --rerun-failed \
        'php app/console doctrine:database:drop --env=test --force; \
        php app/console doctrine:database:create --env=test; \
        php app/console doctrine:schema:update --env=test --force; \
        feature_name=`echo $ENV_TEST_ARGUMENT | sed 's/[\/\.\:]/_/g'`;
        BEHAT_PARAMS="extensions[VIPSoft\\CodeCoverageExtension\\Extension][report][options][target]=app/build/logs/coverage/behat_parallel_${feature_name}.cov" \
        bin/behat \
            --format="progress,junit,html" \
            --out="null,app/build/logs/junit/parallel/${feature_name},app/build/behat_report/behat_report_parallel_${feature_name}.html" \
            -p continuous_integration \
            {} \
        ;' \
;

The key part here is the feature_name variable. It takes the ENV_TEST_ARGUMENT value, which should include the entire feature file path plus the line number of the scenario, and changes slashes, periods, and colons to underscores. This provides a unique enough value for the feature file and the specific scenario that is being executed which could be used as a directory name.

So to adapt to your specific command, I might try something like this:

./bin/behat -c app/config/behat.yml "@AppWebBundle" --list-scenarios |
    ./bin/fastest --verbose --preserve-order
    "feature_name=`echo $ENV_TEST_ARGUMENT | sed 's/[\/\.\:]/_/g'`; ./bin/behat --format=junit --out=build/logs/behat/${feature_name} -c app/config/behat.yml -v {};

Then in your Jenkins configuration, you could change the JUnit report path to something like build/logs/behat/**/*.xml so that it can account for the new sub-directories and find the generated JUnIt *.xml files.

[kristijanhusak]

@wcluijt
Thanks, i will try it.

[diegosainz]

@wcluijt excellent post - @kristijanhusak is this still an issue so I can look into it?

[diegosainz]

@liuggio I think this issue can be closed as resolved - we have been running tests this way for a while now and Jenkins seems to work OK with the junit output.

[kristijanhusak]

I haven't tested @wcluijt solution, stayed with the testing by feature. IF @diegosainz says it is working, it can be closed.

[liuggio]

thanks