New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTML Report: Failure snapshots are overwritten when using Cest with examples #4672

Closed
Tenzian opened this Issue Dec 9, 2017 · 0 comments

Comments

Projects
None yet
1 participant
@Tenzian
Contributor

Tenzian commented Dec 9, 2017

What are you trying to achieve?

Create a test using a Cest with examples to iterate the same test over several scenarios and generate an HTML report with meaningful snapshots of the failed pages.

What do you get instead?

Snapshot files for the failures in _output are overwritten for each failing iteration of the examples.

All the HTML snapshot links in the report end up pointing to the same FailsCest.staticPages.fail.html file generated for the final failure.

Example Screenshots

HTML Report:
screen shot 2017-12-09 at 00 30 56
Snapshot:
screen shot 2017-12-09 at 00 32 01


Suspected Cause

In several places the code assumes that each test has a unique signature.

However, when examples are used, all iterations of the Cest method have the same test signature which causes the snapshots for each failure to have the same filename.

You can see in _failed the filename for the page source is being created using Descriptor::getTestSignature($test) in the following examples:

public function _failed(TestInterface $test, $fail)
{
if (!$this->client || !$this->client->getInternalResponse()) {
return;
}
$filename = preg_replace('~\W~', '.', Descriptor::getTestSignature($test));
$filename = mb_strcut($filename, 0, 244, 'utf-8') . '.fail.html';
$this->_savePageSource($report = codecept_output_dir() . $filename);
$test->getMetadata()->addReport('html', $report);
}

public function _failed(TestInterface $test, $fail)
{
$this->debugWebDriverLogs($test);
$filename = preg_replace('~\W~', '.', Descriptor::getTestSignature($test));
$outputDir = codecept_output_dir();
$this->_saveScreenshot($report = $outputDir . mb_strcut($filename, 0, 245, 'utf-8') . '.fail.png');
$test->getMetadata()->addReport('png', $report);
$this->_savePageSource($report = $outputDir . mb_strcut($filename, 0, 244, 'utf-8') . '.fail.html');
$test->getMetadata()->addReport('html', $report);
$this->debug("Screenshot and page source were saved into '$outputDir' dir");
}


Provide console output if related. Use -vvv mode for more details.

$ codecept run acceptance --html
Codeception PHP Testing Framework v2.3.6
Powered by PHPUnit 6.4.4 by Sebastian Bergmann and contributors.

Acceptance Tests (3) ---------------------------------------------------------------------------------------------------------------
✖ FailsCest: Static pages | "http://www.google.com","Giraffe" (2.43s)
✖ FailsCest: Static pages | "http://www.php.net","Elephant" (1.09s)
✖ FailsCest: Static pages | "http://codeception.com","Zebra" (1.23s)
------------------------------------------------------------------------------------------------------------------------------------


Time: 4.82 seconds, Memory: 10.00MB

There were 3 failures:

---------
1) FailsCest: Static pages | "http://www.google.com","Giraffe"
 Test  tests/acceptance/FailsCest.php:staticPages
 Step  See "Giraffe","h1"
 Fail  Element located either by name, CSS or XPath element with 'h1' was not found.

Scenario Steps:

 2. $I->see("Giraffe","h1") at tests/acceptance/FailsCest.php:13
 1. $I->amOnPage("http://www.google.com") at tests/acceptance/FailsCest.php:12


---------
2) FailsCest: Static pages | "http://www.php.net","Elephant"
 Test  tests/acceptance/FailsCest.php:staticPages
 Step  See "Elephant","h1"
 Fail  Element located either by name, CSS or XPath element with 'h1' was not found.

Scenario Steps:

 2. $I->see("Elephant","h1") at tests/acceptance/FailsCest.php:13
 1. $I->amOnPage("http://www.php.net") at tests/acceptance/FailsCest.php:12


---------
3) FailsCest: Static pages | "http://codeception.com","Zebra"
 Test  tests/acceptance/FailsCest.php:staticPages
 Step  See "Zebra","h1"
 Fail  Failed asserting that any element by 'h1' 
+ <h1>
    <img alt="" class="logo" src="/images/logo.svg"></img>
    Codeception</h1>
contains text 'Zebra'

Scenario Steps:

 2. $I->see("Zebra","h1") at tests/acceptance/FailsCest.php:13
 1. $I->amOnPage("http://codeception.com") at tests/acceptance/FailsCest.php:12


FAILURES!
Tests: 3, Assertions: 3, Failures: 3.
- HTML report generated in file:///Users/Tenz/Repositories/demo/tests/_output/report.html

Provide test source code if related

<?php
namespace { # global namespace

    class FailsCest
    {

       /**
        * @dataprovider pageProvider
        */
        public function staticPages(AcceptanceTester $I, \Codeception\Example $example)
        {
            $I->amOnPage($example['url']);
            $I->see($example['title'], 'h1');
            $I->seeInTitle($example['title']);
        }

        /**
         * @return array
         */
        protected function pageProvider()
        {
            return [
                [
                    'url'   => "http://www.google.com",
                    'title' => "Giraffe"
                ],
                [
                    'url'   => "http://www.php.net",
                    'title' => "Elephant"
                ],
                [
                    'url'   => "http://codeception.com",
                    'title' => "Zebra"
                ]
            ];
        }
    }
}

Details

  • Codeception version: 2.3.6 (@ 75a1f48)
  • PHP Version: 7.1.12
  • Operating System: macOS 10.13.2
  • Installation type: Composer
  • List of installed packages
behat/gherkin                      v4.4.5 Gherkin DSL parser for PHP 5.3
codeception/specify                0.4.6  BDD code blocks for PHPUnit and Codeception
doctrine/instantiator              1.1.0  A small, lightweight utility to instantiate objects in PHP without invoking their con...
facebook/graph-sdk                 5.6.1  Facebook SDK for PHP
facebook/webdriver                 1.5.0  A PHP client for Selenium WebDriver
flow/jsonpath                      0.3.4  JSONPath implementation for parsing, searching and flattening arrays
fzaninotto/faker                   v1.5.0 Faker is a PHP library that generates fake data for you.
guzzlehttp/guzzle                  6.3.0  Guzzle is a PHP HTTP client library
guzzlehttp/promises                v1.3.1 Guzzle promises library
guzzlehttp/psr7                    1.4.2  PSR-7 message implementation that also provides common utility methods
league/factory-muffin              v3.0.1 The goal of this package is to enable the rapid creation of objects for the purpose o...
league/factory-muffin-faker        v1.0.0 The goal of this package is to wrap faker to make it super easy to use with factory m...
monolog/monolog                    1.23.0 Sends your logs to files, sockets, inboxes, databases and various web services
myclabs/deep-copy                  1.7.0  Create deep copies (clones) of your objects
pda/pheanstalk                     v3.1.0 PHP client for beanstalkd queue
phar-io/manifest                   1.0.1  Component for reading phar.io manifest information from a PHP Archive (PHAR)
phar-io/version                    1.0.1  Library for handling version information and constraints
php-amqplib/php-amqplib            v2.7.0 Formerly videlalvaro/php-amqplib.  This library is a pure PHP implementation of the A...
phpdocumentor/reflection-common    1.0.1  Common reflection classes used by phpdocumentor to reflect the code structure
phpdocumentor/reflection-docblock  4.2.0  With this component, a library can provide support for annotations via DocBlocks or o...
phpdocumentor/type-resolver        0.4.0 
phpspec/prophecy                   1.7.3  Highly opinionated mocking framework for PHP 5.3+
phpunit/php-code-coverage          5.2.4  Library that provides collection, processing, and rendering functionality for PHP cod...
phpunit/php-file-iterator          1.4.5  FilterIterator implementation that filters files based on a list of suffixes.
phpunit/php-text-template          1.2.1  Simple template engine.
phpunit/php-timer                  1.0.9  Utility class for timing
phpunit/php-token-stream           2.0.2  Wrapper around PHP's tokenizer extension.
phpunit/phpunit                    6.4.4  The PHP Unit Testing framework.
phpunit/phpunit-mock-objects       4.0.4  Mock Object library for PHPUnit
predis/predis                      v1.1.1 Flexible and feature-complete Redis client for PHP and HHVM
psr/http-message                   1.0.1  Common interface for HTTP messages
psr/log                            1.0.2  Common interface for logging libraries
sebastian/code-unit-reverse-lookup 1.0.1  Looks up which function or method a line of code belongs to
sebastian/comparator               2.1.0  Provides the functionality to compare PHP values for equality
sebastian/diff                     2.0.1  Diff implementation
sebastian/environment              3.1.0  Provides functionality to handle HHVM/PHP environments
sebastian/exporter                 3.1.0  Provides the functionality to export PHP variables for visualization
sebastian/global-state             2.0.0  Snapshotting of global state
sebastian/object-enumerator        3.0.3  Traverses array structures and object graphs to enumerate all referenced objects
sebastian/object-reflector         1.1.1  Allows reflection of object attributes, including inherited and non-public ones
sebastian/recursion-context        3.0.0  Provides functionality to recursively process PHP variables
sebastian/resource-operations      1.0.0  Provides a list of PHP built-in functions that operate on resources
sebastian/version                  2.0.1  Library that helps with managing the version number of Git-hosted PHP projects
squizlabs/php_codesniffer          2.9.1  PHP_CodeSniffer tokenizes PHP, JavaScript and CSS files and detects violations of a d...
symfony/browser-kit                v4.0.1 Symfony BrowserKit Component
symfony/console                    v4.0.1 Symfony Console Component
symfony/css-selector               v4.0.1 Symfony CssSelector Component
symfony/dom-crawler                v4.0.1 Symfony DomCrawler Component
symfony/event-dispatcher           v4.0.1 Symfony EventDispatcher Component
symfony/finder                     v4.0.1 Symfony Finder Component
symfony/polyfill-mbstring          v1.6.0 Symfony polyfill for the Mbstring extension
symfony/process                    v4.0.1 Symfony Process Component
symfony/yaml                       v4.0.1 Symfony Yaml Component
theseer/tokenizer                  1.1.0  A small library for converting tokenized PHP source code into XML and potentially oth...
vlucas/phpdotenv                   v2.4.0 Loads environment variables from `.env` to `getenv()`, `$_ENV` and `$_SERVER` automag...
webmozart/assert                   1.2.0  Assertions to validate method input/output with nice error messages.
  • Suite configuration:
actor: AcceptanceTester
modules:
  enabled:
    - PhpBrowser
  config:
    PhpBrowser:
      url: http://localhost/

Tenzian added a commit to Tenzian/Codeception that referenced this issue Dec 9, 2017

Test\Descriptor: getTestSignatureUnique
Fixes Codeception#4672

- add Descriptor::getTestSignatureUnique

Replace getTestSignature in implementations of PageSourceSaver & ScreenshotSaver
where filenames are being created

- use getTestSignatureUnique in Lib\InnerBrowser
- use getTestSignatureUnique in Module\WebDriver

DavertMik added a commit that referenced this issue Dec 9, 2017

Test\Descriptor: getTestSignatureUnique (#4673)
Fixes #4672

- add Descriptor::getTestSignatureUnique

Replace getTestSignature in implementations of PageSourceSaver & ScreenshotSaver
where filenames are being created

- use getTestSignatureUnique in Lib\InnerBrowser
- use getTestSignatureUnique in Module\WebDriver
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment