Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CakePHP2] Support for running phpunit directly? #12700

Closed
mfn opened this issue Nov 3, 2018 · 13 comments
Closed

[CakePHP2] Support for running phpunit directly? #12700

mfn opened this issue Nov 3, 2018 · 13 comments

Comments

@mfn
Copy link
Contributor

@mfn mfn commented Nov 3, 2018

This is a (multiple allowed):

  • enhancement

  • feature-discussion (RFC)

  • CakePHP Version: 2.x

  • Platform and Target: Any

It's not possible to run phpunit directly because of how CakePHP2 integrates with it. E.g. running it leads to:

app $ Vendor/bin/phpunit Test/
PHP Fatal error:  Class 'CakeTestSuite' not found in /vagrant/project/app/Test/Case/AllComponentTest.php on line 3

Rather, you have to run ./cake test app …

But this means you can't integrate it with tooling like IDE/PhpStorm which expect to run phpunit directly and pass it appropriate flags.

This is very useful for re-running partial tests (i.e. only the failed tests), etc.

There's a 5 year old blog post by Jetbrains talking about creating a wrapper and I also found more recent versions of this script but none works with the current PhpStorm (or at least I couldn't make them work).

Rather then this PhpStorm specific workaround I'm interested or searching for interested parties getting this working with CakePHP2.

I understand it's quite a challenge regarding the whole bootstrapping and fixture process.

@dereuromark dereuromark added this to the 2.10.14 milestone Nov 3, 2018
@ADmad
Copy link
Member

@ADmad ADmad commented Nov 3, 2018

As announced long ago CakePHP 2.x won't be receiving any new features. Hence closing the issue.

@ADmad ADmad closed this Nov 3, 2018
@tenkoma
Copy link
Member

@tenkoma tenkoma commented Dec 4, 2018

@mfn Hi.
I was thinking about the same thing.
With reference to Console/cake.php and ShellDispatchr, I prepared a bootstrap file for phpunit command and it worked, so I will write it down.
(re-running partial tests, run test with coverage, run test with debug)
I think that it is safe unless it is used only in local development and not used in CI build.

Note: When the phpunit command is executed, You may get a "class not defined" error. Apply App::uses() appropriately.

I saw this Issue and tried it, it was a lot easier to write a test, thank you. 😄

bootstrap and phpunit.xml.dist

Define constants and read Config before the test run.

app/Test/bootstrap_for_phpunit_command.php

<?php
/**
 * Bootstrap for phpunit command
 */
/**
 * copy from app/Console/cake.php
 */
$dispatcher = 'Cake' . DS . 'Console' . DS . 'ShellDispatcher.php';
$root = dirname(dirname(dirname(__FILE__)));
$appDir = basename(dirname(dirname(__FILE__)));
$install = $root . DS . 'lib';
$composerInstall = $root . DS . $appDir . DS . 'Vendor' . DS . 'cakephp' . DS . 'cakephp' . DS . 'lib';
// the following lines differ from its sibling
// /lib/Cake/Console/Templates/skel/Console/cake.php
if (file_exists($composerInstall . DS . $dispatcher)) {
    $install = $composerInstall;
}
ini_set('include_path', $install . PATH_SEPARATOR . ini_get('include_path'));
if (!include $dispatcher) {
    trigger_error('Could not locate CakePHP core files.', E_USER_ERROR);
}
unset($dispatcher);
define('ROOT', $root);
define('APP_DIR', $appDir);
define('APP', ROOT . DS . APP_DIR . DS);
// Use methods that initialize constants and environment variables, but the shell does not in ShellDispatcher class.
new ShellDispatcher(array(getenv('_'), '-working', $appDir));
unset($root, $appDir, $install, $composerInstall);

// Required for setup FixtureManager.
App::uses('AppFixtureManager', 'TestSuite/Fixture');

// A class that does not dare to call App::uses() because it calls it anywhere on the application. I will write it as an error
App::uses('ClassRegistry', 'Utility');

app/Test/bootstrap.php

<?php
// Read additional bootstrap.php when executed from phpunit command.
if (!defined('DS')) {
    define('DS', DIRECTORY_SEPARATOR);
    include_once dirname(__FILE__) . DS . 'bootstrap_for_phpunit_command.php';
}

phpunit.xml.dist

Please specify this file path by setting in PhpStorm.
(Preferences > Languages & Frameworks > PHP > Test Frameworks > (Any Setting) > Test Runner > Default Configuration File)

<?xml version="1.0" encoding="UTF-8"?>
<phpunit
    colors="true"
    bootstrap="./app/Test/bootstrap.php"
    >
    <filter>
        <whitelist>
            <directory suffix=".php">app/Config</directory>
            <directory suffix=".php">app/Console</directory>
            <directory suffix=".php">app/Controller</directory>
            <directory suffix=".php">app/Lib</directory>
            <directory suffix=".php">app/Model</directory>
            <directory suffix=".php">app/Routing</directory>
            <directory suffix=".php">app/TestSuite</directory>
            <directory suffix=".php">app/View</directory>
        </whitelist>
    </filter>
</phpunit>

Code to prepare FixtureManager instead of CakeTestRunner

app/TestSuite/AppControllerTestCase.php

<?php
App::uses('ControllerTestCase', 'TestSuite');
class AppControllerTestCase extends ControllerTestCase
{
    public function run(PHPUnit_Framework_TestResult $result = null)
    {
        $this->setUpFixtureManagerForPhpunitCommand();
        return parent::run($result);
    }

    /**
    * When executed from the phpunit command, since FixtureManager is not prepared, it can be prepared with TestCase::run()
    */
    private function setUpFixtureManagerForPhpunitCommand()
    {
        if (is_null($this->fixtureManager)) {
            App::uses('AppFixtureManager', 'TestSuite');
            if (class_exists('AppFixtureManager')) {
                $this->fixtureManager = new AppFixtureManager();
            } else {
                App::uses('CakeFixtureManager', 'TestSuite/Fixture');
                $this->fixtureManager = new CakeFixtureManager();
            }
            $this->fixtureManager->fixturize($this);
        }
    }
}

app/TestSuite/AppTestCase.php

<?php
App::uses('CakeTestCase', 'TestSuite');
class AppTestCase extends CakeTestCase
{
    public function run(PHPUnit_Framework_TestResult $result = null)
    {
        $this->setUpFixtureManagerForPhpunitCommand();
        return parent::run($result);
    }

    /**
    * When executed from the phpunit command, since FixtureManager is not prepared, it can be prepared with TestCase::run()
    */
    private function setUpFixtureManagerForPhpunitCommand()
    {
        if (is_null($this->fixtureManager)) {
            App::uses('AppFixtureManager', 'TestSuite');
            if (class_exists('AppFixtureManager')) {
                $this->fixtureManager = new AppFixtureManager();
            } else {
                App::uses('CakeFixtureManager', 'TestSuite/Fixture');
                $this->fixtureManager = new CakeFixtureManager();
            }
            $this->fixtureManager->fixturize($this);
        }
    }
}

app/TestSuite/Fixture/AppFixtureManager.php (Optional)

<?php
App::uses('CakeFixtureManager', 'TestSuite/Fixture');
class AppFixtureManager extends CakeFixtureManager
{
}
@mfn
Copy link
Contributor Author

@mfn mfn commented Dec 6, 2018

@tenkoma this is incredible!

I only had to to copy&paste your stuff and it literally worked on the first try!

Additional changes I did:

  • in phpunit.dist.xml I added explicitly testsuites so I can run app/Vendor/bin/phpunit without any args
        <testsuites>
            <testsuite name="Application Test Suite">
               <directory>./app/Test/</directory>
            </testsuite>
        </testsuites>
  • very custom: I added an autoloader which recursively scans app/ and can load un-namespaced class names from any folder (performance vs. DX trade-off), see below for code
  • I used the opportunity and upgraded to PHPUnit 7 (namespaced, function type declarations added)
    => works, but had to assimilate CakeTestCase and CakeFixtureManager to adapt some function signatures

The end result is fantastic, because now I can use PhpStorms PHPUnit integration and am now much faster iterating the tests (let alone debugging, which you outlined already).

Thank you very much!

Autoloader code

Append to bootstrap.php

spl_autoload_register(function (string $classNameToFind) {
    static $cakeClassMap = [];

    if (!$cakeClassMap) {
        $findPhpFilesRecursively = function (string $dir) use (&$cakeClassMap): void {
            $dirIter = new RecursiveDirectoryIterator($dir);
            $resursiveIter = new RecursiveIteratorIterator($dirIter);
            $regexIter = new RegexIterator($resursiveIter, '/^.+\.php$/i', RecursiveRegexIterator::GET_MATCH);
            foreach ($regexIter as $file) {
                $file = $file[0];

                // Ignore `Vendor` files, they've their own resolution already
                if (false !== strpos($file, 'Vendor')) {
                    continue;
                }

                $className = str_replace('.php', '', basename($file));
                $cakeClassMap[$className] = $file;
            }
        };

        // Cake itself
        $findPhpFilesRecursively(ROOT . DIRECTORY_SEPARATOR . 'lib');
        // Project specific files; will overwrite clashes with CakePHP!
        $findPhpFilesRecursively(ROOT . DIRECTORY_SEPARATOR . APP_DIR);
    }

    if (isset($cakeClassMap[$classNameToFind])) {
        require_once $cakeClassMap[$classNameToFind];

        return null;
    }

    return false;
});
@kamilwylegala
Copy link

@kamilwylegala kamilwylegala commented Dec 11, 2018

@tenkoma @mfn

What version of PHP do you use?

I use similar setup with PHPUnit 6.* but I gave up using anything related to framework (ControllerTest case etc.) and I also use composer's autoloader for domain specific classes.

@mfn
Copy link
Contributor Author

@mfn mfn commented Dec 11, 2018

@kamilwylegala

What version of PHP do you use?

PHP 7.2

I use similar setup with PHPUnit 6.* but I gave up using anything related to framework (ControllerTest case etc.) and I also use composer's autoloader for domain specific classes.

Fair question: I don't have Controller tests and too use composers autoloader for domain specific classes.

Tried to write controller tests but they turned out too complex or simply not possible properly to mock certain parts so I moved the most stuff into domain classes or the models. Yay for fat models 👎 but turns our due to ClassRegistry::init() you get a test-ready version for it in tests which makes it a breeze to work with.

Mocking is then mostly done in the domain specific classes which can use better patterns (DI, for one).

PHPUnit 6.*

I got above setup by @tenkoma running in 5.* but immediately took the opportunity and upgraded to 7.* by basically copying CakeTestCase and CakeFixtureManager into the app/ and adapting parts (mostly minor function signature adaptions).

The specific project I'm working on is very old but we realized the test shortcomings years ago and thus changed the way we approach creating new code and basically moved off of everything too CakePHP specific 🤷‍♀️

@tenkoma
Copy link
Member

@tenkoma tenkoma commented Dec 16, 2018

@kamilwylegala

What version of PHP do you use?

I'm using PHP7.1 .
PHPUnit is 5.7, CakePHP is 2.10.
In my environment,I installed CakePHP using composer, I use composer autoload in library loading. However, Namespace is not declared in the class definition under the app/ directory.

@mfn
Copy link
Contributor Author

@mfn mfn commented Feb 2, 2019

After two months I would like to share for anyone interested more on how I was able to improve the CakePHP2 PHPUnit integration; I cannot thank @tenkoma enough for coming up with the solution in #12700 (comment)

Notes upfront: I'm not using every aspect of CakePHP2 in my tests; although I've >1k tests, I don't test Controllers or Behaviours for example. Only code in models and custom Service/Repository classes

Fix stack trace to report actual line of assertion failures

For the longest time, when having to refactor code and I want to go quickly to a massive number of failed tests due to this and fix them, the report stacktrace was always ending up in the custom \CakeTestCase::run method:

Failed asserting that 1 is identical to 2.
 /vagrant/project/app/Test/Case/Lib/CustomTest.php:227
 /vagrant/project/app/Vendor/cakephp/cakephp/lib/Cake/TestSuite/CakeTestCase.php
 /vagrant/project/app/Test/TestCase.php:308

The last two lines here are due to the presence of the run method in both cases.

By changing the following:

  • move the fixture manager loading from run to setUp
  • respectively move the unloading from run to tearDown
  • (and implicitly remove the rest what happens in run, i.e. the ob_level* stuff

I was able to this stack trace:

Failed asserting that 1 is identical to 2.
 /vagrant/project/app/Test/Case/Lib/CustomTest.php:227

What's the big deal?

As mention, having to go through a lot of tests using an IDE, I can jump to the next failed assertion with a keystroke which now happens to be exactly the failed assertion line and not the globally used run method. I can say from my experience, this was a massive improvement for me.

Improve runtime of integration tests by over 50% by isolating tests in transactions

The default behaviour of the whole fixture setup is constantly create and try to create tables from the references fixtures (it's possible to opt to not drop/truncate them).

But still, when having a lot of tests this looses unnecessary runtime.

I was able to massively improve this with a few workarounds and some constraints:

  • on the apps TestCase the $fixtures array is defined with a list of all models used in tests but having no data to actual fixturize. Effectively for every model you want to use you define a separate EmptyModelPostFixture. It smells noise at first, but once generated you usually don't have to care about this
    • the actual fixturize data is still left in each tests because it's still consumed
  • ensure to enable \DboSource::$useNestedTransactions
  • ensure (using a global test flag for example), the tables for the empty models are only initialized (really: created) once
  • don't drop tables after each test
  • but wrap each test in a commit/rollback transaction (via setUp/tearDown)

The speed up gained from this has blown me away. A test suite running ~3m30s before was done in 50s.

Final notes

It's unfortunately hard to provide source code here because the CakeFixtureManager in fact had to be cloned/stripped down and the CakeTestCase class is not used at all anymore, but selected calls to the assimilated FixtureManager are included.

Also, another interesting fact: PHPUnit 8 was released recently and by not depending on CakeTestCase anymore I can happily report that it worked with only minor adjustments (i.e. the documented deprecations, really).

I already thanked @tenkoma but also would like to praise the CakePHP project/team/community for still fixing PHP 7.x compatibility bugs. I understand this will stop at some point and I'm actively working on moving away from CakePHP2 but there's still some work left.

@mfn
Copy link
Contributor Author

@mfn mfn commented Nov 28, 2019

@tenkoma to you remember when you posted #12700 (comment) why you did add getenv('_')?

new ShellDispatcher(array(getenv('_'), '-working', $appDir));

I'm asking because I'm testing with PHP 7.4 and I get the following error:
PHP Notice: Trying to access array offset on value of type bool in /vagrant/project/app/Vendor/cakephp/cakephp/lib/Cake/Console/ShellDispatcher.php on line 306
from this line:

		if (!empty($params['working']) && (!isset($this->args[0]) || isset($this->args[0]) && $this->args[0][0] !== '.')) {

Fix looks easy, just some is_array check, but I'd like to also understand the reason behind this?

thx!

@mfn
Copy link
Contributor Author

@mfn mfn commented Nov 29, 2019

After some tinkering I came to the conclusion that the first arg to the ShellDispatcher is simply the executing script, so I'll use this diff for now to silence this notice:

-new ShellDispatcher([getenv('_'), '-working', $appDir]);
+$command = $_SERVER['argv'][0];
+new ShellDispatcher([$command, '-working', $appDir]);
@ZeoKnight
Copy link

@ZeoKnight ZeoKnight commented Jun 16, 2020

@mfn @tenkoma thanks for your continued support on this - if you could provide source for the changes to CakeFixtureManager and CakeTestCase that would be really helpful!

@mfn
Copy link
Contributor Author

@mfn mfn commented Jun 16, 2020

@ZeoKnight

I'll try my best but in my company, we've changed a lot of the test infrastructure to depend less on Cake provided features as they simply don't match our "software reality" in 2020.

I replaced the CakeFixtureManager with our "own" which only has one purpose: setting up the tables.

I do not use the feature of fixtures itself at all, because they're impractical to handle at scale and instead we use our own custom code to create necessary database entries on demand within each test. This also means it's important that the test code runs "protected" in database transactions (so DB gets reset after each test), which might not always be desired. But for us it was just a "practical" choice.

Insofar, I also don't use CakeTestCase; at all. Instead I just extend from PHPUnit\Framework\TestCase and the skeleton we use looks something like this:

    protected function setUp(): void
    {
        // This uses our custom fixture code
        $this->setupCakeDatabaseOnce();

        parent::setUp();

        $this->beginCakeTransaction();
    }

    protected function tearDown(): void
    {
        $this->rollbackCakeTransaction();

        $this->tearDownClassRegistry();

        parent::tearDown();
    }

    private function tearDownClassRegistry(): void
    {
        ClassRegistry::flush();
    }

// …
    private function setupCakeDatabaseOnce(): void
    {
        if (static::$databaseSetupDone) {
            return;
        }

        $fixtureManager = new FixtureManager();
        $fixtureManager->createTables(static::$emptyModelFixtures);

        static::$databaseSetupDone = true;
    }
    private function beginCakeTransaction(): void
    {
        $db = ConnectionManager::getDataSource('test');
        $db->begin();
        // repeat this for every datasource
    }

    private function rollbackCakeTransaction(): void
    {
        $db = ConnectionManager::getDataSource('test');
        $db->rollback();
        // repeat this for every datasource
    }

static::$emptyModelFixtures is a shallow list for all the tables we want to have available in tests (in our case: all…) so it looks like this:

    protected static $emptyModelFixtures = [
        'app.EmptyModels/EmptyModelsPost,
        // list goes on for each table/model
    ];

And as an example, the fixture for above code looks like this:

<?php declare(strict_types = 1);

class EmptyModelsPostFixture extends CakeTestFixture
{
    public $import = Post::class;
}

Here's the fixture manager class; it's blatant rip of the existing one, stripped down to bare bones. Where I can imagine above TestCase skeleton is useful, this one was hacked together until it worked and never looked back. I tried to read our own git history to make sense, but it just doesn't; sorry :/
(it still depends on \CakeTestFixture which was not changed AFAIK)

<?php declare(strict_types = 1);

use App;
use CakeTestFixture;
use ClassRegistry;
use ConnectionManager;
use DataSource;
use Inflector;
use UnexpectedValueException;

/*
 * A factory class to manage the life cycle of test fixtures
 *
 * CakePHP(tm) : Rapid Development Framework (https://cakephp.org)
 * Copyright (c) Cake Software Foundation, Inc. (https://cakefoundation.org)
 *
 * Licensed under The MIT License
 * For full copyright and license information, please see the LICENSE.txt
 * Redistributions of files must retain the above copyright notice.
 *
 * @copyright     Copyright (c) Cake Software Foundation, Inc. (https://cakefoundation.org)
 * @link          https://cakephp.org CakePHP(tm) Project
 * @package       Cake.TestSuite.Fixture
 * @since         CakePHP(tm) v 2.0
 * @license       https://opensource.org/licenses/mit-license.php MIT License
 */

App::uses('ConnectionManager', 'Model');
App::uses('ClassRegistry', 'Utility');

/**
 * A factory class to manage the life cycle of test fixtures
 */
class FixtureManager
{
    /**
     * Holds the fixture classes that where instantiated
     *
     * @var array
     */
    protected $_loaded = [];

    /**
     * Parse the fixture path included in test cases, to get the fixture class name, and the
     * real fixture path including sub-directories
     *
     * @param string $fixturePath the fixture path to parse
     * @return array containing fixture class name and optional additional path
     */
    private function parseFixturePath($fixturePath): array
    {
        $pathTokenArray = explode('/', $fixturePath);
        $fixture = array_pop($pathTokenArray);
        $additionalPath = '';
        foreach ($pathTokenArray as $pathToken) {
            $additionalPath .= DS . $pathToken;
        }

        return ['fixture' => $fixture, 'additionalPath' => $additionalPath];
    }

    /**
     * Looks for fixture files and instantiates the classes accordingly
     *
     * @param array $fixtures the fixture names to load using the notation {type}.{name}
     * @return void
     */
    private function loadFixtures($fixtures): void
    {
        foreach ($fixtures as $fixture) {
            if (isset($this->_loaded[$fixture])) {
                continue;
            }

            $fixtureFile = null;
            $fixtureIndex = $fixture;

            $fixturePrefixLess = substr($fixture, strlen('app.'));
            $fixtureParsedPath = $this->parseFixturePath($fixturePrefixLess);
            $fixture = $fixtureParsedPath['fixture'];
            $fixturePaths = [
                TESTS . 'Fixture' . $fixtureParsedPath['additionalPath'],
            ];

            $loaded = false;
            foreach ($fixturePaths as $path) {
                $className = Inflector::camelize($fixture);
                if (is_readable($path . DS . $className . 'Fixture.php')) {
                    $fixtureFile = $path . DS . $className . 'Fixture.php';
                    require_once $fixtureFile;
                    $fixtureClass = $className . 'Fixture';
                    $this->_loaded[$fixtureIndex] = new $fixtureClass();
                    $loaded = true;
                    break;
                }
            }

            if (!$loaded) {
                $firstPath = str_replace([
                    APP,
                    CAKE_CORE_INCLUDE_PATH,
                    ROOT,
                ], '', $fixturePaths[0] . DS . $className . 'Fixture.php');

                throw new UnexpectedValueException(__d('cake_dev', 'Referenced fixture class %s (%s) not found', $className, $firstPath));
            }
        }
    }

    /**
     * Runs the drop and create commands on the fixtures if necessary.
     *
     * @param CakeTestFixture $fixture the fixture object to create
     * @param DataSource $db the datasource instance to use
     * @return void
     */
    private function setupTable($fixture, $db): void
    {
        $sources = (array) $db->listSources();
        $table = $db->config['prefix'] . $fixture->table;
        $exists = in_array($table, $sources);

        if ($exists) {
            $fixture->drop($db);
        }

        $fixture->create($db);
    }

    public function createTables(array $fixtures): void
    {
        ClassRegistry::config(['ds' => 'test', 'testing' => true]);
        $db = ConnectionManager::getDataSource('test');
        $db->cacheSources = false;

        $this->loadFixtures($fixtures);

        foreach ($this->_loaded as $name => $fixture) {
            $db = ConnectionManager::getDataSource($fixture->useDbConfig);
            $this->setupTable($fixture, $db);
        }

        $this->_loaded = [];
    }
}

I just saw I also adapted the bootstrap_for_phpunit_command.php script:

<?php
/**
 * Bootstrap for phpunit command
 */
/**
 * copy from app/Console/cake.php
 */
$dispatcher = 'Cake' . DS . 'Console' . DS . 'ShellDispatcher.php';
$root = dirname(dirname(dirname(__FILE__)));
$appDir = basename(dirname(dirname(__FILE__)));
$install = $root . DS . 'lib';
$composerInstall = $root . DS . $appDir . DS . 'Vendor' . DS . 'cakephp' . DS . 'cakephp' . DS . 'lib';
// the following lines differ from its sibling
// /lib/Cake/Console/Templates/skel/Console/cake.php
if (file_exists($composerInstall . DS . $dispatcher)) {
    $install = $composerInstall;
}
ini_set('include_path', $install . PATH_SEPARATOR . ini_get('include_path'));
if (!include $dispatcher) {
    trigger_error('Could not locate CakePHP core files.', E_USER_ERROR);
}
unset($dispatcher);
define('ROOT', $root);
define('APP_DIR', $appDir);
define('APP', ROOT . DS . APP_DIR . DS);
// Use methods that initialize constants and environment variables, but the shell does not in ShellDispatcher class.
$command = $_SERVER['argv'][0];
new ShellDispatcher([$command, '-working', $appDir]);
unset($root, $appDir, $install, $composerInstall);

// Emulate \TestShell::_run which unsets any configured error handlers
restore_error_handler();
restore_error_handler();

This might all read a bit unorganized, apologies: it was ~2 years ago and we didn't bother documented this more as the goal was to get it working as fast a possible and don't look back. :-)

@ZeoKnight
Copy link

@ZeoKnight ZeoKnight commented Jun 17, 2020

@mfn Much appreciated! We are in the same position; enterprise application with 10k+ tests but stuck inside the cake nastiness - this will a lot!

@cjonstrup
Copy link

@cjonstrup cjonstrup commented Oct 5, 2020

I have made a quick example how I do it https://github.com/cjonstrup/cake-2-with-unittest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants