Should this project support prebuilt fixtures?

[kayue]

In the README file, @nb said that this project "Doesn't encourage or support the usage of Shared/Prebuilt fixtures". I am wondering what leads to this decision.

I found this project very useful to be used in WordPress theme development. However, some WordPress themes are tightly integrated with plugins (BBPress and BuddyPress for example), and I think we can not test a theme like that without prebuilt fixtures support. In fact, I couldn't even pre-activate a theme without it.

Is it something worth to look into? Or am I doing it wrong?

Thanks.

PS: I made a unit test theme demo in here: https://github.com/kayue/twentytest

[scribu]

Every test already has a giant shared fixture: the WordPress environment, along with it's database, loaded using the --bootstrap option, so I'm not sure how this would be implemented.

Caveat: I'm too much of a noob when it comes to unit testing to have an opinion about whether non-fresh fixtures are a bad idea or not.

[kayue]

After reading the PHPUnit documents, I think the job of wordpress-tests is to provide a clean WordPress environment, and other tasks like activating themes and plugins should be done in test suite's setUp() parse.

This can be done by overriding the WP_UnitTestCase class.

[kayue]

Just noted that theme can not be activated after WordPress loaded (a page refresh is required).

[nb]

This project "Doesn't encourage or support the usage of Shared/Prebuilt fixtures". I am wondering what leads to this decision.

I've made the mistake of using shared and prebuilt fixtures twice (Rails and early GlotPress) and here is why I prefer not to use it:

• If you're using shared fixture, changing the fixture may break other test in totally unexpected ways. For example one test may think the user with ID 1 is called admin, until we change the fixture and many tests break.
• If you're using a prebuilt fixture the data is very far away from the test and this harms readability.
• If you're using prebuilt fixture, you need to pollute it with all sorts of crazy combinations. For example we need to test what happens if a user is both a woman, under 18, and admin, and likes jazz music. Imagine how many of those we need in the fixture. And if we sometimes use the prebuilt fixture and sometimes not, it's an inconsistent mess.
• If you're using a prebuilt fixture and you need to test with an empty database you need to delete stuff, which is lame.
• Loading more information that you need is slow. Re-creating the data every time I need it is in my experience faster, though I don't have data to back it up, so I wouldn't call this a major drawback.
• Starting with a clean slate every time allows us to do the ROLLBACK trick and make the initialization very fast.

@kayue, in case like yours, when your clean slate is different from a vanilla WordPress install, it's perfectly fine to add the Budy/bbPress-specific initialization in your setUp(). This brings none of the problems I described above.

On plugin/theme testing: I would appreciate if you look into having an equivalent of setting $this->plugin_slug for themes. This way you won't need to do that manually in your tests.

[kayue]

I looked into the WP_UnitTestCase class and saw that it is calling activate_plugin() in PHPUnit's set up phrase. I don't even think we should do that because by this point we already missed all the init, wp actions, and it might give us an unexpected result.

In order to create a "real" WordPress environment, I think the proper way to activate a plugin is to modify the result returned by wp_get_active_and_valid_plugins() function that called in wp-settings.php

Below is a code fragment from wp-settings.php and wp-include/load.php

<?php 
// Load active plugins.
foreach ( wp_get_active_and_valid_plugins() as $plugin )
    include_once( $plugin );

<?php 
function wp_get_active_and_valid_plugins() {
    $plugins = array();
    $active_plugins = (array) get_option( 'active_plugins', array() );

    ...

    return $plugins;
}

We can modify the wp_get_active_and_valid_plugins() result by adding a filter to option_active_plugins. Usually we won't be able to do this because it is not possible to do so in a plugin or theme's functions.php.

So now the question is how to make it possible with minimal hard coding. Basically there are 2 ways: modify the default-filters.php, or use SHORTINIT to stop most of WordPress from being loaded until we have our filters / actions added.

Regarding equivalent of activate_plugin(), WordPress has switch_theme() function. Someone wrote a nice article about it in Switch_Theme vs. Theme Switching

[johnpbloch]

Personally, I think the easiest way to get a plugin into wp_get_active_and_valid_plugins() in this case would be to hijack WPMU_PLUGINS_DIR and set it to a subdirectory in wordpress-tests. In this directory would be at least two files: one to add filters to wp_get_active_and_valid_plugins() (based on which tests are currently running, etc.), and the other to find the real mu-plugins directory and include all of its files. This might involve using a special constant, like WPMU_REAL_PLUGINS_DIR.

Of course, the problem with that approach is that now the plugin is running amok (untested!) before PHPUnit will ever start running any tests.

Tests really don't need to be testing whether actions and filters run correctly. Those are tests for WordPress itself, not your plugin. Your plugin should be able to use WordPress' internal APIs without needing to test them. Your tests should be calling those functions directly to test their effects.

[kayue]

I agree with @johnpbloch at some degree that unit test should only test a single function.

However, if you want to test a WordPress theme, then supporting actions and filter is necessary because it can changes the behavior completely. I think what I want is more like a behavior test.

Anyway, I have managed to get a plugin into wp_get_active_and_valid_plugins() without touching the core. A pull request has been submitted. Please review it at #19.

[scribu]

How about this:

• load init.php
• run SHORTINIT procedure
• load lib/*
• {phpunit loads the testcase}
• WP_UnitTestCase child class adds the early filters
• WP_UnitTestCase loads the rest of WP (if needed)

[scribu]

Also, I think it would be good if we have intermediate WP_PluginTestCase and WP_ThemeTestCase classes, instead of stuffing everything into a single class.

[kayue]

I thought about this too but I don't think we can load the WordPress core in individual test case because it can only be loaded once. Both setUpBeforeClass or setUp might be called multiple times in one test.

I am not sure is this something solvable.

[scribu]

We can abort if it's already loaded:

function setUp() {
  ...
  $this->load_rest_of_wp();
  ...
}

function load_rest_of_wp() {
  if ( function_exists( 'some_function_thats_not_part_of_shortinit' ) )
    return;

  // proceed with loading
  ...
}

[scribu]

Or, to be absolutely certain, we can define a constant:

function load_rest_of_wp() {
  if ( defined( 'WP_TESTS_LOADED_WP' ) )
    return;

  define( 'WP_TESTS_LOADED_WP', true );

  // proceed with loading
  ...
}

[kayue]

My concern is once WordPress core is loaded in one of the test, all the global variables and template functions will be carried to all the tests after it.

In another word, the testing environment will be changed from time to time, depending on the previous test case, which is undesirable.

I agree that it is better to load WordPress in individual test case, instead of bootstrap, but until we find a way to "unload" WordPress, I prefer force it to load the core in bootstrap.