Add unit tests

[artfulrobot]

Short description of the enhancement

Tests are really important in software development. They help detect (and thereby can help prevent) regressions, and also can be used to document quirks. ProcessWire is big and complex and deserves this good discipline.

Current vs. suggested behavior

Currently, as far as I (and at least one other person) can see, there aren't any tests. As a new person to PW, this makes me very nervous as I've been around software too long to believe anyone can write it without bugs.

For example, the selector strings - there's some very complex logic in there - that really deserves some tests.

Why would the enhancement be useful to users?

1. They could be more certain of the quality of the product over time; as long as tests are run and pass pre-release, we know that all code with tests is behaving as it did before and as it is designed to.
2. They can see exactly how things are supposed to work - the docs are great, but they can also be wrong.
3. They can document bugs and quirks by submitting PRs that add tests.
4. They can fix bugs, show that they're fixed and show that they've not broken anything else in the process!

Optional: Screenshots/Links that demonstrate the enhancement

For example, here's a crude phpunit test that documents the inconsistencies in support for case-sensitivity and UTF8 in a couple of the Selectors:

<?php
declare(strict_types=1); // PHP requires this first, sorry style guide!
namespace ProcessWire;

use PHPUnit\Framework\TestCase;

class SelectorTest extends TestCase {

  public function testSelectorEquals(): void {
    $w = new WireData();
    $w->set('x', 'axe');

    $s = new SelectorEqual('x', 'axe');
    $this->assertTrue($s->matches($w), "Expected a match but it didn't happen.");

    $s = new SelectorEqual('x', 'Axe');
    $this->assertFalse($s->matches($w), "Expected no match because SelectorEquals is a case sensitive comparison");

  }

  public function testSelectorStarts(): void {
    $w = new WireData();
    $w->set('x', 'axe');

    $s = new SelectorStarts('x', 'axe');
    $this->assertTrue($s->matches($w), "Expected a match but it didn't happen.");

    // Quirk: some selectors are case in-sensitive, others are case sensitive.
    $s = new SelectorStarts('x', 'Axe');
    $this->assertTrue($s->matches($w), "Expected case in-sensitive comparison");

    // Quirk: although SelectorStarts is case insensitive, it does not support multibyte UTF-8
    $w->set('x', 'axé');
    $s = new SelectorStarts('x', 'axÉ');
    $this->assertFalse($s->matches($w), "Expected case in-sensitive comparison to fail - ProcessWire does not support UTF-8 in this Selector");
  }

}

Running these tests gives this:

PHPUnit is an amazing tool for all sorts of tests, from pure unit tests through to integration and end-to-end tests. It has lots of nice features for stubs, mocks, asserting expected exceptions etc.

Tests are costly to write, granted, but they can be added to over time, and added along with new features. Even some tests would be better than none.

[BernhardBaumrock]

I'd love to see tests for PW! There are definitely more people interested in this: See https://processwire.com/talk/topic/19396-using-nette-tester-with-processwire/ or https://github.com/uiii/tense

[teppokoivula]

For the record, Ryan has stated on a few occasions that he's using an updated version of https://github.com/niklaka/ProcessWireTests. I'm not sure if that's still true, and his version of the test suite has never been publicly available (to my best knowledge).

[artfulrobot]

Thanks @BernhardBaumrock yes, they're like a sanity check for me, and a way to think about improvements.

Tense looks great; one of the hardest bits in non-trivial/unit testing is setting up the fixture dbs etc. so it's fab to see that someone's done that grunt work.

I've not come across Tester by Nette before. Seems interest is waning - article is few years old but if if you click the graphs they're bang up to date and show the same trend: Nette ↓ and 1/100th of popularity of phpunit which is rising continually with an upward curve. I note Nette make Tracy debugger, which is super though.

[BernhardBaumrock]

I've almost no experience with testing, but I really like what the folks at Nette are doing. TracyDebugger is great and their forms component is the best I could find. Also Latte is in my opinion absolutely brilliant and far superior than the other template engines. Of course that's a matter of taste.

If download numbers were the only thing that counts we'd all be using laravel:

I've seen another open source project where you had to provide a test-case with every issue report that you made. That could be a good way to force people to do their homework before posting issues and also it could help to improve PW in the long run, as a test that we added once could be run on every new release and find issues before they make problems.

[artfulrobot]

@BernhardBaumrock I'm sure Tester works fine; I was just going on the idea that we have a choice:

• the popular one is phpunit, which doesn't make it a definite, but a counter to your big graph is that just because PW is relatively unpopular doesn't mean we have to use a relatively unpopular testing library 😀
• the existing tests (for what they may be worth) are in phpunit;
• I brought it up because I've used it fairly extensively (I'm a big fan of testing - php at least!) and know it's good.

But anyway, I'd be happy with any framework - we just need tests!

I'm not sure why Ryan would be keeping tests private; maybe he wasn't sure it was in a position for others to help build on yet.

It's great that this idea has some supporters though

[er314]

I fully agree,
nowadays, for many people and for good reason, assessing the quality of an open source software comes with assessing the quality of its unit-tests.
Therefore, not having any unit-tests shipped with PW surely has a negative impact (repelling effect) on some potential new users (developers & customers).
As the most important part to be unit-tested is the selector engine, I guess that the first step would be to adapt & embed the currently private unit-tests of the selector engine into the main PW tree.
Then, having the testing framework in place, this may also encourage contributions by the community.

[hiboudev]

I definitely want tests too. I find too much minor bugs in PW, and even regression like this one.
I start to be scary about that. For example that regression will broke an entire site I built on forms if I update PW.

I'm currently building a somewhat big project using lot of PW features, and am saying to my client: "never update PW without testing the whole site, tools and Web API on staging server". That's not good at all.

A public github repository for unit tests would be awesome, so @ryancramerdesign would not be alone to write tests.

On my side I use PhpUnit.
To run tests against code that affects DB, I think we could use a $db->beginTransaction() and $db->rollback() in setUp() and tearDown() functions. I want to make a try when I'll have some spare time, I never did this before.

[daun]

Adding another vote for a test suite. Any test is better than no tests for a project as complex as ProcessWire. Writing tests for a few hot paths around selectors, data integrity and security would be a great start and an important signal to new adopters.

So many of the recent updates have led to subtle bugs that it has really eroded my trust in the software. I now dread upgrading the core on my customers' sites, which makes me a bit sad: backward-compatibility and long-term stability was one of the main selling points when getting into PW a few years ago.