Humbug's Future (Discussion)

[padraic]

Apologies all, it has been a hectic year or so, and I've allowed ALL of my online activities to come to a stop. I'm currently creeping back onto the open source scene, so I'm getting close to sorting out the Humbug situation. Here's basically where I see things going in brief, and I'm intent on implementing it in the coming weeks:

• I've created a "Humbugged" organisation: https://github.com/humbugged (Yes, it's odd. Yes, the humbug user name was taken. Humbug!)

• I hope to transfer ALL Humbug related repositories, included key supporting dependencies, to the new organisation.

• Organisations need members! If you have emailed, or tweeted, me previously about having a greater involvement in Humbug, I'll reach out to you by email to see if you are still interested (and apologise for the lengthy response wait!). Those who have written PRs, short of having auditions, will be preferred. Ideally, a small team of 2-3 to start with, and I will open it to the floor here if needed to make up numbers.

• While remaining a BDFL figure, I'm generally very relaxed about where my projects go. Members WILL have scope to act independently, champion new features, manage issues, peer review and merge PRs, etc. I'll remain involved, but mostly to ensure things run smoothly as a glorified moderator. Governance can be elaborated on in time. While I expect to be back actively contributing, that's a goal for the medium term. In the meantime, there are issues, PRs, and lots of scope to improve Humbug.

• Humbug logo? I think the NWN avatar I use falls a bit short ;).

The most urgent priorities would, in some basic order:

1. Getting the existing code running error free under Travis (so we can recheck PRs).
2. Merging PRs wherever it makes sense.
3. Issue triage, and solving those where not covered by PRs (within reason).
4. Put out a new release.
5. Have a good think about where Humbug is now, and where it should be for a stable release.

The project will not, at this time, be transferred to a new owner. However, ideally that will become redundant as the project becomes sustainable whether I'm around or not. If I ever do transfer ownership, I would strongly prefer it to be a matter of transitioning administration of the organisation to existing member(s) at that time.

By all means, please discuss below.

[theofidry]

Nice to see you back, although you don't need to apologize.

I've created a "Humbugged" organisation: https://github.com/humbugged (Yes, it's odd. Yes, the humbug user name was taken. Humbug!)

https://github.com/humbug doesn't seem very active, it may be worth to try to reach GitHub to claim that name. That said it's a process that may take a bit of time so having another organisation in the meantime is a good thing IMO.

I hope to transfer ALL Humbug related repositories, included key supporting dependencies, to the new organisation.

👍 GitHub now can transfer repositories whilst keeping all the issues IIRC, so it should go smoothly.

Humbug logo? I think the NWN avatar I use falls a bit short ;).

I actually quite like it, maybe we could use a small variant.

Regarding the current status, from what I see that looks necessary on the short term:

• Ensure it works fine up to PHP 7.1 and prepare for PHP 7.2
• Add support for PHPUnit 6 (PHPUnit 6 Support #204)
• Add support for phpSpec (Support for PHPSpec #192 — [WIP] Support for phpspec (reset) #145 & [Archived] Support for phpspec #88)
• Allow to use phpdbg over xdebug (Don't require xdebug for code coverage #178 — Allow to be run with phpdbg #184)

As the project currently supports PHP 5.4+ and is already quite used (77 depends), I would recommend to fix the issues above which can be done without too much changes and release
a first stable release.

In the future, I would like to push for:

• Support for PHP7.1+ only (I don't see this release before the end of the yeah and at that time PHP7.0 will have reached its EOL)
• AST based mutations: there is already a POC from @MarkRedeman cf. https://github.com/MarkRedeman/ast-based-mutations. And PHP7 has a php-ast extension which should help there.
• Be test runner agnostic: we should be able to make Humbug work with PhpUnit, phpspec, behat etc., be abstractable so that integrating a new testing framework is a mater of building a bridge
• Performance: with some testing framework (phpspec: a spec file is linked to a tested file, in PhpUnit it's doable to check it with the @Covers tags), it should be possible to have a "light" testing mode where only a subset of the tests are run to speed up significantly the tests
• False positive: much more tricky, but there should be a way to note false positive to not always get them. Always having 50 false positives make it much harder to see the new ones (or the ones that have been removed), and make it significantly less attractive to any user
• I would also like to make it possible to have a report detailed enough to do a diff between two reports and be able to analyse the changes. This way it would make it possible to integrate Humbug to CI tools with an evolution tracker

On a side note, I also thing it would be a good idea to promote other mutation testing frameworks, i.e. to promote the practice itself (maybe have an awesome list?). A little least of other mutation testing frameworks which can also be interesting to check for ideas/features:

• http://pitest.org/
• http://mutation-testing.org/
• https://stryker-mutator.github.io/
• https://github.com/mbj/mutant

And an interesting document: http://crestweb.cs.ucl.ac.uk/resources/mutation_testing_repository

[MarkRedeman]

Hey @padraic, @theofidry

I would love to help out on this project. I still have a couple of other proof of concepts laying (including one that works with many test frameworks, but doesn't include all performance optimizations that Humbug has) around which I haven't open sourced yet.
I will try to explain some of my ideas and push some code samples next week, after I've finished my exams.

[davedevelopment]

@padraic I wouldn't want to commit to being a major contributor, but if you struggle to get people I can be someone who's around to quickly glance over issues as they come in etc. I do a similar thing for sculpin, I'm the person who says "looks good enough to me" and hits the merge button.

[lt]

@theofidry @padraic I've reclaimed dormant names for a couple of businesses, the process was very quick (same day), you have to be ready to jump on it as soon as they release it though.

[kelunik]

See #201 for a PR that fixes the tests on Travis.

[stof]

Add support for PHP6 (#204)

@theofidry I think you meant PHPUnit 6 😄

[padraic]

@stof And here I was thinking we'd finally found a use for those PHP6 books... :D

[theofidry]

edited, soz^^

[padraic]

I'm pleased to add that @theofidry will be the first full member.

I hope to send out two additional invitations over the next day or so. Once I have two more acceptances, I'll confirm them here. This will bring together the small 2-3 person team.

Thanks to everyone who volunteered whether it was recently or some time ago! For those I don't send invitations to, it has not been easy to narrow down the list I have, but please remember that you don't need to be a member to write PRs ;). A few of you expressed ideas on how to improve or evolve Humbug, and I'd love to see those reach the Issues/PR lists.

[theofidry]

The pleasure is shared, let's make 2017 the year of mutation testing :)

[mallardduck]

While I'm not too familiar with the projects code specifically, I'd be happy to help contribute as I can. I really love using this tool as I dev and would love to be involved however I can help!

[simondel]

Good to see that you're back! I'm not very experienced with PHP but perhaps you can introduce an 'easy' tag for people like me if you find a small issue :)

[maks-rafalko]

Hello, I'm not sure you saw my email @padraic, so I'm posting here to share my experience with humbug, thoughts, plans, and to ask questions:

Introduction (click me)

First time I saw mutation testing framework I was very impressed with this idea and found Humbug for PHP. Of course, I have immediately tried it for my projects. While it showed me bad tests, I had many problems with false-positives, and this one moved me forward to investigate Humbug's source code:

// code is very simplified just to show the case of false-positive
function test($a)
{
    if ($a === 0) { // line 3
         return false;
    }

    return $a < 0; // line 7
}

Humbug created a mutation for line 7 with <= and it was not killed, because when $a === 0 we already have a return value false on line 4. I wanted to fix it and started to investigate the source code. But the main problem to start contributing to Humbug is a difficult token-based logic. I mean using get_token_all() format.

So I decided to google how I can improve this situation and found PHP-Parser project, which works with Abstract Syntax Tree. (unfortunately, I didn't see @MarkRedeman's implementation)

Since I had no idea what is the future of the Humbug and seeing your inactivities, dead issues and PRs, I started my own new project - to have a challenge and to familiarize with PHP-Parser / PHPUnit internals.

...So currently I have working AST-based mutators similar to @MarkRedeman's ones, but with the same API as Humbug. Some examples:

Plus mutator - AST (click me)

class Plus implements Mutator
{
    public function mutate(Node $node)
    {
        return new Node\Expr\BinaryOp\Minus($node->left, $node->right, $node->getAttributes());
    }

    public function shouldMutate(Node $node) : bool
    {
        if (!($node instanceof Node\Expr\BinaryOp\Plus)) {
            return false;
        }

        if ($node->left instanceof Array_ && $node->right instanceof Array_) {
            return false;
        }

        return true;
    }
}

Example of ported from Humbug unit tests for FunctionCall mutator - AST (click me)

class FunctionCallTest extends AbstractMutator
{
    public function test_not_mutates_with_value_return_true()
    {
        $code = '<?php return true;';
        $nodes = $this->getNodes($code);

        $this->assertFalse($this->mutator->shouldMutate($nodes[0]));
    }

    public function test_not_mutates_with_value_return_true_for_new_objects()
    {
        $code = '<?php return new Foo();';
        $nodes = $this->getNodes($code);

        $this->assertFalse($this->mutator->shouldMutate($nodes[0]));
    }

    public function test_mutates_with_value_return_function_call_no_params()
    {
        $code = '<?php return array_filter();';
        $nodes = $this->getNodes($code);

        $this->assertTrue($this->mutator->shouldMutate($nodes[0]));
    }

    public function test_mutates_with_value_return_function_call_with_params()
    {
        $code = '<?php return array_filter([]);';
        $nodes = $this->getNodes($code);

        $this->assertTrue($this->mutator->shouldMutate($nodes[0]));
    }

    public function testGetsMutationSettingReturnValueNullAndPreservingFunctionCall()
    {
        $code = '<?php return array_filter([]);';
        $mutatedCode = $this->mutate($code);

        $expectedMutatedCode = <<<'CODE'
<?php

array_filter([]);
return null;
CODE;

        $this->assertSame($expectedMutatedCode, $mutatedCode);
    }

    protected function getMutator() : Mutator
    {
        return new FunctionCall();
    }
}

The difference is that it uses shouldMutate() method to understand whether we need to do mutations or not (e.g. we don't need to mutate [] + [] with Plus mutator).

I understand how much efforts you brought to the Humbug, hundreds of human-hours, and I understand that my pet-project may not see the open source world, but:

1. do you think I can help with something to Humbug, since I already have an AST based mutators / mutant generator and know internals of the humbug (how coverage works, time collectors, phpunit listeners, etc.)?
2. do you think I should continue writing my own version of mutation framework? btw, It's very interesting to write it and I don't want to stop the development, even just for fun ;)

Suggestions about future plans:

1. IMO, to improve DX of this project and to move forward, the first major task is to rewrite humbug using AST and PHP-Parser. As I understand @MarkRedeman can help with it (and I am of course ;))
2. I suggest to not fix any false-positives with current implementation because it's useless and (hopefully) will be fixed easier with AST eventually
3. Start development of humbug's 2.0 project with PHP 7.1 requirements, with AST, and with new features (e.g. mutating code only for diff in CI server like Ruby's Mutant does for each push)

So, if you guys think humbug can benefit from what I already have, just let me know.

[padraic]

@borNfreee To emphasise my approach to open source - I'm basically a terrible dictator. If someone writes a PR, and others are open to reviewing/adopting it, it will more than likely be merged. However, for the short-term, we do have to prioritise bugs+maintenance over new features/rewrites, and then assess where we want to be for a stable (implying a backwards compatability retaining) 1.0 version. There will be discussions in the future on the topics you've raised, and I'd love to see you take part in those when we get there!

I've restricted the core team to three members to keep it simple, and ensure Humbug can continue without me being its single point of failure, but there won't be any restrictions on PRs, or discussing issues, and all of the other ways in which people can contribute.

Being a tinkerer for fun myself, you don't have to stop working on your own project even if helping us here, and you may find your own ideas developing into something of value whether its to Humbug or a future competitor ;).

[GrahamCampbell]

I've created a "Humbugged" organisation: https://github.com/humbugged (Yes, it's odd. Yes, the humbug user name was taken. Humbug!)

You can check with GitHub support if the organization is actually used. Their terms ban squatting on organization names, so they can release it for you.

[stof]

@GrahamCampbell looks like it is done now, as https://github.com/humbug is now an organization with @theofidry in it

[padraic]

@GrahamCampbell @stof Yep, Theo was able to get hold of it, so good news all around. We managed to secure Mockery yesterday also.

[theofidry]

Actually @padraic is also already in that organization, but he needs to change his visibility status to "public" :P

Unless there's something missing we can already move the repositories no? Might need a quick update on packagist, but that should be it

[padraic]

Now we are three with @pamil ;). Just waiting on one more response here, or refusal - you never know!

[GrahamCampbell]

👍

[padraic]

Thanks to @MarkRedeman who brings us up to four!

Thanks to everyone who sent emails! Unfortunately, I could not bring everyone of you on board with write access, but we're going to be far more responsive from here on out to your PRs, issues, and suggestions.

[snapshotpl]

Good luck! Waiting for first stable release!

[padraic]

@humbug/core AKA: @theofidry @pamil @MarkRedeman

I've transferred humbug itself, and few dependencies across to the organisation. You should all be added to the core team, with the team granted write access. Anyone here can address the entire team using @humbug/core in issues/PRs.

The repositories include:

• https://github.com/humbug/phar-updater (as a significant sub-project)

• https://github.com/humbug/phpunit-extensions

• https://github.com/humbug/phpspec-extensions (prototyped helpers)

[theofidry]

👌 quite like the branding. Not really what I had in mind but I couldn't find something satisfying after looking a bit.

I'll try to do a turn on the ongoing issues/PRs to cleanup a bit ASAP.

[padraic]

@humbug/core I did some basic labelling yesterday, just to see what's out there. Everyone should, I think, have access to the Github review features too. Make sure you assign any issues/PRs so nobody is duplicating work ;).

@theofidry We can replace it any time, but didn't want to leave the org logo empty ;).

[padraic]

I'll open a few new issues in the next day or so for discussing organising/releasing, where to talk, and the usual "please shoot me already before I die of boredom" stuff 😁.

[padraic]

I think we've wrapped this up for now. Repos are added to the new organisation. There's a logo. There's a roadmap issue open. Ready. Set. G...<vanishes in puff of smoke> Kidding!

[Pablodotnet]

What about making a channel on Slack or Gitter to create a community for Humbug future assistance?

[padraic]

@Pablodotnet We have a #humbug channel now running on the symfony-dev's slack.

[theofidry]

See #238