FR: detect tests via annotation

[tyler36]

Thank you for the extension.

I've been looking for a replacement for "recca0120.vscode-phpunit" for a while and came across this one.

I'm havign trouble getting this extension to recognise my tests.
Basically,

• all my tests files are sub-folders in ./tests
• all my test files end in Test.php
• all the test are marked with annotation

<?php

namespace Tests\Unit;

use PHPUnit\Framework\TestCase;

class ExampleTest extends TestCase
{
    /** @test */
    public function this_should_work()
    {
        $this->assertTrue(true);
    }

    /**
     * @test
     * @group error
     */
    public function this_should_not_work()
    {
        $this->assertTrue(false);
    }
}

When I install this extension, I can't see the test icon unless I open a test file. At that point, it populates with every test in my workspace, including vendor.

I think my tests are not displaying because this extension assumes a "functionRegex", and can't detect @test annotations, but I'm not really sure. Regex is a crazy powerful, but I still struggle writing & parsing it.

My current config is the following.

    "phpunit.folderPattern": "./{tests}/**/*Test.php",
    "phpunit.fileRegex": ".*(Test).php",
    "phpunit.functionRegex": "/\\s*(public\\s+){0,1}function\\s+(\\w*test\\w*)\\s*\\(",

What am I doing wrong?

PHPunit Extended with TestExplorter: 0.2.5
Test Explorer: 2.21.1
Test Adapter: 0.1.4
Vscode: 1.62.3
OS: Remote: WSL 0.58.5, Ubuntu via Win10 WSL2

[RobertOstermann]

That looks to be a problem with your functionRegex. The function regex currently only searches a single line, so there is no regex that will recognize a function is a test function based on the annotation. If there is no consistent naming for test functions I would recommend using the following functionRegex.

    "phpunit.functionRegex": "\\s*(public\\s+){0,1}function",

This will recognize all functions that start with any number of spaces followed by public function or function as test functions.
Depending on your test files, you may get some false positives if there are non-test functions that are not private within those test files.

Also, if you do not want the tests within your vendor folder showing up, you would have to change the glob pattern to exclude that folder. The folderPattern glob determines the folders to look at when discoverAllTests=true, then the files within those discovered folders are compared to the fileRegex to determine if they should be included in the test explorer.

I typically use the site regex101.com to test out different regular expressions.

If you are still not seeing those tests let me know and I will look further into this issue.

[RobertOstermann]

In version 0.2.7 I have added the configuration option multilineFunctionRegex, which could be useful for you to find tests if you want to limit to only tests that have the @test annotation. The boolean value enables the functionRegex to evaluate all the lines since the last test that was found with the regex flags gis. Here are some settings that should work with the example tests you have shown me.

"phpunit.functionRegex": "\\/\\*.*?@test.*?\\*\\/\\s*?(public\\s+){0,1}function",
"phpunit.multilineFunctionRegex": true,

[tyler36]

Thank you for the update.

I tried 0.2.7 with the following settings.

    "phpunit.discoverAllTests": true,
    "phpunit.folderPattern": "./tests/**/*Test.php",
    "phpunit.fileRegex": ".*(Test).php",
    "phpunit.functionRegex": "\\/\\*.*?@test.*?\\*\\/\\s*?(public\\s+){0,1}function",
    "phpunit.multilineFunctionRegex": true,

The "testing" icon does not appear until I open a test file. Then, icon appears and I see a list of the file test.
If I open another test file, it's contents then appear.

This feels like the "phpunit.discoverAllTests" isn't working for my setup.

[RobertOstermann]

I currently have the extension only initializing once a php file is opened, I will make an update to change that so the tests are recognized when VSCode is opened.

However, as it currently works you should see all your tests once any php file is opened, so if you are not seeing that functionality it probably means you need to change the phpunit.folderPattern setting. It utilizes a glob pattern to recognize the folder/files to add to the test explorer. I am not as familiar with glob patterns, and there might be some changes you need to make on linux (not really sure there), so if you want something that exactly fits your specific cases you would have to look that up yourself. Could you try this setting and let me know if that works?

"phpunit.folderPattern": "**/tests/**/*Test.php"

or possibly

"phpunit.folderPattern": "**/tests/**/*.php"

The folderPatter setting is used in combination with the fileRegex when finding tests, which is why you probably would not need to be specific about the ending and could use *.php instead of *Test.php.

[RobertOstermann]

On second thought, if the files are showing up in your view explorer, but just the individual tests are not showing up then that might be a problem with parsing unopened files using the functionRegex for multiple lines. If that is the case I will have to dig into the code and get back to you.

[tyler36]

Thanks for the follow-up.

Tests show up after I open a test file, or if I have previously opened a file. It sounds to me like it might be infunctionRegex.

[RobertOstermann]

I have released version 0.2.8 which should allow this extension to startup when the workspace contains any php files.

I took a quick look at the code and do not see a reason that the functionRegex would be causing this problem, though I thought that was a possibility earlier. Have you tried the folderPattern configurations that I gave examples of? If you have not, I still think that may solve your problem. Though either of those globs would still include the tests from the vendor folders. I am not sure how to exclude that folder using a glob, but you could alter your fileRegex to exclude tests that include the word vendor, which might be what you are looking for. Try the following configuration and let me know if it works for you.

  "phpunit.discoverAllTests": true,
  "phpunit.folderPattern": "**/tests/**/*.php",
  "phpunit.fileRegex": "^(?!.*vendor).*Test.php$", // if you want to exclude files with "vendor" in the path
  // "phpunit.fileRegex": ".*(Test).php" // if you want to include all test files
  "phpunit.functionRegex": "\\/\\*.*?@test.*?\\*\\/\\s*?(public\\s+){0,1}function",
  "phpunit.multilineFunctionRegex": true,

[tyler36]

Updated to version 0.2.8

I'm using the following settings and restarted VScode

    "phpunit.discoverAllTests": true,
    "phpunit.folderPattern": "./tests/*Test.php",
    "phpunit.fileRegex": ".*(Test).php",
    "phpunit.functionRegex": "\\/\\*.*?@test.*?\\*\\/\\s*?(public\\s+){0,1}function",
    "phpunit.multilineFunctionRegex": true,

The "testing" icon appears but says:

"No tests have been fouind in this workspace yet."

If I open a test file, or switch focus to an already opened test file, the tests appear in the panel.

I tested with the following functionRegex as you suggested, restarted between each but had the same results.

    "phpunit.functionRegex": "\\/\\*.*?@test.*?\\*\\/\\s*?(public\\s+){0,1}function",

    "phpunit.functionRegex": "\\s*(public\\s+){0,1}function",

[RobertOstermann]

Have you tried changing the folderPattern? In my previous comment I listed out some settings I thought might work, but it looks like your current settings has not changed the folderPattern, or am I missing something?

[tyler36]

Just tried the following, reloading the extension after change:

"phpunit.folderPattern": "**/*Test.php",

"phpunit.folderPattern": "**/tests/**/*.php",

"phpunit.folderPattern": "**/*.php",

"phpunit.folderPattern": "./**/*.php",

[RobertOstermann]

Hmm, that is unfortunate. When you say you reloaded the extension do you mean reloaded the VSCode window, or exited out of VSCode? Simply disabling and re-enabling the extension might not be enough. I should probably add a reload window prompt whenever the folderPattern is changed.

But assuming you did reload the window then I am out of ideas. Given that you are able to see, and I assume run, the tests once you open a test file that means the fileRegex, functionRegex, and multilineFunctionRegex values are all good and working as expected. I think the folder pattern being

"phpunit.folderPattern": "./tests/*Test.php"

would not work, but using

"phpunit.folderPattern": "**/*.php"

should effectively watch all the php files in the workspace. Also, knowing that those other values are working that should mean the test files would get added to the test explorer. I have not been able to recreate this problem on my end so I am stumped as to the problem here. You mentioned in your first comment that once you opened a test file then all the tests are populated, do you have any idea why this stopped happening? I do not think any of the updates I made would have changed that.

[tyler36]

OK ... I think my problem was with VSCode settings/cache somewhere.

• Install portable copy of VSCode with only this extension.
• Use the following complete settings.json file

{
    "phpunit.discoverAllTests": true,
    "phpunit.folderPattern": "**/tests/**/*.php",
    "phpunit.fileRegex": "^(?!.*vendor).*Test.php$", // if you want to exclude files with "vendor" in the path
    // "phpunit.fileRegex": ".*(Test).php" // if you want to include all test files
    "phpunit.functionRegex": "\\/\\*.*?@test.*?\\*\\/\\s*?(public\\s+){0,1}function",
    "phpunit.multilineFunctionRegex": true
}

And everything worked; saw icon withonly my tests (no vendor tests).

Next, replaced my "normal" Vscode install with the above settings and same old "problems".
I disabled all extensions, re-enable only this extension and same old "problems".

I completely uninstalled VScode, and renamed my %appdata%/code to force a regeneration. I slowly activated each extension and copied over "chunks" of my old settings files, everything appears to be working now.

Thank you for taking the time to help debug the problem. I really appreciate it.

[tyler36]

More feedback. While I had it working on Win10, when I tried to access a WSL projects, it failed.

I tried deleting the .vscode-server folder in WSL but the problem persisted. Then noticed that some old extentions were syncing into WSL ( but they didnt appear on Win10); "hbenl.vscode-test-explorer", "connorshea.vscode-test-explorer-status-bar".

I had "Setting Sync" & VSCode sync turned on so perhaps there was conflict between these (which might have caused the original problem too).

[RobertOstermann]

Interesting...thanks for the feedback. And glad you were able to figure it out and get everything working! Once you have some time to try out the extension I'd appreciate if you could leave a review on the marketplace. And if you have any other problems or think something could be improved you can always create another issue.