Assert-VerifiableMocks silently does not report anything if 'Verifiable' is spelled wrong on one mock

[johlju]

This code block reports the correct error.

Describe 'Tests' {
    Mock Get-Service -MockWith { return 'wuausrv' } -Verifiable
    Mock Get-Process -MockWith { return 'setup' } -Verifiable

    Context 'When something happens' {
        It 'Should return correct values' {
            Get-Service | Should Be 'wuausrv'
            #Get-Process | Should Be 'setup'
        }
    }

    Assert-VerifiableMocks
}

Assert-VerifiableMocks throw the correct expected error

Describing Tests

  Context When something happens
    [+] Should return correct values 124ms
  [-] Error occurred in Describe block 27ms

     Expected Get-Process to be called with $True
    At C:\Program Files\WindowsPowerShell\Modules\Pester\4.0.3\Functions\Mock.ps1:428 char:9

But if we make a typo in the parameter -Verifiable for the mock Get-Process

Describe 'Tests' {
    Mock Get-Service -MockWith { return 'wuausrv' } -Verifiable
    Mock Get-Process -MockWith { return 'setup' } -Verfiable

    Context 'When something happens' {
        It 'Should return correct values' {
            Get-Service | Should Be 'wuausrv'
            #Get-Process | Should Be 'setup'
        }
    }

    Assert-VerifiableMocks
}

Assert-VerifiableMocks reports success

Describing Tests

  Context When something happens
    [+] Should return correct values 76ms

I don't know if there is something you can do to fix this. But wanted to report it just in case. 😄

[alx9r]

@johlju Thanks for reporting this. I have reproduced it. Misspelling any of the parameter names is an easy mistake to make. I think it would be better if Mock threw an exception in that case.

Proximate Cause

For functions that do not use "CmdletBinding" PowerShell seems to stuff any arguments that are not mentioned in params() into $args without error.

Context, for example, doesn't use [CmdletBinding()], why isn't it affected?

Merely mentioning [Parameter()] seems change a function to use "CmdletBinding" which seems to cause bogusly-name arguments to throw an exception.

Which public functions are affected?

I think

Get-Command -Module Pester | ? { -not $_.CmdletBinding } | % Name

Is adequate to discover the affected public functions. Invoking those functions with bogusly-named parameters does not throw an exception:

Describe 'vulnerable to misspelled parameters' {
    AfterAll {} -bogus
    AfterEach {} -bogus
    Assert-VerifiableMocks -bogus
    BeforeAll {} -bogus
    BeforeEach {} -bogus
    Get-TestDriveItem -bogus
    In . {} -bogus
    function f {}
    Mock f -bogus
    SafeGetCommand -bogus
    It 'test' {
        Set-TestInconclusive message -bogus
    }
}

Possible Fix

It seems like using [CmdletBinding()] or mentioning [Parameter()] on each of the affected functions would fix this problem.

[alx9r]

It looks like invocation of the scriptblocks following AfterAll, AfterEach, BeforeAll, and BeforeEach is defeated also without error by bogus named parameters. Here is the repro:

Describe 'setup/teardown scriptblocks defeated without error' {
    BeforeAll { New-Item TestDrive:\success }
    BeforeAll { New-Item TestDrive:\defeated } -bogus

    It 'normal BeforeAll invokation successfully invokes setup scriptblock' {
        Test-Path TestDrive:\success | Should be $true
    }
    It 'bogus parameter defeats invokation of setup scriptblock' {
        Test-Path TestDrive:\defeated | Should be $false
    }
}

[alx9r]

In seems to either function correctly or throw an exception even when a bogus named parameter is provided. Here are some tests:

Describe 'in vulnerability' {
    It 'confirm location' {
        Get-Location | Should not be 'TestDrive:\'
    }
    In TestDrive:\ {
        It 'correctly sets location' {
            Get-Location | Should be TestDrive:\
        }
    }
    It 'confirm location' {
        Get-Location | Should not be 'TestDrive:\'
    }
    In TestDrive:\ {
        It 'setting location is not defeated by bogus parameter' {
            Get-Location | Should be TestDrive:\
        }
    } -bogus
}

[alx9r]

@dlwyatt I see you've self-assigned this. I've got a bunch of tests and the beginnings of a PR to fix this. Do you want to do it instead? Or do you want me to submit a PR?

[dlwyatt]

No worries, I'll look at your PR when it's ready.

[alx9r]

Assert-VerifiableMocks takes no parameters and does not seem to be vulnerable to bogus named parameters:

Describe 'Tests' {
    function f {}
    Mock f {} -Verifiable
    It 'invoke mock' { f }
    Assert-VerifiableMocks -bogus
}

[alx9r]

Set-TestInconclusive is vulnerable to misspelled parameters:

Describe 'Set-TestInconclusive vulnerability' {
    It 'this should be inconclusive and have a message' {
        $true
        Set-TestInconclusive -Message 'inconclusive message'
    }
    It 'so should this' {
        $true
        Set-TestInconclusive -Mesage 'inconclusive message'
    }
}

The second It{} does not show the message because -Mesage is misspelled.

[dlwyatt]

In terms of tests needed, I think your original command sums it up just fine: Get-Command -Module Pester | ? { -not $_.CmdletBinding } | % Name

Making sure that we don't export any commands that don't have CmdletBinding is sufficient to avoid this type of error.

Don't worry about the Before/After commands; I need to take a fresh look at those anyway.

[alx9r]

Changing the affected functions to use CmdletBinding will cause user tests that inadvertently misspelled certain parameters to begin to fail (per OP's example). As a user I think I'd be grateful for those test to start failing when, say, switching from Pester 3.x to 4.x. But it might be a bit painful if that change came through unexpectedly as a minor patch via Windows Update or otherwise.

I'm thinking this change should be included in 4.x ASAP and not backported to 3.x. Does that sound right to you @dlwyatt?

[alx9r]

The vulnerability and deprecation status of commands I think are affected by bogus named parameters seems to be as follows:

• AfterAll - vulnerable
• AfterEach - vulnerable
• Assert-VerifiableMocks - not vulnerable
• BeforeAll - vulnerable
• BeforeEach - vulnerable
• Get-TestDriveItem - deprecated in 4.0
• In - not vulnerable
• Mock - vulnerable
• SafeGetCommand - Pester internal
• Set-TestInconclusive - vulnerable
• Setup - deprecated