Code coverage analysis#148
Merged
nohwnd merged 12 commits intopester:Betafrom Jul 4, 2014
Merged
Conversation
Uses the AST to identify all commands in whatever script files the caller specifies, and sets empty PSBreakpoints on each of those commands. After the tests finish, the breakpoints' HitCount properties are checked to determine coverage, and a report of any commands that have 0 HitCount is displayed. Seems to work so far, but the tricky part is using this to analyze Pester's own test suite. I've added several calls to Suspend-CoverageAnalysis and Resume-CoverageAnalysis (which basically correspond to Disable-PSBreakpoint and Enable-PSBreakpoint) to try to keep the number of false positives to a minimum, but I don't think we're ever going to be 100% accurate for our own internal tests.
Performance hit was pretty deadly when I ran Pester's own tests using this feature. Repeatedly disabling and enabling nearly 900 breakpoints brings the speed to its knees, and this performance hit would have been passed on to the users who weren't even affected by the problems of having Pester test itself. However, I did get a single report run against Pester before I changed the code back, and it reported around 50% code coverage. Definitely some room for improvement there. I'll try to come up with a way to do a useful coverage analysis without a bunch of false positives against Pester in the future (maybe having two copies of the module with different prefixes, or something. One to run the tests, the other to be analyzed for coverage.)
When displaying missed commands, instead of only outputting the "command" element which comes after the return or throw keyword, the code now displays the entire context of the return / throw statement, which is more useful output. May add more of this sort of context to the coverage output later.
Experimenting with Git; it was reporting changes to Describe.ps1 and It.ps1 which I had rolled back manually, but left some differences in whitespace. Using Git Checkout -p option to try to check out the actual copies from the Beta branch.
Invoke-Pester's -Coverage parameter (renamed from -CoveragePath) can now either accept strings (paths), or hashtables. If you use a hashtable, it must contain a Path key (which can actually be named Path, p, FilePath, File, or f... maybe went overboard there), and may optionally contain StartLine (or start or s) and EndLine (or end or e) keys which allow you to narrow down the coverage analysis to a particular part of a file. Coverage paths may now contain wildcards as well. Nice-to-haves: Maybe allow Function names in these hashtables, instead of requiring line numbers. The AST makes it easy to find those anyway; it's just a matter of parsing the user's input in an organized manner. Polymorphism would be nice right about now. Do we have those v5 classes yet? :) This would allow for a more persistent way of analyzing coverage, without having to look up new line numbers every time you change a file.
The hashtable that is passed to Invoke-Pester's -Coverage parameter can now have a key named Function (or f) to generate a coverage report for all commmands in that function. The value assigned to this key may contain wildcards. Some of the aliases for Path have been removed; now only Path and p are acceptable for that key. The script doesn't currently generate an error if a hashtable contains both Function and StartLine and/or EndLine values, but if Function is present, StartLine / EndLine are ignored. I should probably add either an error or warning output in this condition.
Regardless of how users pass input, only one breakpoint will be generated per command in each file, keeping the report of commands analyzed accurate and the output cleaner (no duplicate lines for the same command in the report.) Renamed -Coverage parameter on Invoke-Pester to -CodeCoverage. I'm still not sure if that's the right name, but it'll do for now. Also renamed -InputObject parameter on Enter-CoverageReport to -CodeCoverage as well, and removed pipeline input from Enter-CoverageReport. (It wasn't being used anyway.) While making these changes, also extracted some code from a couple of longer functions into their own functions.
There were a few comments added with the Isolation branch which are not really needed, so I took them out to reduce clutter. Added documentation to Invoke-Pester's comment-based help around the new CodeCoverage parameter.
CodeCoverage feature now automatically limits itself to attempting to parse File objects with extensions of PS1 or PSM1. This makes it easier for users to pass in a value of '*' to the CodeCoverage parameter, without having to worry about the wildcard resolving to Directories and/or non-PowerShell files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added -CodeCoverage parameter to Invoke-Pester. This allows users to specify files (or parts of files) which should be analyzed to see if any commands were not executed by the tests. The parameter's usage is somewhat like the Property parameters of Select-Object and Format-*; it can be passed either simple strings, or hashtables for finer control. Strings are paths to source files (which may contain wildcards), and the hashtables can contain the following keys (for convenience, each key may be abbreviated to its first letter, as shown):
The coverage report looks something like the following sample (and will also show up as a property on the -Passthru object, if this switch is used):