Help generate a report card for modules

[KirkMunro]

I wasn't quite sure where to post this suggestion, but since I feel the work starts with PSScriptAnalyzer I decided to post it here.

The quality of many modules published into the PowerShell Gallery is quite poor, including modules coming from teams inside of Microsoft (not to point fingers, but cough Azure *cough). There are plenty of community modules that are of poor quality also, and scripts published into the Gallery will no doubt follow suit.

PSScriptAnalyzer is being used on publish to notify authors about issues, but authors can choose to simply ignore the issues and every user who discovers a module in the Gallery has to review it themselves. The review step is necessary, and I don't think that should go away, but I think that review should be focused on security and technical issues, not module design issues that can be corrected automatically. We were chatting on Twitter about having moderators in the mix, but really moderation should be about serious technical problems that can have adverse side effects on infrastructure, not about pluralized nouns, lack of documentation, or many other issues that can be automatically detected. PSScriptAnalyzer should be used as a tool to not only identify issues for script/module authors, but to identify issues to end users looking at community content. Helping authors is necessary, but helping end users is even more necessary so that they can identify the cruft more easily. I'm thinking a report card of sorts, for script files and for modules, that ranks them according to how well they pass (or fail) tests. Authors should be able to pragma-out rules that are actually inaccurate for their work, but such a tool would be highly useful for the community, and I think it would give authors much more incentive to do it right the first time.

Case in point: AzureAD was just added to the Gallery very recently, and it is just full of design problems that can be identified by anyone with a bit of PowerShell experience in a matter of a few minutes. The issues are egregious enough that several MVPs feel they should never have released the module in this state, and since it's release and the issues have been raised via numerous channels trying to get their attention, it appears that the authors are summarily ignoring the feedback and the requests to pull the module until its design issues are fixed, before there are many dependencies on the bad design issues. This is one prime example of a module from a very influential and highly visible group that influences the PowerShell Community in a shockingly bad direction.

Having a report card sent back to the author when the module is published and shared on the PowerShell Gallery would go much further towards keeping problem modules like this in line, I think. You could even have the PowerShellGallery refuse to publish modules that fall below a certain grade, so that people wanting to share their content are forced to learn how to do it the right way. Then, after all of that is in place, moderators and/or the community could also review a module for other issues that are less obvious, provide feedback into PSScriptAnalyzer for things that could/should be auto-detected that are being missed, contacting module authors about general technical issues, and providing feedback to the Gallery administrators for other more serious problems that may warrant a module being unlisted for one reason or another.

I would love PSScriptAnalyzer to get to this point (maybe with the help of the OneGet team because this directly impacts them) so that I wouldn't feel like getting module authors to follow best practices is such a hopeless task. That seems to be the only way to get certain people to raise the quality of PowerShell product that they produce, inside Microsoft and out.

[rsdovers]

I like the idea of a report card of some type that is independent of the rating based on user feedback. When I look for a script on the Gallery, I review the script to make sure it meets my standards, and then I read the comments to see what issues others have had, and then I usually have to fix something because it doesn't work as described, or doesn’t create objects of the output. If there was a report card that outlined some of the best practices the script doesn't follow, or identifies if the script contains a potential performance issues, then I would skip the scripts that people just through together and upload. It would almost be like an automated moderation. How to automate the report card and how stringent the ratings should be will always be up for debate.

[juneb]

I agree. The automatic report card is a great start, but I would appreciate the opportunity to provide individual reviews and a 5-star rating system of modules. For some people, missing or broken help files, or cmdlets that do something, but not what they're advertised to do, are trivial. For others, they're critical. I'd like to read reviews or scan a rating before I install a module.

[BlackV]

Agreed, modules that are below a threshold (i.e. 2 starts out of 5 or something) shouldn't show up in search , or should only show up when you tick a box to (or a -lowestrating switch on find-module) include low rated scripts

Good feedback with respect to wanting a rating system on the Gallery, it is something we have considered. We are also thinking about linking the results of the automated ScriptAnalyzer scan with the modules on the Gallery.
As an FYI, feature requests for the Gallery are generally better placed into UserVoice (see http://windowsserver.uservoice.com/forums/301869-powershell), but we'll likely catch them here, too.

[joeyaiello]

I love the idea of a report card for modules, but I would ask for your help in forming some initial thoughts on what rules would be included for this analysis and how those rules might be weighted to emit some kind of "score".

It might be a good exercise here to discuss some of the more egregious design mistakes that you've felt that the AzureAD module has made, and how we might be able to catch those mistakes.

Also, to clarify: are you saying, @KirkMunro that stuff like pluralized nouns and lack of docs should or should not be part of the "report card score"?

[KirkMunro]

The biggest one (and the easiest to detect, so really it should never make it into the wild) is pluralized nouns. The AzureAD module (which was pulled from the Gallery) has a lot, and not just pluralized nouns, but variants. For example: Get-AzureADApplication and Get-AzureADApplications, both cmdlets, each with different syntaxes/parameters: the singular noun takes an -AppObjectId string parameter to get a single application, and the pluralized one takes a -Top int parameter to get the top N applications (or all if that parameter is not used). That is one example of many in this module where there are two flavors of what should be a single cmdlet with multiple parameter sets. And for this specific example, I can easily imagine (and have seen in the past) them simply changing the pluralized noun to something non-pluralized but equivalent (e.g. renaming Get-AzureADApplications as Get-AzureADApplicationList and thinking that was the right change to make).

I decided since the module was pulled that it might be worthwhile sharing it to use as an example, so I've attached an archive containing it here:
AzureAD.zip

There are probably a set of rules that could be categorized as "command signature", that check for pluralized nouns, approved verbs, suffixes that try to fake it (like List), etc. Those are the issues that I usually identify in the first 10 minutes reviewing a module, and almost always I identify them without running a single command, sometimes without even loading the module. They are definitely issues that I would want checked on the report card because they are the first experience a user has with a module, and they are costly to change after the fact.

SD01-SD05 would be a good place to start looking for rules that identify the command signature. If any of those don't apply or have changed or are not properly documented, then that document should be updated (aside: that document should clearly define singular noun as a RULE, not as a "Strongly Encouraged Development Guideline", because the latter leaves too much opportunity for interpretation).

And yes, I am saying that pluralized noun use as well as the lack of docs should be part of the "report card score" (and maybe that will light the fire needed for me to put docs in place where my modules are lacking them).

[Jaykul]

For what it's worth, the idea of assigning an automated grade for modules based on the output of the rules we've got now just makes me laugh. 😭