Add support for extension methods (LINQ)

[vors]

Copied from TFS item 819298

Original Connect item: https://connect.microsoft.com/PowerShell/feedback/details/635454/add-support-for-linq

When LINQ is available, it can push queries into the native language of the underlying storage without requiring users to learn that query language. This transformation can speed up queries to perform hundreds times faster, and when the alternative is to ship large amounts of data over the wire to be filtered locally, thousands of times faster.

In many cases using LINQ can convert a script which ran so slow as to be infeasible into scripts that we can run every day, and in fact, can run whenever we like -- and it also drastically reduces CPU impact.

For a trivial example, see this post on Stack Overflow http://stackoverflow.com/questions/4559233/technique-for-selectively-formatting-data-in-a-powershell-pipeline-and-output-as where the switch to using LINQ (by embedded the LINQ into a C# function via Add-Type and calling it from PowerShell) provided 1000 times speedup. It's currently not possible to write that LINQ query at all in pure PowerShell, and the alternative (as it was written in the other answer on that StackOverflow question, using Where-Object) is extremely slow even on modest data sets.

The bottom line is that the Where-Object and Select-Object cmdlets are very slow, and using them to query databases, and other remote systems results in not only the slowdown of processing thousands of items in a Where-Object, but potentially sending them all over the wire as well.

Since LINQ to events is becoming available through the Rx project, it's more useful, and more important than ever, and we really need a way to write LINQ natively in PowerShell.

[Jaykul]

Oh please, oh please, oh please ... 😉

[BrucePay]

(@powercode you'll probably be interested in this discussion).

Some random thoughts on what implications LINQ/extension methods have for PowerShell...

@Jaykul I added an updated answer to your answer on StackOverflow that replaces all the pipeline use with .where(). This removes the last usage of the pipeline from the code. The result is pretty fast.

@vors The example that @Jaykul provided used XLinq which is perfectly usable from PowerShell as it stands. (XLinq isn't "really" LINQ anyway, it's just an alternative model for manipulating XML).

To see what the experience might be like, I've been playing with wrapping the [Linq.Enumerable] methods in ScriptMethods so I can write things like:

# Find processes with the most threads on this machine
(Get-Process).select{
    param($p) 
    [pscustomobject] @{
        Name=$p.Name; 
        Id=$p.Id; 
        Threads=$p.Threads.Count}}.
            OrderByDescending{param ($p) $p.threads}.
                Take(10)

The result is usually somewhat to much faster than the pipeline, even without lazy evaluation and a lot of copying. But it's also rather more complex to use. Now, while I am personally fond of this sort of thing, is this the experience we want to provide? For example, @mklement0 seems adamantly opposed to pushing methods as a first-class experience. So perhaps we should have a real language experience like from $x where $_ .Size -gt 10 select .... Or a set of cmdlets that built up the query. Or even automatically rewrite parts of the pipeline into LINQ queries to make regular Where/Select/etc. automatically faster. Also note that there is a LINQ for PowerShell module on the gallery. It's incomplete and hasn't been updated in a couple of years (only has 1200 downloads) so maybe people really aren't that keen on LINQ in PowerShell after all.

And finally, what are the core scenarios for doing this? Is it just to be faster (a good thing)? Or is there specific functionality that you want to unlock. What about LINQ-to-SQL? We have classes now. Is this something we should invest in?

[BrucePay]

Another "stunt-query" that gets the 10 most common words from the text of "War and Peace". It uses a few extra operators I created (split(), trim() and match()) that aren't standard query operators.

$h=@{}
[io.file]::ReadAllLines("c:\temp\WarAndPeace.txt").
    trim().
        split().
            match('^[a-z]+$').
                aggregate{param ($x, $y) 
                    if ($x -isnot [hashtable]) { $h[$x]++; $x = $h } $x[$y]++; $x }.
                        select{param ($t) 
                            @($t.GetEnumerator()).
                                select{ param ($w)
                                    [pscustomobject] @{
                                        Word = $w.Key; 
                                        Count = $w.Value}}.
                                            OrderByDescending{$args[0].Count}.
                                                Take(10)} 

[mklement0]

So perhaps we should have a real language experience like from $x where $_ .Size -gt 10 select ....

To quote @Jaykul: Oh please, oh please, oh please ...

The query-like syntax fits better with the rest of PowerShell - a "whitespacey" experience that doesn't cause the confusion with argument mode that method syntax entails (though it's still an embedded language users will have to learn).

rewrite parts of the pipeline into LINQ queries to make regular Where/Select/etc. automatically faster.

Is that feasible? Certainly would be great.

And finally, what are the core scenarios for doing this?

As you say, a performance gain alone would be useful, but LINQ is such a wonderfully expressive, general-purpose feature that I can see its utility in a wide range of scenarios, including LINQ-to-SQL.

[BrucePay]

the confusion with argument mode that method syntax entails

I'm confused :-) What confusion? Method syntax is all expression mode. Every bit of the query I wrote above is in expression mode.

As you say, a performance gain alone would be useful,

Playing with this a bit more, I see that there will be a bound on how much faster we can get. LINQ query operators depend heavily on lambdas or, for us, scriptblocks, So the overhead of invoking a scriptblock will be a limiting factor on how fast we can get. (Perhaps things like PSLambda could help though at the risk of introducing another syntax mode. )

[mklement0]

That's good to know; the PSLambda project sounds interesting; unless a solely method-based approach is chosen (see below), another syntax mode sounds unavoidable, though ideally it would feel like a variant of the existing (main) ones (argument mode and operators).

I'm confused :-) Method syntax is all expression mode.
Every bit of the query I wrote above is in expression mode.

And my head is still spinning.

To recap previous arguments:

Method syntax invites confusion with PowerShell's shell-like argument [parsing] mode (notwithstanding the fact that the option of using method syntax on .NET types directly is a great extensibility feature):

From the perspective of a shell user mostly familiar with command syntax (argument mode), the tricky differences are:

• parentheses are required (vs. whitespace in argument mode)
• arguments must be ,-separated (vs. whitespace in argument mode, where , constructs arrays)
• bareword arguments (unquoted strings) aren't allowed

With the exception of the - little-known and little-used - .Where() and .ForEach collection "operators" (see PowerShell/PowerShell-RFC#126 for how to make them more PowerShell-like), PowerShell's own, core functionality relies only on cmdlets / functions (argument mode) and operators (expression mode, but resembling argument mode), and my vote is to keep it that way.

[BrucePay]

@mklement0

PowerShell's own, core functionality relies only on cmdlets / functions (argument mode)

I really don't understand where you got this idea from. It has never been the case that we relied only on cmdlets. There just wasn't enough time to cover all the scenarios (simple example: getting the substring of a string.) We targeted the most common scenarios with cmdlets because they gave the best user experience and then depended on .NET to back fill what remained.

and operators (expression mode, but resembling argument mode),

What do you mean by "resembling" here? They are distinct modes: barewords are allowed in argument mode but not in expression mode. Operators are allowed in expression mode but not in argument mode. They both allow simple expressions (including method calls) so I suppose that's a resemblance. Method calls are part of expression mode and I can't say I've ever seen someone try to use a method like a command but I've seem a lot of instances where command invocations are written like method calls (and subsequently don't work).