Close the feature gap between the `Where-Object` cmdlet and the `.Where()` array method, introduce -First, -Last, ... switches

[mklement0]

Summary of the new feature/enhancement

Update for clarification: The gist of this proposal is to bring valuable filter functionality that currently only exists in the in-memory .Where() array method to its pipeline counterpart, Where-Object.
(Always using .Where() isn't an option with large input sets, because you'd have to collect all input in memory first.)

The .Where() array method, which is the in-memory equivalent of the pipeline-based Where-Object cmdlet, offers a number of additional features, such as the ability to optionally stop matching after the first match is found, which can be an important optimization technique, and the ability to output all objects that come before/after a matching one:

First / -First example:

# OK - very fast, because - despite the large input collection - processing stops after the first match.
(1..1e6).Where({ $_ -eq 10 }, 'First')  # -> 10

# WISHFUL THINKING: Add a -First switch (among several others - see below).
1..1e6 | Where-Object { $_ -eq 10 } -First

# Without it, ALL elements are processed - SLOW
1..1e6 | Where-Object { $_ -eq 10 }

# Current workaround is cumbersome.
1..1e6  | Where-Object { $_ -eq 10 } | Select-Object -First 1

SkipUntil / -SkipUntil example:

(1, 2, 42, 43).Where({ $_ -eq 42 }, 'SkipUntil') # -> 42, 43

# WISHFUL THINKING: Add a -SkipUntil switch.
1, 2, 42, 43 | Where-Object { $_ -eq 42 } -SkipUntil

Based on the current .Where() features, the following switches should be introduced with the same behavior, mirroring the WhereOperatorSelectionMode enumeration values:

-First, -Last, -SkipUntil, -Until, -Split

Note:

• -Split partitions the input into two and returns two collections, so it would in effect require collecting all input in memory first before producing output, which would have to be clearly documented.

• The .Where() method has an optional numeric parameter, numberToReturn, that modifies the operations, such as 2 with First returning the first two matches; since we don't have a syntax for switches with optional (non-Boolean) arguments for commands (see Feature Request: Variable Parameter with switch-like capability #12104), we have the following options:

  • Option A: Simply omit this aspect of the functionality and invariably default to 1, which is likely fine in the majority of cases; if a different number is needed, Select-Object -First/-Last $n can be piped to.

  • Option B: Implement -First and -Last not as _switches, but as -First and-Last ` ; the potential downside is that a number argument is then mandatory.

  • Option C: Given that numberToReturn also modifies all other functionality - though there's likely less of a need for that - implement a separate -Count <int> parameter, which in the absence of any of the switches would imply -First.

• For these behaviors to be implemented efficiently, they have to stop the pipeline on demand, as Select-Object already does. However, the latter does so in a problematic fashion - not giving other cmdlets a chance to run their End blocks - which should be addressed (independently) as well: see Select-Object -First X skips End{ } blocks of previous cmdlets #7930

On a meta note: You can skip the obsolete comments that follow and resume reading at this comment.

[vexx32]

Eh, I'd say the only reason .Where() handles that is because we don't have a .Select() magic method.

I don't see a real need to add this to Where-Object given that Select-Object already performs this function. 🤷

[mklement0]

@vexx32:

Select-Object is for transforming input and/or extracting objects by position, which is completely separate from the filtering that Where-Object performs.

As an aside, there's no need for a .Select() array method, because the .ForEach() array method and indexing fulfill that role for in-memory collections.

The point is that .Where() provides useful functionality for in-memory collections that would be equally useful for Where-Object's pipeline processing, and in the case of -First offers an important performance optimizations.

[vexx32]

Select-Object -First $x already short-circuits the pipeline.

Adding it to Where-Object as well would 1) duplicate the code path (unless you move that code somewhere common) and 2) not really gain a whole lot above that, I wouldn't think.

[mklement0]

You misunderstand the intent of what I'm proposing:

• Select-Object -First $n unconditionally selects the first $n input objects (possibly modified by the equally positional -Skip $m).

• Where-Object { <condition> } -First would select (only) the first input object that matches the filter criterion.

E.g.:

# WISHFUL THINKING
PS> 'foo', 'bar', 'baz', 'bar', 'bar' | Where-Object { $_ -eq 'bar' } -First
bar  # only the first 'bar' match

The above is the pipeline equivalent of the following in-memory operation based on the Where()method:

PS>  ('foo', 'bar', 'baz', 'bar', 'bar').Where({$_ -eq 'bar' }, 'First')
bar

[vexx32]

No, I understand well enough.

I'm just saying I don't think that's meaningfully or usefully better than just doing $stuff | Where-Object { $_ -eq 'bar' } | Select-Object -First 1

[mklement0]

No, I understand well enough.

I guess I was confused by you repeating the cumbersome workaround already shown in the OP.

First, the -First and -Last switches alone have two benefits:

• primarily: concision

• secondarily: performance (though probably negligible in practice)

Second, the -SkipUntil, -Until, -Split functionality doesn't even have straightforward-but-cumbersome Select-Object workarounds.

These are natural extensions, especially given that the conceptually equivalent method already offers these features.

On a more general note: Duplicating functionality is nothing new, and has proven helpful in other cases:

Get-Content -First 2 file.txt
# vs.
Get-Content file.txt | Select-Object -First 2

'foo' | Select-String 'o' -Raw
# vs.
'foo' | Select-Sting 'o' | Select-Object -ExpandProperty Line

[vexx32]

In the case of Get-Content it's able to take additional shortcuts due to how the IO reading works.

For the case of whether there's a switch/parameter for Where-Object vs Select-Object I think there's even less reason to do it. If you really want it I guess I have nothing specifically against it, I just don't consider it worth the time to do. 🙂

[mklement0]

While that is a matter of preference with respect to -First and -Last, your Select-Object workaround doesn't apply to the remaining 3 switches proposed: -SkipUntil, -Until, -Split

[vexx32]

Yeah, those are probably worth adding and more aligned with Where-Object's general role IMO.

[mklement0]

@vexx32, let me try to summarize, and, if we agree, I suggest we hide all previous comments:

The gist of this proposal is to bring valuable filter functionality that currently only exists in the in-memory .Where() array method to its pipeline counterpart, Where-Object.
(Always using .Where() isn't an option with large input sets, because you'd have to collect all input in memory first.)

The proposed new switches, -First, -Last, -SkipUntil, -Until, -Split would directly correspond to the WhereOperatorSelectionMode operation enumeration values (except for Default) accepted by the .Where() method.

You, @vexx32, disagree with this proposal in part:

• You agree that the functionality of -SkipUntil, -Until, -Split is worth adding,
• while thinking that -First and -Last aren't needed, because you can pipe to Select-Object -First 1 and Select-Object -Last 1 instead and therefore do not strictly need to this functionality in Where-Object itself.

I disagree with this for the following reasons:

• Concision and convenience: 1..1e6 | Where-Object { $_ -eq 10 } -First is much shorter to type than
  1..1e6 | Where-Object { $_ -eq 10 } | Select-Object -First 1

  • There is precedent for duplicating functionality in-cmdlet that can also be had by combining cmdlets, notably Get-Content -First $n instead of Get-Content | Select-Object -First $n and Select-String -Raw instead of Select-String | Select-Object -ExpandProperty Line

• Symmetry: Only adding some of the operations of the analogous .Where() method is awkward.

• Performance: While the performance gain is negligible in the -First case, with -Last it matters:

  • If many input objects match in a Where-Object call, an internal -Last implementation could keep only the most recent match and replace it if another one comes around; by contrast, if the -Last functionality is provided by Select-Object, Where-Object would send all matches through the pipeline before Select-Object can select the last.

[SeeminglyScience]

What about new commands? Select-Object -First 1 doesn't feel great interactively. That's why I have first, last, skip, and at (indexing) defined in my profile (Source).

e.g.

gci | skip 2 | at -1 | % children | first

Also would be nice to be able to stop the pipeline without killing it completely like Select-Object -First 1 does (well, an easier way then the nightmare of reflection and implementation detail abuse I linked above).

[KirkMunro]

If this moves forward, I'd love to see -First and -Last implemented as <Switchable[int]> values, so that you don't need to provide a number if you are getting the very first one or the very last one. If you don't know what I mean, I proposed on another PR that there are scenarios where we might want a parameter to have a default value when used as a switch, or accept a specific value as input as well. This is one of those scenarios.

Are these new parameters being proposed only for the -ScriptBlock parameter sets? Or for all parameter sets. Maybe we could have a new poster boy for parameter set nightmares. Let's see...31 parameter sets times 5 new incompatible, optional switches per parameter set, so 135 parameter sets in total. 🤣

Also FWIW, I would agree with all 5 new parameters to cover all options, especially given @mklement0's argument for -Last.

[mklement0]

@SeeminglyScience, those are interesting ideas, but it's really a separate discussion.

@KirkMunro :

😁 re parameter sets, though perhaps that will inspires us to improve matters there, such as implementing simpler mutual exclusion... - ideally, we wouldn't have to restrict this to the script-block parameter set solely for technical reasons.

Re <Switchable[int]> - something like that was also proposed in #12104, which I mention in the OP (couldn't find the PR you're referring to).

While I like the idea in general, in this particular case I'm leaning toward option C from the (updated) OP: A separate, optional -Count <int> parameter, given that such a quantifier applies to all operations supported by the .Where() method - we would need that for full feature parity.