implement indexer in loop

[p0W3RH311]

Hi guys !

loops in powershell is various and its awesome but it missing an indexer for example:

foreach($index,$item in $list) {
    "$index = $item"
}

is more elegant and short:

$index =  0
foreach($item in $list) {
    "$index = $item"
     $index++
}

[doctordns]

What specific problems does this solve?
Wasn't this suggestion recently made and declined?

[p0W3RH311]

What specific problems does this solve?
Wasn't this suggestion recently made and declined?

@doctordns

What specific problems does this solve in many languages like python, ruby, perl, rust, javascript, Go, Scala, Php....etc

[mklement0]

I like the idea; it complements @iRon7's proposal to introduce an automatic $PSIndex variable in the pipeline: #13772

@doctordns, it's helpful syntactic sugar; just a couple of use cases off the top of my head; I'm sure there are more.

^{Note: The original syntax was changed to reflect the later discussion; the index variable now comes after the iteration expression, separate with ;}

# Parellel array processing
$a = 'foo', 'bar'
$b = 'baz', 'qux'
$c = 'and' 'so on'
foreach ($aElem in $a; $i) { # WISHFUL THINKING
  $bElem, $cElem = $b[$i], $c[$i]
  # ...
}

# Get only every 2nd element
$a = 'foo', 'bar', 'baz', 'quux'
foreach ($aElem in $a; $i) { # WISHFUL THINKING
  if ($i % 2) { continue }
}

As a largely moot aside: Quite some time ago, in #3830, I had proposed allowing syntax such as foreach ($a) { ... } - not having to specify an iteration variable explicitly and defaulting to $_ / $PSItem; the proposal was rejected, but it would have allowed us a unified approach with respect to an automatic $PSIndex as well.

[rkeithhill]

I'd move that indexer variable to after the condition and if it's optional you don't break existing code e.g.:

foreach ($item in $items, $ndx) { ... }

FWIW it do not like the idea of an automatic variable like $PSIndex. That falls apart in nested loops.

[mklement0]

@rkeithhill, foreach ($item in $items, $ndx) { ... } is not an option due to ambiguity - such a statement already works, and loops over the array $items, $ndx.

However, ; as the separator would work (currently a syntax error): foreach ($item in $items; $ndx) { ... }

falls apart in nested loops.

In nested loops, $PSIndex could refer to the respective enclosing loop.
If referencing indices across the loop hierarchy is needed, the explicit index-variable syntax proposed here can be used.

[rkeithhill]

Good point on the , versus ;. In nested loops, it is not uncommon at all to want to access the indexer of outer loops which is tricky with a single variable e.g.:

foreach ($item in $items; $ndx1) {
     foreach ($child  in $item.Children; $ndx2) {
         ....
     }
 }

[p0W3RH311]

$psindex in my opinion as magic variable is better because:

it's not break change and it not break the loop:

foreach($item in $list) {
   "$psindex => $item"
}

[rkeithhill]

IMO PowerShell already has too many magic variables. And to be clear, I do like the overall concept.

[p0W3RH311]

@rkeithhill

yes ps has many magic variable but it not has an indexer magic variable

[p0W3RH311]

another idea is to implement an indexer in $foreach enumerator

foreach($item in $list) {
   $foreach.index
}

[rkeithhill]

I think you wouldn't get too far before folks demanded a way to access an outer loop's indexer. At that point, I don't think you need (or want) two ways to access indexers.

[mklement0]

Agreed, @rkeithhill, but my point was that you would have a choice, i.e. I was advocating for both proposals to be implemented: if you do need cross-hierarchy index access, use the explicit syntax; otherwise, rely on $PSIndex for convenience. But, again: this is really a moot point, because $PSIndex should only be implemented in tandem with allowing foreach ($collection) { ... } (implicit iterator variable $_ / $PSItem), which, as stated, has been rejected.

In short: I think we can conclude this part of the discussion, unless enough people feel strongly enough to revisit this, in a separate proposal.

---

Come to think of it: foreach ($item in $items; $ndx) { ... } (; as the separator, index variable specified after) is preferable due to avoiding potential conceptual confusion due to $ndx, $item in $items looking like a destructuring assignment.

Also, it would leave the door open for a further, separate enhancement that could further simplify the parallel-collection enumeration use case by using a destructuring assignment as follows (which could still be combined with an explicit index variable):

$a = 'foo', 'bar'
$b = 'baz', 'qux'
$c = 'and' 'so on'
foreach ($aElem, $bElem, $cElem in $a, $b, $c) { # WISHFUL THINKING
   # $aElem is $a[<ndx>]
   # $bElem is $b[<ndx>]
   # $cElem is $c[<ndx>]
}

The largest among the RHS collections would drive the number of iterations, and the iterator variables would contain $null for those collections that have run out of items.

Update: See #14732

[p0W3RH311]

@rkeithhill

I think the common point is that powershell lacks an indexer variable like many other languages ​​... now how to implement this indexer ... there are several ways on the design side:

powershell way

foreach($item in $list) {
   $foreach.index # another property
}

or

foreach($item in $list) {
   $PSindex # magic variable
}

the traditional way

foreach($i, $item in $list) {
   $foreach.index
}

[rkeithhill]

@p0W3RH311 I'm with you on the "common point". I like the feature suggestion. I'm just not a fan of using a magic variable for the indexer. 🤷‍♂️

[237dmitry]

another idea is to implement an indexer in $foreach enumerator

foreach($item in $list) {
   $foreach.index
}

It probably won't work with $foreach. What will $foreach.index be in the case of $foreach.Reset()?

$ind = 0; foreach ($i in 'a','b','c')    
 {                                       
     $foreach.Current                    
     if ($ind -eq 1) { $foreach.Reset() }
     $ind++                              
 }                                       

[SeeminglyScience]

What specific problems does this solve in many languages like python, ruby, perl, rust, javascript, Go, Scala, Php....etc

This isn't a very compelling argument imo. There are a lot of patterns in other languages that just don't make as much sense in PowerShell.

It's definitely useful occasionally to have an index in PowerShell, but with the way pipelines work it's significantly less important. Personally I worry that this would be used too infrequently to warrant language changes.

[mklement0]

Agreed re the general point that not all patterns are a good fit for PowerShell, however:

the way pipelines work it's significantly less important.

Can you elaborate on how the pipeline mechanism makes this feature less important?

#13772 makes a pretty good case for a an automatic $PSIndex variable in the pipeline; for symmetry alone it makes sense to provide an analogous feature in foreach statements (with an explicit index variable, analogous to how you need an explicit iterator variable).

To me, it's a useful enhancement, both in ForEach-Object / Where-Object-based pipelines (and, by extension, in the .ForEach() and .Where() methods) and foreach loops.

• In the former case, implementing $PSIndex is trivial - no new syntax needed, just a new variable you're free to use or not.

• In the latter case, explicit (opt-in) syntax is needed, but foreach ($name in 'foo', 'bar'; $ndx) { ... } strikes me as a perfectly reasonable extension to the current syntax (the ; should already be familiar from for loops).

[SeeminglyScience]

Can you elaborate on how the pipeline mechanism makes this feature less important?

In other languages it's very common to allocate an array up front and then populate it. Or mutate an array in place. In PowerShell it's significantly easier to just let the pipeline make one for you.

#13772 makes a pretty good case for a an automatic $PSIndex variable in the pipeline;

With the exception of "Combine single arrays into columns" I don't really see any real world examples in that issue. I think it's just as unlikely to see all that much usage.

for symmetry alone it makes sense to provide an analogous feature in foreach statements

Symmetry is nice, but I disagree that it's a compelling enough reason on it's own.

[mklement0]

• An automatically maintained index provides utility irrespective of whether it is used in a streaming pipeline or during iteration of an in-memory array.

• Because it is useful in both scenarios, it should be implemented in both (that is to say, the need for symmetry is driven by the feature's utility).

This comes down to whether one indeed sees the utility, which is ultimately a subjective assessment. Let's see how the community feels.

[mklement0]

I've created a new issue for the separate foreach ($aElem, $bElem, $cElem in $a, $b, $c) { ... } proposal mentioned above (enumerating multiple collections in tandem): #14732

Also, to make it more obvious how people feel about this issue, I encourage everyone who was commented so far to give the initial post here a thumbs-up or -down.

[SeeminglyScience]

• An automatically maintained index provides utility irrespective of whether it is used in a streaming pipeline or during iteration of an in-memory array.

Sorry, to clarify, I was giving examples of some of the most common scenarios one would use an index in other languages. Scenarios that are significantly less common in PowerShell.

[mklement0]

I see, but the fact there are additional scenarios in other languages where indices are not only useful but required as an intrinsic part of the enumeration is ultimately a moot point. To put it differently: this argument doesn't preclude utility in PowerShell scenarios.

The relevant question is: Is the feature useful in PowerShell?

It is to me, and I have personally wished for it in the past; we'll see how the community feels over time.

[SeeminglyScience]

I see, but the fact there are additional scenarios in other languages where indices are not only useful but required as an intrinsic part of the enumeration is ultimately a moot point. To put it differently: this argument doesn't preclude utility in PowerShell scenarios.

That's correct. It was in response to your question asking why the pipeline makes it less important in PowerShell.

It is to me, and I have personally wished for it in the past; we'll see how the community feels over time.

I think one of the best ways to show the utility would be to point to some examples of existing code that would greatly benefit from this. Like some code in an already published module or script that would be simplified significantly.

I don't doubt that it has uses. I've wanted it myself a few times, but in all of those few times either a for loop, incrementing my own var or just a different pattern was perfectly suitable. Remember that language changes need a pretty dramatic effect to be worth it.

[p0W3RH311]

What specific problems does this solve in many languages like python, ruby, perl, rust, javascript, Go, Scala, Php....etc

This isn't a very compelling argument imo. There are a lot of patterns in other languages that just don't make as much sense in PowerShell.

Personally I worry that this would be used too infrequently to warrant language changes.

in PS there are many variable too infrequently used like $^ and $$ but this variable exist...indexer is usefull and perphaps is more usefull in future....these variable ($^ and $$...etc) and others esoteric like ${c:\hello.txt} = 'foo' exist but nobody said they are useless..

[mklement0]

@SeeminglyScience:

Fair point, but I was confused by your mentioning the pipeline specifically, because enumeration of collections with foreach ($element in $collection) too obviates the need for indices (as it does in the equivalent C# statement, for instance), if (read-only) enumeration alone is needed.

Optionally - if the business logic rather than the technical underpinnings requires it - having an automatically maintained index at one's disposal, without having to forgo the convenience of foreach ($element in $collection) seems beneficial to me.

need a pretty dramatic effect to be worth it.

Point taken, but even though we are undoubtedly talking about syntactic sugar here, such sugar is often important for sweetening the developer experience, if you will.

I'll see if I can come up with examples in published code and/or additional compelling examples.
I'm hoping others will too.

[SeeminglyScience]

Fair point, but I was confused by your mentioning the pipeline specifically, because enumeration of collections with foreach ($element in $collection) too obviates the need for indices (as it does in the equivalent C# statement, for instance), if (read-only) enumeration alone is needed.

Ah yes sorry it wasn't obvious what I was referring to, but most things use the pipeline plumbing at some point. For example:

$a = foreach ($b in (0..30)) { $b }

# or 

$a = gci

# or 

$a = $(0; 1)

Behind the scenes that'll use a very similar code path to build $a even though there is no PipelineAst language wise (except in the gci example). I guess it would be more accurate to refer to it as "output semantics" maybe?

[mklement0]

Good point, @SeeminglyScience, and thanks for clarifying.

Even though it's technically not entirely accurate, I generally suggest using and interpreting the term "pipeline" as informal shorthand for "connecting commands with a pipe symbol" (including the case where the first segment is an expression; e.g. 1..10 | ForEach-Object { "[$_]" }), as distinct from the world of expressions (e.g. 1 + 1) and statements
(e.g. foreach ($num in 1..10) { "[$num]" }).

Here, we have the contrast between the ForEach-Object command with its (invariably) implicit iterator variable, $_ ($PSItem) and the foreach statement with its (invariably) explicit, user-chosen iterator variable.

Therefore, in the pipeline (loosely speaking), the proposed automatic index variable must be implicit too, with a name such as $PSIndex, as proposed by @iRon7 in #13772.

(As stated, it would make sense to me to also allow implicit iterator variables in foreach statements too -
foreach (1..10) { "[$_]" } - but only if $_ / $PSItem were to be supported - which was rejected - would it make sense to me to also support $PSIndex)

---

The following uses cases are more directly relevant to #13772, but since the unified discussion seems to be happening here, here goes:

In the pipeline, it isn't just ForEach-Object and Where-Object that would benefit from $PSIndex, but also:

• delay-bind script blocks
• calculated properties

In both cases the script blocks run in a child scope - see #7157 - which makes maintaining a cross-invocation index variable via the parent scope both obscure and cumbersome:

Consider the scenario of renaming files to names that incorporate a sequence number via a delay-bind script block (a real-world scenario that comes up repeatedly on Stack Overflow):

# Delay-bind script block
# Cumbersome and obscure, because you must refer to the $i variable in the *parent* scope.
# Renames the input files to "file1.txt", "file2.txt", ...
$i = 0
Get-Item *.txt | Rename-Item -NewName { "file" + ++(Get-Variable -Scope 1 i).Value + ".txt" } -WhatIf

With $PSIndex this simplifies to:

# WISHFUL THINKING.
Get-Item *.txt | Rename-Item -NewName { "file" + (1 + $PSIndex) + ".txt" } -WhatIf

The same applies to a (script block-based) calculated property; consider the case of wanting to create sequence number-based identifiers via Select-Object:

# WISHFUL THINKING.
Get-Item *.txt | Select-Object @{ n='Id'; e={ 1 + $PSIndex } }, FullName

While the scoping problem doesn't arise with the foreach statement, an automatic indexer still makes for a nice, expressive simplification:

# WISHFUL THINKING
foreach ($file in Get-Item *.txt; $i) {
  [pscustomobject] @{
    Id = 1 + $i
    FullName = $file.FullName
  }
}

[SeeminglyScience]

Here, we have the contrast between the ForEach-Object command with its (invariably) implicit iterator variable, $_ ($PSItem) and the foreach statement with its (invariably) explicit, user-chosen iterator variable.

Yeah it doesn't use the pipeline for enumeration, though it can emit to an output pipe still.

# Delay-bind script block
# Cumbersome and obscure, because you must refer to the $i variable in the *parent* scope.
# Renames the input files to "file1.txt", "file2.txt", ...
$i = 0
Get-Item *.txt | Rename-Item -NewName { "file" + ++(Get-Variable -Scope 1 i).Value + ".txt" } -WhatIf

If you're renaming files, aren't you going to want to keep the number of the existing name? Like if you're renaming FileX.txt to SomethingX.txt you probably don't want the index to be based on the order they're returned in most cases. The only time I can really think of that I've needed to do this is creating test files, where you typically do 0..10 | % { New-Item File$_.txt } anyway.

The rest of the examples are similar in that while it is an example of how it could be used, it's not clear why you'd need to.

[mklement0]

though it can emit to an output pipe still.

If I understand correctly, directly only stand-alone (possibly in the context of an assignment); in a pipeline (with |), only via $() / @{) (collecting all output up front) or via & { ... } / . { ... } (streaming), but I think we have clarity now on the syntactic scenarios we want to contrast.

If you're renaming files, aren't you going to want to keep the number of the existing name?
it's not clear why you'd need to.

• There may be no numbers (lexical order), or the numbering may be different (e.g., start with an offset) and/or may have gaps.
• Renaming files is just one example; needing to produce sequence numbers is a common requirement in many different scenarios, as is index-based filtering.

[SeeminglyScience]

If I understand correctly, directly only stand-alone (possibly in the context of an assignment); in a pipeline (with |), only via $() / @{) (collecting all output up front) or via & { ... } / . { ... } (streaming), but I think we have clarity now on the syntactic scenarios we want to contrast.

Anything that isn't captured or in a class method is emitted to the pipeline.

For example:

This is what the script `0` compiles to

private static void <ScriptBlock>(FunctionContext funcContext)
{
    try
    {
        context = funcContext._executionContext;
        locals = ((MutableTuple<object, Object[], object, object, PSScriptCmdlet, PSBoundParametersDictionary, InvocationInfo, string, string, Null, Null, Null, Null, Null, Null, Null>)funcContext._localsTuple);
        funcContext._functionName = @"<ScriptBlock>";
        funcContext._currentSequencePointIndex = 0;

        context._debugger.EnterScriptFunction(funcContext);
        try
        {
            funcContext._currentSequencePointIndex = 1;
            if (context._debuggingMode > 0)
            {
                context._debugger.OnSequencePointHit(funcContext)
            }

            funcContext._outputPipe.Add(((object)0));
            context.QuestionMarkVariableValue = true;
        }
        catch (Exception exception)
        {
            ExceptionHandlingOps.CheckActionPreference(funcContext, exception);
        }

        funcContext._currentSequencePointIndex = 2;
        if (context._debuggingMode > 0)
        {
            context._debugger.OnSequencePointHit(funcContext);
        }
    }
    finally
    {
        context._debugger.ExitScriptFunction();
    }
}

The relevant line being:

funcContext._outputPipe.Add(((object)0));

• There may be no numbers (lexical order), or the numbering may be different (e.g., start with an offset) and/or may have gaps.
• Renaming files is just one example; needing to produce sequence numbers is a common requirement in many different scenarios, as is index-based filtering.

My suggestion if your aim is to convince someone to pick up this work (or convince someone that this should be brought up with the committee) would be to use real world examples. Theoretical examples aren't as helpful for determining the level of impact a feature could provide.

[mklement0]

Anything that isn't captured or in a class method is emitted to the pipeline.

Yes, but my point was that you can't do foreach ($i in 1..3) { $i } | Measure-Object, for instance - you need & { foreach ($i in 1..3) { $i } } | Measure-Object, for instance; that is, you cannot use a statement in a multi-segment pipeline - see #10967 (comment)

My suggestion if your aim is to convince someone to pick up this work (or convince someone that this should be brought up with the committee) would be to use real world examples.

Honestly, to me the examples given so far - both concrete and abstract - are so self-evidently compelling that I don't see value in spending more time on finding real-world examples.

Perhaps others feel inspired to do so, and perhaps the count of thumbs-up on the issue will make a statement of its own over time.

The primary hurdle, I'd say, is to get this - and #13772 - committee-reviewed; I wouldn't want anyone spending time on an implementation without knowing that the feature will be approved.
@iSazonov, any thoughts?

[iSazonov]

Just yesterday, I thought PowerShell is at the same conceptual level as it was 20 years ago and is still at the foot of a magnificent tower - we haven't even climbed one floor in all this time. One must have limitless inspiration to develop primary ideas to these new heights.
Thinking so, I believe that it makes no sense to invest too much in traditional operators, but in PowerShell pipeline makes sense.

[mklement0]

Thanks, @iSazonov. I too value the conceptual elegance of PowerShell's OO pipeline and welcome improvements to it, but, as is often the case, this is not an either-or proposition (more on that later).

Unquestionable, what was initially proposed here in the context of the foreach statement and what is proposed with respect to the pipeline (loosely speaking) in #13772 can be decided on separately - even though I personally think that if it's worthwhile in one context, it is worthwhile in the other (see below).

So, if you already think #13772 is worthwhile, please tag it for committee review.

(The generalization of #13772, which started with a focus on ForEach-Object and Where-Object is: provide an automatic index variable in any collection-processing script-block context where $_ / $PSItem is defined, which also includes calculated properties and delay-bind script-block arguments, the .ForEach() and .Where() methods, the switch statement, and the -replace operator (possibly more).)

---

it makes no sense to invest too much in traditional operators

Cmdlets such as ForEach-Object and language statements such as foreach loops complement each other and are appropriate for different scenarios.

Neither is going away.

Both should provide an expressive developer experience with (fundamental) feature parity (which, regrettably, already falls short in one case: the .Where() method has useful features that Where-Object lacks: see #13834)

A foreach loop far outperforms a ForEach-Object-based solution, and that alone makes it indispensable. This is especially pronounced with collections that are already in memory in full, but it even applies to foreach ($obj in <command>); it also outperforms the .ForEach() method.

It comes down to a tradeoff between memory use and performance, and while in many cases you may be able to choose any of these without too much real-world impact, there are cases where are forced to choose one (foreach for performance) or the other (ForEach-Object to avoid running out of memory).

All three serve the same fundamental purpose: iterating over something enumerable (loosely speaking; strictly speaking, it is the pipeline itself that does the enumeration in the case of ForEach-Object)

All three are expressive in that you needn't worry about the details of the enumeration: is it an index-based collection or a (potentially lazy) enumerable?

ForEach-Object / .ForEach() do not even require you to pick a variable name for the enumeration element at hand; the name is fixed in the form of the automatic $_ / $PSItem variable.

Providing an automatic index indicating the 0-based position of the element at hand in the enumeration therefore similarly calls for such a variable, $PSIndex, as proposed in #13772

By contrast, foreach does require you to pick an iterator variable name - $_ / $PSItem isn't supported.
Therefore, it makes sense to also require picking an variable name for the automatic index - unlike with $PSItem, this necessitates an addition to the syntax, the proposed foreach ($element in $enumerable; $index) { .. }.

If such an automatically maintained index makes sense with ForEach-Object (among others), it also makes sense in a foreach loop - especially given that, as stated, sometime you have no choice but to use foreach over ForEach-Object.

---

I've said it before: the examples in #13772 and here to me amply demonstrate that such an automatic index is called for as expressive syntactic sugar that serves real-world needs, which, as @p0W3RH311, has pointed out, several other languages have recognized too (notably including the systems programming language Rust)

Syntactic sugar isn't a luxury; it's what makes a language enjoyable to use and boosts productivity.

And here's a secret: foreach ($element in $enumerable) { $element } is itself syntactic sugar:

• In the case of an index-based enumerable, it is syntactic sugar for:
  for ($i=0; $i -lt $enumerable.Count; ++$) { $enumerable[$i] }

• In the case of an enumeration-interface-based enumerable, it is syntactic sugar for:
  while ($element = $enumerable.MoveNext()) { $element }

And with foreach you don't even have to worry which kind of enumerable you're dealing with.

Also note how the for loop forces you to deal with indices - even though you may not actually need them.
Conversely, if you do need them in the .MoveNext() while loop, you have to add them manually.

• foreach already takes the drudgery out of the enumeration part.

• Wouldn't it be nice if it also took out the drudgery of the index part, if and when you actually need an index, for business-logic reasons, not as a syntactic necessity?

[Jaykul]

I have to say that whenever I've needed something like this, I've just changed to a for(){} loop, or added a counter variable.

I don't mind the idea of adding syntax to foreach(){} to define a counter variable to be incremented for me, but I'm not convinced the savings are worth it. I mean, is this:

foreach($item in @($List); $index) {           
  $item | add-member Index $index -PassThru    
}                                              

Really better than this:

$index=0; foreach($item in @($List)) {         
  $item | add-member Index ($index++) -PassThru
}

Remember that it's not just about saving you a few keystrokes, it's also about clarity and readability ...

To make it clearer, we could add a parameter (there's precedent in switch):

foreach($item in @($List)) -counter index {           
  $item | add-member Index $index -PassThru    
}

[mklement0]

Really better than this

To me it is unequivocally better:

• no need to declare a variable outside the loop (even though based on PowerShell's scoping rules even the syntactic-sugar index variable would live on beyond the loop, as the iteration variable already does, but that's a separate issue)

• no messing with $index++ for incrementing, which is notoriously error-prone.

Remember that it's not just about saving you a few keystrokes, it's also about clarity and readability ...

Agreed, but to me foreach($item in @($List); $index) fulfills that criterion (syntactically simple, easily documented and remembered) - even though it isn't as verbose as -counter index, but I am wary of this mixing of PowerShell's parsing worlds (argument vs. expression mode):

Yes, switch is the precedent, but the only one, and an awkward one at that; e.g., in switch -File file.txt { ... }, file.txt is parsed as an argument and may therefore - despite being a string - be unquoted; by contrast, a direct input object must be supplied inside (...), which forces a new parsing context where a bareword string is not supported, as it would be interpreted as a command; e.g., switch (mystring) { ... } looks for a command named mystring. Similar confusion can arise in the branch conditions - see #3668.

[rjmholt]

After discussing this with the Engine working group, we don't think this should be baked in at the language/syntax level. This is probably best accomplished with a function like Python's enumerate. This could be implemented in an external module and later evaluated for inclusion in PowerShell.