Builtin variable for loop index in foreach

[p5pRT]

Migrated from rt.perl.org#130038 (status was 'rejected')

Searchable as RT130038$

[p5pRT]

From @epa

Created by @epa

Perl's foreach loop is very handy. But sometimes you write some code
where you realize you need the array index from 0 instead. So where
you had

  foreach my $thing (@​things) {
  do_something($thing);
  }

you now have to rewrite your code as

  foreach my $i (0 .. $#things) {
  my $thing = $things[$i];
  say "processing thing number $i";
  do_something($thing);
  }

Moreover, to get back the original semantics (where $thing is an alias
to each item in the list) requires more code still.

I was wondering, if perl knows internally how far along in @​things we
are, perhaps it could expose that list index to the programmer? This
would be both more concise and more efficient, and avoid the need to
rewrite code when it turns out you do want the index after all.

Suppose that the currently unused $# variable were used for this.
Then you could write

  foreach my $thing (@​things) {
  say "processing thing number $#";
  do_something($thing);
  }

There are snags with this (even assuming it could be implemented with
little work). The $# variable would get set to the array index
applying to the foreach loop you are currently inside. If having
nested loops the programmer would have to be a bit careful, or else
fall back to doing it the longhand way with an $i variable. But this
problem has already been addressed with the default $_ used in foreach​:

  foreach (qw(a b c)) { say $_; foreach (qw(x y z)) { say $_ } say $_ }

and I imagine a similar rule could apply to $#.

I suggest this new feature because I hope it is something perl already
knows and can expose to the programmer. If so, it would be a useful
enhancement to the language. If implementing it would be more
difficult, obviously the perl5-porters may feel there are better
things to work on. Thanks for considering it.

Perl Info

Flags:
    category=core
    severity=wishlist

Site configuration information for perl 5.22.2:

Configured by Red Hat, Inc. at Wed Aug  3 14:06:20 UTC 2016.

Summary of my perl5 (revision 5 version 22 subversion 2) configuration:
   
  Platform:
    osname=linux, osvers=4.6.3-300.fc24.x86_64, archname=x86_64-linux-thread-multi
    uname='linux buildvm-08.phx2.fedoraproject.org 4.6.3-300.fc24.x86_64 #1 smp fri jun 24 20:52:41 utc 2016 x86_64 x86_64 x86_64 gnulinux '
    config_args='-des -Doptimize=none -Dccflags=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches  -m64 -mtune=generic -Dldflags=-Wl,-z,relro  -Dccdlflags=-Wl,--enable-new-dtags -Wl,-z,relro  -Dlddlflags=-shared -Wl,-z,relro  -Dshrpdir=/usr/lib64 -DDEBUGGING=-g -Dversion=5.22.2 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dprefix=/usr -Dvendorprefix=/usr -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl5 -Dsitearch=/usr/local/lib64/perl5 -Dprivlib=/usr/share/perl5 -Dvendorlib=/usr/share/perl5/vendor_perl -Darchlib=/usr/lib64/perl5 -Dvendorarch=/usr/lib64/perl5/vendor_perl -Darchname=x86_64-linux-thread-multi -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Duseshrplib -Dusethreads -Duseithreads -Dusedtrace=/usr/bin/dtrace -Duselargefiles -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dscriptdir=/usr/bin -Dusesitecustomize'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fwrapv -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='  -g',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fwrapv -fno-strict-aliasing -I/usr/local/include'
    ccversion='', gccversion='5.3.1 20160406 (Red Hat 5.3.1-6)', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags ='-Wl,-z,relro  -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib64 /lib64 /usr/lib64 /usr/local/lib /usr/lib /lib/../lib64 /usr/lib/../lib64 /lib
    libs=-lpthread -lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
    perllibs=-lpthread -lresolv -lnsl -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.22.so, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.22'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,--enable-new-dtags -Wl,-z,relro '
    cccdlflags='-fPIC', lddlflags='-shared -Wl,-z,relro  -L/usr/local/lib -fstack-protector-strong'

Locally applied patches:
    Fedora Patch1: Removes date check, Fedora/RHEL specific
    Fedora Patch3: support for libdir64
    Fedora Patch4: use libresolv instead of libbind
    Fedora Patch5: USE_MM_LD_RUN_PATH
    Fedora Patch6: Skip hostname tests, due to builders not being network capable
    Fedora Patch7: Dont run one io test due to random builder failures
    Fedora Patch15: Define SONAME for libperl.so
    Fedora Patch16: Install libperl.so to -Dshrpdir value
    Fedora Patch22: Document Math::BigInt::CalcEmu requires Math::BigInt (CPAN RT#85015)
    Fedora Patch26: Make *DBM_File desctructors thread-safe (RT#61912)
    Fedora Patch27: Make PadlistNAMES() lvalue again (CPAN RT#101063)
    Fedora Patch28: Make magic vtable writable as a work-around for Coro (CPAN RT#101063)
    Fedora Patch29: Fix duplicating PerlIO::encoding when spawning threads (RT#31923)
    Fedora Patch30: Do not let XSLoader load relative paths (CVE-2016-6185)
    Fedora Patch31: Avoid loading optional modules from default . (CVE-2016-1238)
    Fedora Patch200: Link XS modules to libperl.so with EU::CBuilder on Linux
    Fedora Patch201: Link XS modules to libperl.so with EU::MM on Linux


@INC for perl 5.22.2:
    /home/eda/lib64/perl5/
    /usr/local/lib64/perl5
    /usr/local/share/perl5
    /usr/lib64/perl5/vendor_perl
    /usr/share/perl5/vendor_perl
    /usr/lib64/perl5
    /usr/share/perl5


Environment for perl 5.22.2:
    HOME=/home/eda
    LANG=en_GB.UTF-8
    LANGUAGE (unset)
    LC_COLLATE=C
    LC_CTYPE=en_GB.UTF-8
    LC_MESSAGES=en_GB.UTF-8
    LC_MONETARY=en_GB.UTF-8
    LC_NUMERIC=en_GB.UTF-8
    LC_TIME=en_GB.UTF-8
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/eda/bin:/home/eda/bin:/usr/local/bin:/usr/bin:/sbin:/usr/sbin:/sbin:/usr/sbin
    PERL5LIB=/home/eda/lib64/perl5/
    PERL_BADLANG (unset)
    SHELL=/bin/bash

[p5pRT]

From @iabyn

On Mon, Nov 07, 2016 at 03​:46​:12AM -0800, Ed Avis wrote​:

I was wondering, if perl knows internally how far along in @​things we
are, perhaps it could expose that list index to the programmer?

Currently perl doesn't always know the index. For ranges, e.g. (1..10) and
('a'..'z'), it only records the current value and the last value.
Recording the index for these cases would involve adding an extra field to
the loop part of the context union, which might increase the size of the
context struct; also it would require extra bookkeeping work for each loop
iteration.

Suppose that the currently unused $# variable were used for this.

If we did this this, we would have to either​:

* update $# on each loop iteration, which would slow down all loop
  iterations, or
* give $# get magic that searches the context stack for a loop context,
  then retrieves the current loop index stored in there. That wouldn't
  slow down normal loops, but accessing $# itself might be quite slow.

--
My Dad used to say 'always fight fire with fire', which is probably why
he got thrown out of the fire brigade.

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @epa

Dave Mitchell wrote​:

For ranges, e.g. (1..10) and ('a'..'z'), it only records the current value and the last value.

But for numeric ranges, if you know the current value it is just a subtraction to find its index.
For string ranges like 'aa'..'zz' it is trickier but still possible. So fetching $# could do that?

give $# get magic that searches the context stack for a loop context,
then retrieves the current loop index stored in there. That wouldn't
slow down normal loops, but accessing $# itself might be quite slow.

I think it would still be faster than the equivalent handwritten code using an $i variable, though?

My concern would be whether in all cases 'searching the context stack for a loop'
gives exactly the same semantics as we currently have for $_ in nested loops.
Or whether the odd differences are not worth worrying about.

--
Ed Avis <eda@​waniasset.com>

This email is intended only for the person to whom it is addressed and may contain confidential information. Any retransmission, copying, disclosure or other use of, this information by persons other than the intended recipient is prohibited. If you received this email in error, please contact the sender and delete the material. This email is for information only and is not intended as an offer or solicitation for the purchase or sale of any financial instrument. Wadhwani Asset Management LLP is a Limited Liability Partnership registered in England (OC303168) with registered office at 9th Floor Orion House, 5 Upper St Martin’s Lane, London, WC2H 9EA. It is authorised and regulated by the Financial Conduct Authority.

[p5pRT]

From @demerphq

On 8 November 2016 at 11​:35, Ed Avis <eda@​waniasset.com> wrote​:

Dave Mitchell wrote​:

For ranges, e.g. (1..10) and ('a'..'z'), it only records the current value and the last value.

But for numeric ranges, if you know the current value it is just a subtraction to find its index.
For string ranges like 'aa'..'zz' it is trickier but still possible. So fetching $# could do that?

give $# get magic that searches the context stack for a loop context,
then retrieves the current loop index stored in there. That wouldn't
slow down normal loops, but accessing $# itself might be quite slow.

I think it would still be faster than the equivalent handwritten code using an $i variable, though?

My concern would be whether in all cases 'searching the context stack for a loop'
gives exactly the same semantics as we currently have for $_ in nested loops.
Or whether the odd differences are not worth worrying about.

An alternative would be to introduce a new keyword, say "for_with_index".

Then you could say

for_with_index (LIST) {
}

and perl would know it has to do the extra bookkeeping to populate $#.

But personally I would not use $#, i would use something like ${^LOOP_INDEX}.

Yves

[p5pRT]

From @iabyn

On Tue, Nov 08, 2016 at 10​:35​:07AM +0000, Ed Avis wrote​:

Dave Mitchell wrote​:

For ranges, e.g. (1..10) and ('a'..'z'), it only records the current value and the last value.

But for numeric ranges, if you know the current value it is just a subtraction to find its index.

No, at the point where you have iterated 'for (11..20) {}' say, 7 times,
exactly two values are currently stored​: 17 and 20. There is no way (in
general) to deduce that value '7' from that. Extra bookkeeping of one form
or another would need to be added.

--
A walk of a thousand miles begins with a single step...
then continues for another 1,999,999 or so.

[p5pRT]

From @epa

Dave Mitchell wrote​:

No, at the point where you have iterated 'for (11..20) {}' say, 7 times,
exactly two values are currently stored​: 17 and 20.

Ah I see. So when looping over a range, the start of the range would need to be stored
in case of later calls to $#. At least that is only once per loop, not per iteration.

--
Ed Avis <eda@​waniasset.com>

This email is intended only for the person to whom it is addressed and may contain confidential information. Any retransmission, copying, disclosure or other use of, this information by persons other than the intended recipient is prohibited. If you received this email in error, please contact the sender and delete the material. This email is for information only and is not intended as an offer or solicitation for the purchase or sale of any financial instrument. Wadhwani Asset Management LLP is a Limited Liability Partnership registered in England (OC303168) with registered office at 9th Floor Orion House, 5 Upper St Martin’s Lane, London, WC2H 9EA. It is authorised and regulated by the Financial Conduct Authority.

[p5pRT]

From @ap

* Dave Mitchell <davem@​iabyn.com> [2016-11-08 11​:36]​:

If we did this this, we would have to either​:

* update $# on each loop iteration, which would slow down all loop
iterations, or
* give $# get magic that searches the context stack for a loop context,
then retrieves the current loop index stored in there. That wouldn't
slow down normal loops, but accessing $# itself might be quite slow.

The latter would be fine, just as a new builtin `loopindex` instead of
get magic on yet another new superglobal.

How much overhead (size, speed) on scopes would we incur if each frame
had a pointer to the innermost containing loop context? (On each scope
check what type the immediate containing scope is, then either set the
new scope’s containing loop pointer to the containing scope or copy the
containing scope’s loop context pointer.) Could `next`/`last` benefit
from that too?

Regards,
--
Aristotle Pagaltzis // <http​://plasmasturm.org/>

[p5pRT]

From @kentfredric

On 28 November 2016 at 13​:48, Aristotle Pagaltzis <pagaltzis@​gmx.de> wrote​:

* update $# on each loop iteration, which would slow down all loop
iterations, or
* give $# get magic that searches the context stack for a loop context,
then retrieves the current loop index stored in there. That wouldn't
slow down normal loops, but accessing $# itself might be quite slow.

There's a third option, its not well employed, but it could be​:

At parse time, the Perl compiler parsing the for { } block *lexically*
parses for a variable ( we'll just say $# for now ).

And then upon detecting that variable's usage in its block, then instruments
the for loop to behave differently.

This would mean *no* performance impact for existing code at runtime,
and the associations of loops and variables would be compile-time bound.

Though it could be tricky.

for ( ... ) {
  for ( ... ) {
  $#
  }
}

Here $# is only instrumenting its direct parent.

Logically, it would be like imagining there was a builtin called
"for_idx" , which looked as follows​:

for_idx my($#, $value) ( .... ) {

}

And I think that latter form is quite easy to reason about.

You don't have to worry about distant subs secretly poking at $# and
sniffing your index and all those kinds of spooky side-effects.

Also, my memory tells me lexicals are rather fast in comparison to
globals, but my memory could be wrong.

--
Kent

KENTNL - https://metacpan.org/author/KENTNL

[p5pRT]

From @iabyn

On Mon, Nov 28, 2016 at 01​:48​:23AM +0100, Aristotle Pagaltzis wrote​:

How much overhead (size, speed) on scopes would we incur if each frame
had a pointer to the innermost containing loop context? (On each scope
check what type the immediate containing scope is, then either set the
new scope’s containing loop pointer to the containing scope or copy the
containing scope’s loop context pointer.) Could `next`/`last` benefit
from that too?

Well there are two approaches​: the one you describe, or to add a new
interpreter var, PL_last_loop_ctx say, whose old value is stored in the
block_loop struct part of the context union and is set and restored when
entering or returning from a loop context. That would probably be cheaper.

In addition to extra field needed in the loop struct to store or calculate
the iteration value (as dicusssed earlier), the extra old_last_loop_ctx
field would make the struct block_loop the largest field of the context
union, increasing the size of every context union (from about 17 to 18
64-bit words).

pp_next() etc are already optimised to check if the top context is a loop
context and if so, to not go searching for one; and if it isn't, the extra
contexts need to be popped anyway - so PL_last_loop_ctx probably wouldn't
make much difference to next/last's performance.

--
All wight. I will give you one more chance. This time, I want to hear
no Wubens. No Weginalds. No Wudolf the wed-nosed weindeers.
  -- Life of Brian

[p5pRT]

From @epa

Translating the loop at compile time into a new builtin foreach_idx sounds sensible, and it effectively automates what the programmer currently has do to in the code. But if $# is a lexical scoped at compile time to the for loop, it won't be available in subroutines called from the loop. This makes it inconsistent with next/last/redo, which do work from subroutines (matching the enclosing loop at run time). Personally speaking I'd be fine with that (I consider always allowing 'next' from a subroutine something of a wart on the language) but I just mention the asymmetry.

One advantage of a new builtin loopidx rather than $# would be that you could specify loopidx(LABEL). This might be another point in favour of a compile-time translation approach, that it could allow loopidx(LABEL) to be implemented more easily.

[p5pRT]

From @iabyn

On Mon, Nov 28, 2016 at 08​:14​:23PM +1300, Kent Fredric wrote​:

At parse time, the Perl compiler parsing the for { } block *lexically*
parses for a variable ( we'll just say $# for now ).

And then upon detecting that variable's usage in its block, then instruments
the for loop to behave differently.

what about​:

  for (...) {
  eval '$#';
  }

Also, my memory tells me lexicals are rather fast in comparison to
globals, but my memory could be wrong.

Marginally faster. For example '$x = 1' uses about 3% fewer CPU instructions
and 2% fewer data reads when $x is lexical.

--
This is a great day for France!
  -- Nixon at Charles De Gaulle's funeral

[p5pRT]

From @kentfredric

On 28 November 2016 at 21​:53, Dave Mitchell <davem@​iabyn.com> wrote​:

what about​:

for \(\.\.\.\) \{
    eval '$\#';
\}

Yeah. That's the annoyingest part of this idea. I don't know how to
make that seem unweird.

If we take a different tack and propose, instead of some arcane magic,
that we define a new keyword,

for_idx

That can take *two* variables​:

for_idx my $key, $value ( @​array ) {
  ...
}

Then we get the benefit of explicitness, the benefit of clarity, and
the benefit of
working even in evals.

Theoretically, this idea could even be prototyped on CPAN.

If anyone goes in this direction, may I also tempt them with​:

  for_assoc my $a, $b, $c ( @​array ) {

  }

Which would be a "more perlish" way of the sort of idea people are trying to
get over in the signature introspection thread, and it would be really
awesome if
that gave us something like

  for_assoc my $key, $value ( %hash ) {

  }

#justsayin

But as you can see, this is kinda bikeshed territory​: Its clearly the
sort of thing we all hate bumping into all the time as
a feature that perls missing, but its not really that clear how we
should implement it sanely. So CPAN prototypes
would be really nice to prove which strategy is the nicest for Perl.

I'm not in any hurry to shoehorn one of these proposals into Perl wrongly. :)

--
Kent

KENTNL - https://metacpan.org/author/KENTNL

[p5pRT]

From @iabyn

On Mon, Nov 28, 2016 at 08​:53​:34AM +0000, Dave Mitchell wrote​:

On Mon, Nov 28, 2016 at 08​:14​:23PM +1300, Kent Fredric wrote​:

At parse time, the Perl compiler parsing the for { } block *lexically*
parses for a variable ( we'll just say $# for now ).

And then upon detecting that variable's usage in its block, then instruments
the for loop to behave differently.

what about​:

for \(\.\.\.\) \{
    eval '$\#';
\}

Another approach would be to extend the for syntax, e.g.

  for -> $i (....) { say "$i​: $_" }
  for $x -> my $i (....) { say "$i​: $x" }
  for my $x -> my $i (....) { say "$i​: $x" }

for a suitable bikeshedding of '->'.

--
Indomitable in retreat, invincible in advance, insufferable in victory
  -- Churchill on Montgomery

[p5pRT]

From @kentfredric

On 28 November 2016 at 22​:34, Dave Mitchell <davem@​iabyn.com> wrote​:

for       \-> $i    \(\.\.\.\.\) \{ say "$i​: $\_" \}
for $x    \-> my $i \(\.\.\.\.\) \{ say "$i​: $x" \}
for my $x \-> my $i \(\.\.\.\.\) \{ say "$i​: $x" \}

for a suitable bikeshedding of '->'.

Yeah. There's also the possibility of a future of list expanding +
iteration step.

for my $iter -> my $a, $b, $c ( @​items ) {
  say "$iter​: $a, $b, $c";
}

Where "$iter" would only increment once for every three @​items pooled
in via $a,$b,$c

Though I would expect any implementation we choose to have the values
in the order "index/value" or "key/value",
as that's pretty much how every other language with similar features does it.

--
Kent

KENTNL - https://metacpan.org/author/KENTNL