Feature Request: Recursive Iterators (non-quadratic)

[gafter]

@AdamSpeight2008 commented on Fri Jan 16 2015

Current if you want write a recursive based iterator function.

Iterator Function TreeWalk(Of T) ( curr As BinaryNode(Of T)) : IEnumerable(Of BinaryNode(Of T))
  If curr Is Nothing Then Return Enumerable.Empty(Of BinaryNode(Of T))
  ForEach node In TreeWalk( curr.Left )
    Yield node
  Next
  Yield curr
  ForEach node In TreeWalk( curr.Righ )
    Yield node
  Next
End Function

it ends up being Quadratic runtime

If I could express the Iterator / Yielder as a parameter I could linearise the runtime.

Iteration Function TreeWalk( n : BinaryNode, iterator As ?? ) : IEnumerable(Of T)
  If n Is Nothing Then Exit Functon
  If n.Left IsNot Nothing Then TreeWalk(n.Left, Iterator)
  Yield n.Value on Iterator
  If n.Righ IsNot Nothing Then TreeWalk(n.Righ, Iterator)
End Function

---

@ufcpp commented on Fri Jan 16 2015

👍
I'd like this fearure to be added to C# too.

---

@mattwar commented on Fri Jan 16 2015

It's a nice feature, and we had it on the list when we did IEnumerable, or the release after, not entirely sure. But it is kind of niche feature. I think the syntax was going to be something like:

yield foreach x

Erik Meijer had an algorithm to make it near non-recursive.

---

@AdamSpeight2008 commented on Sat Jan 17 2015

@mattwar
Don't confuse

yield foreach recFN with yield x On iter

yield foreach recFN is equivalent to

foreach(x in recFN) 
{
 yield x;
}

Which is still Quadratic.

This does something different. Let's use another example. Yielding the permutations of a list of items.

private iterator _Perm<T>( items : T[], len : int, res : T[] , iter : Iterator<T[]> ) : IE< T[] >
{
  if( items.Length = len )
  {
   Yield res On iter;
  } else
  {
    for(var i = 1 To items.Length)
    {
      _Perm<T>( items.Skip(1).ToArray(), len, res.Concate( items.Take(1) , iter );
      items = items.Rotate();
    }
  }
}


public Rotate<T>( this a : T[] ) : T[]
{
  return a.Skip(1).Concate( a.Take(1) ).ToArray();
}`

Let's create the public method the initiates the iterator and calls the method.

public Perm<T>( this items : T[] , size : int ) : IE< T[] >
{
  if( items == null ) return Empty<T[]>();
  if( items.Length = 0 ) return Empty<T[]>();
  if( size < 1 ) return Empty<T[]>();
  if( size > items.Length ) size = items.Length;
  return _Perm<T>( items , size, {} , Iterator.Create( _Perm<T> ) );
}

Iterator.Create( _Perm<T> ); create the single instance of the iterator / yielder that is use through out the recursive calls.
Maybe an attribute[Iterator(Recusive:= True)] on the function, creates the "linearised" state-machine.

---

@mattwar commented on Sat Jan 17 2015

The yield foreach I was referring to was just a syntax. It did not translate to a foreach instruction and required a very different codegen for the iterator methods.

---

@theoy commented on Tue Jan 20 2015

I think it already has the VB tag, right?

Am I confused about something?

Cheers,
--Theo

From: Adam Speight [mailto:notifications@github.com]
Sent: Tuesday, January 20, 2015 3:21 PM
To: dotnet/roslyn
Cc: Theo Yaung
Subject: Re: [roslyn] Feature Request: Recursive Iterators (non-quadratic) (#15)

@theoyhttps://github.com/theoy Co-Evolution? Why no VB.net tag?

—
Reply to this email directly or view it on GitHubhttps://github.com/dotnet/roslyn/issues/15#issuecomment-70755229.

---

@AdamSpeight2008 commented on Tue Jan 20 2015

I missed the tag, then replied. Then saw the tag, so removed to comment. DOH!

---

@HaloFour commented on Tue Jan 20 2015

Didn't we discuss this in some length on the codeplex forums? If I recall while there was an alternate algorithm which made these scenarios much faster they were also found to be much slower for the common non-nested cases.

---

@weltkante commented on Fri May 01 2015

For reference, the paper about "yield foreach" with non-quadratic runtime (part of the Spec# work)

http://research.microsoft.com/en-us/projects/specsharp/iterators.pdf

---

@AlexRadch commented on Tue Jan 12 2016

I think it may be more comfortable to add yield foreach of IEnumerable as syntax sugar for loop with yield return.

IEnumerable<BinaryNode<T>> TreeWalk<T>(BinaryNode<T> curr)
{
    if (curr == null) yield break;
    yield foreach TreeWalk(curr.Left); // Syntax  sugar for loop with yield return
    yield return curr;
    yield foreach TreeWalk(curr.Rigth); // Syntax  sugar for loop with yield return
}

---

@AlexRadch commented on Tue Jan 12 2016

Here dotnet/roslyn#7630 suggested yield foreach

yield foreach TreeWalk(curr.Left); // Syntax  sugar for loop with yield return

---

@AdamSpeight2008 commented on Tue Jan 12 2016

@AlexRadch
My opinion yield foreach xs wouldn't be a good addition to the language, makes it too easy to implement the quadratic case. What do we gain by having it? 3 lines of into 1.
Whereas linearisation of the recursive iterator, would be a lot more beneficial. It would be more efficient in both runtime and memory. As it wouldn't create an additional iterator for each level of recursion.

---

@AlexRadch commented on Tue Jan 12 2016

What is linearisation of the recursive iterator? What code compiler should create? Store all local variables in arrays or where? Or your linearisation of the recursive iterator should create result array in memory?
Now each yield fuction create object where local variables stored but where should be stored local variables in linearised iterator? I think if you can create linearised code for your iterator you can create linearised code for yield foreach also.
I do not see difference in code yield foreach TreeWalk(curr.Left) and code TreeWalk(n.Left, Iterator).
Can you describe difference?
linearisation of the recursive iterator means not more than perpetuum mobile.

---

@AlexRadch commented on Tue Jan 12 2016

I think you mean that linearisation of the recursive iterator should create List<T> to store result and can be coded as

void TreeWalk<T>(BinaryNode<T> curr, IList<T> magicIterator)
{
    if (curr == null) return;
    TreeWalk(curr.Left, magicIterator);
    magicIterator.Add(curr);
    TreeWalk(curr.Rigth, magicIterator);
}

Am I right?

---

@AdamSpeight2008 commented on Tue Jan 12 2016

No. Suggested reading material ( http://www.dreamincode.net/forums/topic/332455-iterators/ ).

You should now see that each recursive call (use the foreach yield) is generates a new instance of a ienumerable and ienumerator.

It should be possible generate a single instance of ienumerable and ienumerator, that encompasses all of the subsequent recursive method calls.

The compiler should capable of building a state machine for a recursive iterator function. It could internally keep a stack of method calls, where it is in each of those methods.

Non tail-recursive would potentially result in stack overflow when evaluated. Tail-recursive (along with tail call optimisation) could potentially never return. (infinite iteration).

---

@AlexRadch commented on Tue Jan 12 2016

It should be possible generate a single instance of ienumerable and ienumerator, that encompasses **all** of the subsequent recursive method calls. is wrong!
If it is possible to do automatic with compiler for your magic iterator, it means that it is possible do the same for yield foreach also. So yield foreach can be compiled to the same magic single IEnumarable with smart tail-recursion also without creation many IEnumerable!

---

@weltkante commented on Wed Jan 13 2016

You may want to read the Microsoft research paper I linked above (which was also mentioned by mattwar) before doing handwaving and talking of magic IEnumerables. You are making too many assumptions without knowing the already presented facts.

Non-quadratic iteration (and implementation of yield foreach compiler magic) is possible without building a list beforehand, but for an efficient implementation it needs a new interface or some other way to extend the current IEnumerable iteration protocol.

Also keep in mind that the paper only presents one possible solution, I know of at least one other possible implementation which is also non-quadratic, but it also needs a different iteration protocol.

Also note that even if new interfaces are introduced they can be optional and the iteration protocol can gracefully fall back to the classic iteration for enumerators which don't implement the new interface (at the cost of the quadratic iteration we currently have).

---

@AdamSpeight2008 commented on Wed Jan 13 2016

I've read the paper again, it doesn't require a new interface.
It still utilises IEnumerable<T> and IEnumerator<T>
Done a VB implementation

---

@weltkante commented on Wed Jan 13 2016

@AdamSpeight2008 then you didn't understand the code you wrote, nor the paper

    Public Shared Function GetNestedEnumerator(e As IEnumerable(Of T))
        Dim ne = TryCast(e, NestedEnumerable(Of T))
        Return If(ne Is Nothing, New EnumeratorAdapter(Of T)(e), ne.GetNestedEnumerator)
    End Function

That's testing for the new iterator protocol. If its not present then it falls back to quadratic behavior using a simple foreach-loop.

Using a class instead of an interface to test for a feature is just an implementation detail. It's been a year or so since I read the paper so I didn't remember their proposed implementation exactly, but the concept is the same.

If you want your custom enumerables support the new protocol they need to implement an NestedEnumerable<T> instead of an IEnumerable<T>. When using the yield language feature the compiler can do it for you, but existing code needs to be recompiled and explicit implementations of IEnumerable must be manually updated.

---

@AlexRadch commented on Sat Jan 16 2016

I made hyperloop https://github.com/AlexRadch/YieldForEachDN/blob/master/Src/YieldForEachApp/Hyperloop.cs to make recursive yielded loops without quadratic runtime performance.

For example next yielded method have quadratic runtime performance because it have recursion.

        static IEnumerable<int> FromToNestedStandart(int b, int e)
        {
            if (b > e)
                yield break;
            yield return b;
            foreach(var v in FromToNestedStandart(b + 1, e))
                yield return v;
        }

To make it with liner runtime performance you should rewrite it with Hyperloop usage:

        static IEnumerable<int> FromToNestedHyperloopWithTail(int b, int e)
        {
            var hl = new Hyperloop<int>();
            hl.AddLoop(FromToNestedHyperloopWithTailLoop(b, e, hl).GetEnumerator());
            return hl;
        }

        private static IEnumerable<int> FromToNestedHyperloopWithTailLoop(int b, int e, IOldHyperloop<int> hl)
        {
            if (b > e)
                yield break;
            yield return b;
            // yield foreach replaced on hyperloop
            hl.GetHyperloop().AddTail(FromToNestedHyperloopWithTailLoop(b + 1, e, hl).GetEnumerator());
        }

AddLoop() used to replace yield foreach calls in middle recursion for more performance.
AddTail() used to replace yield foreach calls in tail recursion for little more performance than AddLoop().

---

@paulomorgado commented on Sat Jan 16 2016

I wonder if local function definitions could be used by the compiler to generate that linear state machine.

---

@weltkante commented on Sat Jan 16 2016

@paulomorgado As far as I understood it, local functions add no benefit for the compiler because its only a language feature and not an IL feature, the compiler could already generate the IL for local functions if he needed it.

Anyways, the whole reason why you need a state machine in the first place is because the control flow is non-local (you return to the caller between states) so local function definitions are unlikely to help.

---

@AlexRadch commented on Sat Jan 16 2016

To reduce quadratic performance in recursive yielded methods you should create fist loop (I called it hyperloop) and send to that hyper loop workitems without creating local loop in local loop in local loop and so on for each recursive call.

Here hyperloop code https://github.com/AlexRadch/YieldForEachDN/blob/master/Src/YieldForEachApp/Hyperloop.cs

Hyperloop execute recursive work in one loop without layered loops for each recursive call and then return control to continue execution if AddLoop() was used. For tail recursion you can use AddTail() so it does not return control.

Next code create Hyperloop and add first loop to them

        static IEnumerable<int> FromToNestedHyperloopWithTail(int b, int e)
        {
            var hl = new Hyperloop<int>();
            hl.AddLoop(FromToNestedHyperloopWithTailLoop(b, e, hl).GetEnumerator());
            return hl;
        }

Next code is like usual yielded method with recursion but it does not create layered loops for each recursive call but add loops to Hyperloop and Hyperloop execute work without layers (than speedup performance from quadratic to linear)

        static IEnumerable<int> FromToNestedHyperloopWithTail(int b, int e)
        {
            var hl = new Hyperloop<int>();
            hl.AddLoop(FromToNestedHyperloopWithTailLoop(b, e, hl).GetEnumerator());
            return hl;
        }

        private static IEnumerable<int> FromToNestedHyperloopWithTailLoop(int b, int e, IOldHyperloop<int> hl)
        {
            if (b > e)
                yield break;
            yield return b;
            // yield foreach replaced on hyperloop
            hl.GetHyperloop().AddTail(FromToNestedHyperloopWithTailLoop(b + 1, e, hl).GetEnumerator());
        }

---

@AdamSpeight2008 commented on Sat Jan 16 2016

@weltkante @paulomorgado VB has iterator / async lambda functions, they won't help in this situation. As any state information held by them is local to that lambda.

Iterator Function Foo() As IEnumerable(Of int)
  Dim lamba = Iterator Function()
                 Yield 0
                 Yield 1
                 Yield 2
                 ' These "yield" from the lambda not from the enclosing function "Foo" 
              End Function 

`

---

@paulomorgado commented on Mon Jan 18 2016

@weltkante, local functions can be reentrant and do not need a delegate invocation. There's a lot that can be done and done better with local functions then with delegates.

Of course the compiler can generate the IL. It already does.

---

@gafter commented on Mon Mar 20 2017

We are now taking language feature discussion on https://github.com/dotnet/csharplang for C# specific issues, https://github.com/dotnet/vblang for VB-specific features, and https://github.com/dotnet/csharplang for features that affect both languages.

[aluanhaddad]

This sure would be nice.
That said, current solutions involving a stack are ugly but they work.
Its probably orthogonal, but I wonder how this cross cuts with #390.

[MgSam]

This feature would of course be really nice, but when it was originally proposed on CodePlex Lucian specifically addressed Erik Meijer's paper and mentioned that implementing this would make the common case more slow. I'm guessing that's a tradeoff the team is unlikely to make.

[MillKaDe]

I would like to add two use-cases to the discussion.

Most discussions of recursive iterators simply assume that

• all collection items are yield-returned unprocessed.
• the recursive iterator just calls itself.

A simple binary tree:

class Tree {
  Tree L; // left subtree (node values smaller than V)
  int V; // node value
  Tree R; // right subtree (node values larger than V)
}

can be enumerated by this recursive iterator:

IEnumerable<int> Iterator1 (Tree t) {
  if (null == t) return;
  foreach (var vl in Iterator1 (t.L)) yield return vl;
  yield return t.V;
  foreach (var vr in Iterator1 (t.R)) yield return vr;
}

In a future version of C#, this might be shortened to:

IEnumerable<int> Iterator2 (Tree t) {
  if (null == t) return;
  yield foreach Iterator2 (t.L);
  yield return t.V;
  yield foreach Iterator2 (t.R);
}

But what if some processing of all items is required ?

1) Iterators with item processing

For example, this iterator converts each int item to string and appends
a letter indicating the items original position in the tree:

IEnumerable<string> Iterator3 (Tree t) {
  if (null == t) return;
  foreach (var vl in Iterator3 (t.L)) yield return $"{vl}L";
  yield return t.V;
  foreach (var vr in Iterator3 (t.R)) yield return $"{vr}R";
}

which converts the balanced tree: (((1)2(3))4((5)6(7)))
to the string sequence: 1LL,2L,3RL,4,5LR,6R,7RR

The new, shorter syntax needs some way to specify an optional operation for each item,
maybe as a lambda function:

IEnumerable<string> Iterator4 (Tree t) {
  if (null == t) return;
  yield foreach Iterator4 (t.L) vl => $"{vl}L";
  yield return t.V;
  yield foreach Iterator4 (t.R) vr => $"{vr}R";
}

If the lambda function is omitted, x => x is assumed instead.

2) Mutual recursive iterators

Lets add another, completly artificial requirement:
Swap the order from ascending to descending (and back) on each recursion level:

IEnumerable<string> Iterator5a (Tree t) { // ascending (L,V,R)
  if (null == t) return;
  foreach (var vl in Iterator5d (t.L)) yield return $"{vl}L";
  yield return t.V;
  foreach (var vr in Iterator5d (t.R)) yield return $"{vr}R";
}
IEnumerable<string> Iterator5d (Tree t) { // descending (R,V,L)
  if (null == t) return;
  foreach (var vr in Iterator5a (t.R)) yield return $"{vr}R";
  yield return t.V;
  foreach (var vl in Iterator5a (t.L)) yield return $"{vl}L";
}

or shorter:

IEnumerable<string> Iterator6a (Tree t) { // ascending (L,V,R)
  if (null == t) return;
  yield foreach Iterator6d (t.L)) vl => $"{vl}L";
  yield return t.V;
  yield foreach Iterator6d (t.R)) vr => $"{vr}R";
}
IEnumerable<string> Iterator6d (Tree t) { // descending (R,V,L)
  if (null == t) return;
  yield foreach Iterator6a (t.R)) vr => $"{vr}R";
  yield return t.V;
  yield foreach Iterator6a (t.L)) vl => $"{vl}L";
}

Yes, the example is idiotic. Instead assume traversing some arbitrary object graph
or serializing an expression tree or ...

So my questions are:

1. Will / shall the requested feature support item processing ?
2. Will / shall the requested feature support mutual recursion ?

[AlexRadch]

@MillKaDe #378 (comment)

So my questions are:

1. Will / shall the requested feature support **item processing** ?

2. Will / shall the requested feature support **mutual recursion** ?

1. No. It is impossible to linearise recursions with item processing.

2. It is possible if there is no item processing in recursions. Your mutual recursion can not be linearised because have item processing in recursions.

[aluanhaddad]

I think the solution in Eric Lippert's blog post is still simple enough (10 extra lines of code and can easily be generalized for any iterator) is simple enough that it really isn't necessary. It would be nice, but we can already get the optimization today.

https://blogs.msdn.microsoft.com/ericlippert/2007/12/19/immutability-in-c-part-seven-more-on-binary-trees/

[svick]

@MillKaDe I think adding item processing does not make any sense. Your code has a quadratic number of string formatting operations and I don't see how could the compiler fix that. And, if I understand it correctly, the proposed implementation would not support this.

When it comes to mutual recursion (without item processing), I think that should work just fine.

[tuespetre]

Keywords for anyone else who might come along looking for this: yield from in Python

[jhbertra]

@MillKaDe

IEnumerable<string> Iterator4 (Tree t) {
  if (null == t) return;
  yield foreach Iterator4 (t.L) vl => $"{vl}L";
  yield return t.V;
  yield foreach Iterator4 (t.R) vr => $"{vr}R";
}

This could also be accomplished with existing language features + yield foreach:

IEnumerable<string> Iterator4 (Tree t) {
  if (null == t) return;
  yield foreach (from v in Iterator4(t.L) select $"{v}L");
  yield return t.V;
  yield foreach (from v in Iterator4(t.R) select $"{v}R");
}

Or, even better:

IEnumerable<string> Iterator4 (Tree t) {
  if (null == t) return;
  yield foreach Iterator4(t.L).Select(v => $"{v}L");
  yield return t.V;
  yield foreach Iterator4(t.R).Select(v => $"{v}R");
}

If anything, an extension to LINQ query syntax might make more sense than lambdas floating around, something like

IEnumerable<string> Iterator4 (Tree t) {
  if (null == t) return;
  from v in Iterator4(t.L) yield $"{v}L";
  yield return t.V;
  from v in Iterator4(t.R) yield $"{v}R";
}

But I'm not sure even that is really necessary... the existing LINQ tools are more than adequate for manipulating IEnumerable<T>s

[jhbertra]

👍 for this feature!

[AlexRadch]

I created next suggestion for roslyn compiler dotnet/roslyn#25052

[Thaina]

We have syntax for yield return x So I think we could just have syntax yield array and it would not conflict with anything right?

[ErikSchierboom]

I would love to have this!

[chrisdunelm]

Would definitely like this functionality. And I can't imagine it's too difficult to design and implement.

[Brondahl]

There's been no actual discussion of this feature for > 3 years. Just ~20 distinct requests for it.
Is there any indication of whether this is going to get any traction?

Is it still being considered as a useful feature by the languages development team?

[CyrusNajmabadi]

@Brondahl there is no one currently championing this.

[DaveInCaz]

Adding a link to this article: https://docs.microsoft.com/en-us/archive/blogs/wesdyer/all-about-iterators because I don't think it has been mentioned yet in this issue, and it has a good description of how a "yield foreach" might result in performance optimizations for some cases, which I think is what has been discussed in the above comments somewhat.

If instead of the yield foreach, there was the foreach expansion that yielded each result then the FromTo method would suffer from quadratic performance; however, with nested iterators the performance would be linear.

The article was written by Wes Dyer of Microsoft in 2007.

[Rekkonnect]

From here, the proposed syntax could be

yield return foreach x;

which would be directly replaced with

foreach (var x0 in x)
    yield return x0;

[seanblue]

I run into this issue from time to time, and would also like to save some characters with a yield return foreach or similar syntax. The simple example as already discussed goes from

yield return foreach x;

to

foreach (var y in x)
    yield return y;

Now, it's been said that this syntactic change may not be worth it to just save a few characters, given that it might "hide" the fact that it's a heavy operation. However, I think there's a situation that hasn't been discussed yet. The current lack-of-syntax for this can also discourage adding methods from being added, resulting in very large methods when yield return is being used. I recently ran into this use case when writing a validation method using System.ComponentModel.DataAnnotations.IValidatableObject, where there were several conditional validation cases for unrelated data points.

Option 1 (all in one method):

public IEnumerable<ValidationResult> Validate()
{
    if (DataPoint1Condition1)
        yield return new ValidationResult(...);
    if (DataPoint1Condition2)
        yield return new ValidationResult(...);

    if (DataPoint2Condition1)
        yield return new ValidationResult(...);
    if (DataPoint2Condition2)
        yield return new ValidationResult(...);

    if (DataPoint3Condition1)
        yield return new ValidationResult(...);
    if (DataPoint3Condition2)
        yield return new ValidationResult(...);
}

Option 2 (smaller methods, current syntax):

public IEnumerable<ValidationResult> Validate()
{
    foreach (var validationResult in ValidateDataPoint1())
        yield return validationResult;

    foreach (var validationResult in ValidateDataPoint2())
        yield return validationResult;

    foreach (var validationResult in ValidateDataPoint3())
        yield return validationResult;
}

public IEnumerable<ValidationResult> ValidateDataPoint1()
{
    if (DataPoint1Condition1)
        yield return new ValidationResult(...);
    if (DataPoint1Condition2)
        yield return new ValidationResult(...);
}

public IEnumerable<ValidationResult> ValidateDataPoint2()
{
    if (DataPoint2Condition1)
        yield return new ValidationResult(...);
    if (DataPoint2Condition2)
        yield return new ValidationResult(...);
}

public IEnumerable<ValidationResult> ValidateDataPoint3()
{
    if (DataPoint3Condition1)
        yield return new ValidationResult(...);
    if (DataPoint3Condition2)
        yield return new ValidationResult(...);
}

Option 3 (smaller methods, new syntax):

public IEnumerable<ValidationResult> Validate()
{
    yield return foreach ValidateDataPoint1();
    yield return foreach ValidateDataPoint2();
    yield return foreach ValidateDataPoint3();
}

public IEnumerable<ValidationResult> ValidateDataPoint1()
{
    if (DataPoint1Condition1)
        yield return new ValidationResult(...);
    if (DataPoint1Condition2)
        yield return new ValidationResult(...);
}

public IEnumerable<ValidationResult> ValidateDataPoint2()
{
    if (DataPoint2Condition1)
        yield return new ValidationResult(...);
    if (DataPoint2Condition2)
        yield return new ValidationResult(...);
}

public IEnumerable<ValidationResult> ValidateDataPoint3()
{
    if (DataPoint3Condition1)
        yield return new ValidationResult(...);
    if (DataPoint3Condition2)
        yield return new ValidationResult(...);
}

Note that our actual use case is more complicated than this, but I wanted to keep the example on the simpler side, while hopefully still getting my point across.

Right now our only two options were the first two, neither of which were particularly appealing from a code cleanliness perspective. Since we have to validate several data points (again, much more than in this example), we don't like having all of the validation logic in one method. But we also didn't like having several instances of a foreach loop to yield return the contents returned by each method call. So while I get that there is a concern around the "obviousness" of what a yield return foreach syntax would be doing, I think the benefits from a code cleanliness perspective outweigh those concerns. In cases like this, the new syntax isn't just saving a few characters, but encourages other best practices.

[HaloFour]

You can also use LINQ:

public IEnumerable<ValidationResult> Validate()
{
    return ValidateDataPoint1()
        .Concat(ValidateDataPoint2())
        .Concat(ValidateDataPoint3());
}

[seanblue]

Yes, I had considered that option as well. I just didn't want to complicate the example with a fourth option. Even with this, I still think the new proposed syntax is by far the more succinct and understandable at a glance.

[CyrusNajmabadi]

Even with this, I still think the new proposed syntax is by far the more succinct and understandable at a glance.

To help out with future posts on this topic:

The limiting factor in the designs here is not syntactic. In other words, the reason we haven't done this yet is not related to syntax. The reason is that we have no design for how this can be implemented without there being an unpalatable runtime cost for non recursive iteration. We'd need a solution for that in order to be able to really proceed, or we'd need enough confidence that the runtime cost of the proposed solution so far on normal iterators would not be an issue in practice. However, given how massively widespread iterators are, it would currently be very hard to get confidence that there wouldn't be an issue here in practice.

[CyrusNajmabadi]

What is your basis for that claim?

Working on the ldm and being part of the discussions every time this topic has come up.

This is why I laid out the blocking issues from those prior discussions. They are the things that would have to be solved to get any traction here.

[CyrusNajmabadi]

and repeatedly in favour of adding the syntactic-sugar-only change.

I've explained the reasoning for why that is viewed as a negative. Nothing presented so far changes that calculus.

[CyrusNajmabadi]

it would immediately get closed with "Duplicate ...

If that happens, @ me and I'll reopen and explain why the discussions are separated.

[CyrusNajmabadi]

"yeah ... personally, I'd like that syntactic sugar."

That's my point. Reiterating that you want the sugar doesn't change the blocking issues here. So it weighs be best to focus on that as that is the actual viable path to get traction here.

Thanks!

[Brondahl]

I understand your position (don't agree with it :D but I understand it; I think you've articulated it very clearly now) and wasn't disputing the position itself.
I was disputing the assertion that "most people who use C# don't want this sugar".

Asserting "we won't do it even though most people would like it, because {justification}" is entirely reasonable.
But denying that users want it ... is a very different position. :)

[Thaina]

I think the point here is, we just want one line of

foreach(var a in enumerable)
    yield return a;

And I can't see why making yield enumerable directly translate into the above foreach pattern will change quardraticalness or make anything differ from above code. If the above foreach pattern is not quardratic then let it be

And one use case of this to chain a complicate linq in one yield

yield myCollection.Select((item) => item.SomeProperty).Where((prop) => prop.Valid).Select((prop) => {
     return Calculate(prop.Value);
});

It was not convenient to use foreach even we can write a one line of foreach

[CyrusNajmabadi]

@HaloFour It's unavoidable when it just need multiple line for calculation, consider

in that case, you still need multiple lines. but you have multiple lines already. So... there's even less of a need for a single-line form for your needs.

[CyrusNajmabadi]

This code above has no compact syntax with foreach and yield pattern

Sure it does:

foreach (var x in myCollection.Select((item) => item.SomeProperty).Where((prop) => prop.Valid).Select((prop) => {
    return Calculate(prop.Value);
})) yield x;

But, again, solving the case of removing a few characters from a foreach/yield is a non-goal for us. Repeating that you would like it doesn't change that here. We actively do not want such a feature as we feel strongly that it would be misleading and would actually lead to pit-of-fail scenarios.

So, until someone can propose a way to address teh pit-of-fail scenario, we're not going to take a syntactic solution that is nothing more than a terser way of expressing the foreach-statement syntax.

[CyrusNajmabadi]

But this become dirty

This is a non-reason for us. You have chosen stylistically to not do something. But that's a choice that itself has consequences. The language isn't going to design new features just because you're artificially chosen to constrain yourself here. We're going to do the features if:

1. we feel like it's actually a net benefit (not a net negative in this case) syntactically, and
2. it doesn't mislead users and, indeed, limit our options in the future if/when we are able to solve the perf overhead for normal iteration.

[CyrusNajmabadi]

@Thaina It comes down to this:

I think the point here is, we just want one line of

Yes, what you want is understood. However:

1. you can already write these htings on one line, you choose not to. That's an artificial restriction that the language is not enforcing, but you are choosing to encumber yourself with. We generally do not create language features to workaround people choosing to not use the pre-existing features that solve that issue already (but which people do not like stylistically).
2. solutions that allow a terser syntax here have a serious problem in that:
   a. they can potentially cause confusion with people who think that that syntax means they can do non-quadratic recursive iteration. which they can't.
   b. they limit our design space in the future if we can actually provide such a feature. in other words (if we were to ignore 2.a), we don't want a terse syntax here for the bad case as that would mean we wouldn't have that syntax for the good case.

What you want is very understood. However, it is not considered important (because of '1') and it is actively considered harmful currently because of '2'. Reiterating that you want this doesn't change the conversation at all because '1' and '2' still remain true. Until they can be addressed, this issue is at an impasse.

[Thaina]

@HaloFour It's unavoidable when it just need multiple line for calculation, consider

in that case, you still need multiple lines. but you have multiple lines already. So... there's even less of a need for a single-line form for your needs.

Because it was more obvious to make it into pattern that have only one keyword on the front instead of tail the yield at the back and wrap it in foreach. Calling it dirty meant it was cause confusion and make it hard to discern what it really did. While making it as the start of the chain make it more obvious and tidy. Yours alternative solution are not wieldy when it has more line of code and push the yield return x farther out of screen

I would say it comparable to implicit scoped using #114. In the same way that we could assume the dispose will be at the end of scope. We make it into one line pattern but actually we could write it in many line of initialization (include linq and take First), however the benefit is still the same, it was intend to be disposed in the scope and we don't need to wrap everything in parentheses, more benefit for multiple line to not require wrapping in parentheses

As I said I was using word one line to describe it but it not just about making it into one line, but making it into pattern. Sorry that I could not express the intention clearly because I don't know which vocabulary I should use

they can potentially cause confusion

If you think just have foreach in front of collection would make it more obvious then just make it be syntax yield foreach in /* someCollection */

[austinw-fineart]

Correct me if I'm wrong but isn't the non-quadratic part of this request an implementation detail? That should be left for Roslyn to figure out. Such a request could then be generalized to all recursive iterators and not necessarily just for this new syntax.

[CyrusNajmabadi]

No. It would not be an implementation detail. It would have different semantics, and would need to be spec'ed so that other languages could use it as well.

[austinw-fineart]

Was it not implied by the implementors that they could have implemented iterators non-quadratically for all iterators but chose not to for the sake of the common use case? In other words syntax is not the limiting factor here.

[austinw-fineart]

To put it another way, let's say there's an implementation of C# out there where iterators already have non-quadratic performance. You now introduce this new iterator whose specification specifically states it to be non-quadratic. Do you then retroactively go back and say that the vanilla iterators must be quadratic? If so, what purpose does that serve?

[CyrusNajmabadi]

@austinw-fineart i think the point is: No one has a magical way to make this work. We only know of one solution to get non-quadratic iteration. And that solution literally makes normal iteration slower. So we have an unpalatable situation:

1. add the feature to support this, but almost all existing programs get slower and potentially use more memory.
2. do not add the feature. programs continue to work the same way as before, but there's no solution for yield+quadratic iterator. You need to write your own efficient iterator instead.

These are our options. Both aren't great. BUt we're sticking with '2' as we are loath to introduce anything that penalizes people who aren't even using it.

If you have an alternate option, then please lay it out in detail so it can be explored and assessed.

[Brondahl]

@Thaina , @austinw-fineart ...

@CyrusNajmabadi has been very clear that he understands the request, and that he understands that the sugar syntax doesn't change the performance. But that in his view the risk of adding a syntax that could give a misleading impression that they have solved the perf problem, out weighs any possible advantage of the one-line syntactic sugar.

The cost side is so significant in your view that no syntactic benefit could possible outweigh the costs.

This is correct. There is literally no syntactic benefit that would be high enough to justify {cost}.

[Brondahl]

Essentially ... "yes ... anyone could blow off their own foot with an existing gun ... that doesn't mean that you should start making guns that have 'point at foot' written on them ... even if people say they would like it to be easier to point their guns at an opponents feet".

Not a perfect analogy, but I think close enough? @CyrusNajmabadi ?

[Joe4evr]

I think you meant to ping @CyrusNajmabadi, not Jones. 🙃

[Brondahl]

TY; fixed

[znakeeye]

JS dev: "What's up C# guys! What's the yield* equivalent?
C# dev: "Hang on a second. I'll just write this epic for loop..."

F# dev: "What's up C# guys! What's the yield! equivalent?
C# dev: "Hang on a second. I'll just write this epic for loop..."

Seriously, it's just embarrassing.

[CyrusNajmabadi]

@znakeeye we've been clear. The quadratic equivalent is already there. The non-quadratic solution doesn't exist yet because no one has been able to figure out how to provide it without penalizing every existing user of foreach today.

[znakeeye]

You keep talking about quadratics, whereas some of us are just here for the syntactical sugar. There is no way syntactical sugar would automatically penalize existing users.

The quadratic equivalent was there in C# 1.0 too. Back then we also had nifty properties. Guess what happened? The syntax for getters/setters got improved. What ever happened with syntactical improvements for e.g. yield? What, we can't have syntactical improvements anymore? Did we eliminate "syntactical improvements" from the dictionary?

yield! items;
vs
foreach (var item in items) { yield return item; } 🤣

[CyrusNajmabadi]

You keep talking about quadratics, whereas some of us are just here for the syntactical sugar.

This is literally the tracking discussion for non-quadratic iteration. If you want to discuss pure syntax sugar, then the discussion for that is in: #5303

[CyrusNajmabadi]

There is no way syntactical sugar would automatically penalize existing users.

We disagree. We think it would penalize us in terms of future design flexibility.

What ever happened with syntactical improvements for e.g. yield

We don't have a design that is suitable.

yield! items;

This is not an improvement. And it falls afoul of many of the issues already specified, including potentially being highly misleading to users as to the actual behavior this would have and the performance they should expect of it. We will be very unlikely to take such a syntax as it stands a high risk of being excessively misleading which is something we do not want.

[CyrusNajmabadi]

What, we can't have syntactical improvements anymore?

We have added syntactic improvements in every version of the language shipped so far. But in all cases, we felt the net positive was there, and there weren't large negatives coming along with the change as well.

That should be an indication to you why we haven't taken anything in the yield space so far. It's because we both do not see much improvement at all, and we also see it being saddled with severe negatives at the same time. Given that, we would not take such a change as we actually think it would be a net negative for the language both in the near term (for the reasons listed above), and hte long term as it would then limit the design space even further.

[seanblue]

@CyrusNajmabadi I want to circle this conversation back to the main topic/blocker, being the performance issue. I have a couple questions, just so I have a better idea of the likelihood of this feature being added in the near-ish future.

1. Do you (or anyone else you've discussed this with) already have an idea for how to implement non-quadratic recursive iteration, but can't move forward due to time/priorities/complexity/etc.?
2. If not, based on your experience adding features to C#, does this seem unlikely to ever be solved?

[CyrusNajmabadi]

Do you (or anyone else you've discussed this with) already have an idea for how to implement non-quadratic recursive iteration,

Yes. one way to do this would be to have these recursive iterators expose a different interface that can be queried to allow them to directly participate in a callers iteration chain. However, such an interface means that normal iteration would have to query for this and would pay a price even when non-recursive iteration (the common case) was in play. For more details see: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/specsharp-iterators.pdf

Of relevant, see section 4.3 Translation of nested iterators and how it now introduces:

1. extra allocations
2. extra virtual dispatch
3. extra type checking

The issue here is not time/priorities/complexity, it's that we don't have a way currently to support recursive iteration without potentially paying a price for non-recursive iteration.

If not, based on your experience adding features to C#, does this seem unlikely to ever be solved?

It definitely seems solvable. Things like shapes might help out here as they might potentially allow us a way to have thigns like SEnumerable and SRecursiveEnumerable, and we might end up saving enough in the new approach (like no virtual, potentially no allocs) that the cost of the additional typechecks might be ok... but it's very hard to tell as those features are very much in flux.

[Brondahl]

I want to circle this conversation back to the main topic/blocker, being the performance issue.

Can you please open a different discussion for that. This issue really is targeted at "(non-quadratic)" iteration. I'm happy to have that discussed separately.

I've now raised #5503 which addresses the sugar-only ... non-quadratic aspect of the discussion 👍

I imagine that some of the people that voiced opinions on this thread, re the syntactic sugar may wish to summarise / reproduce them over there.

[lbmaian]

Should be #5303

[mburbea]

As a possible dumb solution, why not add a hint via an attribute or contextual keyword to do the special casing for recursive iterators to enable the syntax?

public recursive IEnumerable<T> TreeWalk<T>(BinaryNode<T> root)
{
       if(root is null) yield break;
       yield return foreach TreeWalk(root.Left);
       yield return root.Value;
       yield return foreach TreeWalk(root.Right);
}

Without the recursive keyword, the compiler will generate the equivalent of foreach(x in xs) yield return x;, and in the case of the keyword it could generate something akin to what's mentioned in the research paper? An analyzer could be written to check for misuse of the keyword citing the potential performance downsides.

If a new contextual keyword is unpalatable (as previous discussions have suggested), could it maybe that you can write the method with a different return type? IRecursiveEnumerable<T>. The interface would inherit from IEnumerable<T>.

[Joe4evr]

As a possible dumb solution, why not add a hint via an attribute or contextual keyword to do the special casing for recursive iterators to enable the syntax?

Because the moment someone does

var walk = TreeWalk(root).MaybeSomeLinq();
foreach (var item in walk) { /*.....*/ }

that information is lost and you're back to quadratic evaluation again.

[CyrusNajmabadi]

Sorry, i meant to respond to this yesterday. But yes, @Joe4evr hit the nail on the head. That approach might work, but only for people directly consumign the iterator directly through the method call. in other words: foreach (var item in TreeWalk(root)). However, it would be entirely non-composable. Place the call into a variable, and all we know about it is that it's an IEnumerable<...>. Similarly, use anything else that operates on IEnumerable (like any linq mehtod) and both the linq method, and the conusmer of the resultant IEnumerable will both not only not get the benefit, they'll now observe the awful perf version.

The only suitable way we've foudn that makes this work is to take the approach in the paper i listed above. However, that approach involves both allocations and runtime checks. And you now get that perf issue for all iteration everywhere. It would be a case of us adding a feature, and then literally 100% of all programs having extra overhead even if they never used that feature.

That's a very bitter pill. Imagine you have written apps with tons of iteration in them, and all of a sudden things get slower on upgrade. You didn't do anything, you're not making use of any recursive iteration, but you still have to pay the cost for it everywhere.

We do not like that, and until that thinking changes, it's effectively dead in the water here.

[AartBluestoke]

could we have :

    Interface IRecursiveEnumerable: IEnumerable{
         IRecursiveEnumerator GetRecursiveEnumerator()
    }
    
    Interface IRecursiveEnumerator: public bool MoveNext(IRecursiveEnumerator existingItr){
     ... recursive code can recieve an existing iterator to yield extra items on ...
    }

The compiler can compile the foreach for IRecursiveEnumerable differently to just a recursive enumerable.

unwraping a recursiveEnumerable to an enumerable by enumerating it is fine, as you have already unfolded the quadratic.

It feels like we could setup something where a recursiveEnumerable is convertable to a normal enumerable, and that cast contains the machinery for passing the existing enumerator down through the chain, absorbing the quadratic cost.

downsides

• It would mean that people wanting this behavior would have to opt in deliberately, so quadratic behaviour is still on the books for those who don't expect a trap.
• compiling a foreach would be slightly more expensive as there would be 2 patterns to check for - is this a recursiveEnumerable, is this a normal enumerable
• runtime cost of the conversion from recursive to normal enumerable at whatever boundary the coder ends up (probably the next linq call), but this is still much cheaper than the quadratic cost before, so probably reasonable.

Thoughts?

[AartBluestoke]

• Is it viable to determine whether a method is recursive or not?

Isn't this pretty much the same question as solving the Halting Problem?
@Joe4evr not if we require people wanting a recursive Enumeration to flag those functions themselves.

[lbmaian]

Assuming that the compiler can distinguish between recursive and non-recursive iterators (whether through attribute or interface or whatever), and that yield foreach only works on non-recursive iterators, to preclude the possibility of quadratic runtime, shouldn't iterators be assumed to be recursive by default (and thus ineligible for yield foreach)? That is, marking an iterator as non-recursive needs to be "opt-in" (and thus eligible for yield foreach) rather than "opt-out"?

[AartBluestoke]

what i was suggesting was a way for efficient recursive iteration to be used within yield foreach, be providing for a place where recursive iteration could be explicitly given an efficient implantation.

it is the efficient implementation that is opt-in. you can still write inefficient quadratic iterators, but there is now a path for the implementer of that function (not the consumer) to provide an efficient implementation.

[lbmaian]

Sure, but this statement from Cyrus:

We would not want any feature that gives hte appearance of not being that expensive actually be that expensive. In other words, it's reasonable for people to assume that yield return foreach x; would be efficient and would not have a quadratic perf cliff you could fall off of. Adding it and having that cliff would nto be desirable.

strongly suggests to me that opt-in efficient implementation is insufficient, that yield foreach must not result in quadratic runtime due to recursive iteration. Since iterators currently have no restriction on recursion, doesn't that mean the compile-time indicator for non-recursive iteration for yield foreach eligibility must be opt-in?

[Brondahl]

Is it viable to determine whether a method is recursive or not?

Isn't this pretty much the same question as solving the Halting Problem?

not if we require people wanting a recursive Enumeration to flag those functions themselves.

Perhaps I've mis-understood / mis-expressed something.

I think the exact question we're considering is "does this method (which uses the return yield operator) call itself, either directly or transitively via other methods?"

In which case, it's not the Halting Problem, its just "is there a cyclic loop in this finite graph, passing through this point".

• A codebase is a finite collection of methods (or props, or anonymous functions, etc. etc.).
• Each method is a node in a directed graph where the edges represent "this method calls that method".
• A recursive method is one which ultimately calls itself again, which represents a cycle in the graph.

I would be a bit surpised if the compiler doesn't already construct that graph, and I assume there are efficient algorithms for finding cyclic loops in graphs already exist. (But even if not, in practice, I suspect brute forcing it would be unpleasant, but not actually that computationally expensive, since the graphs aren't going to be that large or have that many connections from any given node.)

[AlexRadch]

I created the DrNet.Hyperloop library https://github.com/AlexRadch/DrNet.Hyperloop to rewrite recursive iterators to non-quadratic recursive iterators with minimal code refactoring.

The library has benchmarks to show that quadratic complexity was decreased to linear complexity. Also, those benchmarks are examples of code refactoring.

I am planning to create a source generator with such automatic code refactoring.

If you have any suggestions or comments on the library, please write about it. I will also be appreciated in any help to create the documentation in good English.

[AartBluestoke]

this type of thing was what i was thinking of above - wrap the potentially quadratic recursive code in an correct handler, presenting this linearly enumerable data source to any downstream consumers. It requires anyone wanting to work around quadratic behaviour to "opt in" to the better behaviour. It should be reasonably easy to make a diagnostic for directly recursive programs suggesting that IEnumerable f(){ ... f();} should instead be a IRecursiveEnumerable.

[AlexRadch]

I created the DrNet.Hyperloop NuGet package https://www.nuget.org/packages/DrNet.Hyperloop/.

[oliver-unifii]

Is this on a roadmap ? recursive yields are very common when processing tree structures such (e.g. Json)

Both JS and F# seem to have a good solution.

[hughesjs]

Would this proposal capture yield return all anotherEnumerable behaviour rather than just specifically recursive enumerables?

Similar to what's being discussed here: https://stackoverflow.com/questions/78438765/multi-yield-mechanism-in-c-sharp?noredirect=1#comment138285851_78438765

[hez2010]

With .. operator being introduced in C#, how about:

yield return ..list;

[HaloFour]

.. outside of collection literals is for ranges, not spreads. Either way, the syntax was never the issue with this feature, it was the behavior.

[hez2010]

list is a collection, not an integer nor an index, so it can be interpreted as a spread operator, not a range operator.

[HaloFour]

See: #8053 (reply in thread)

Either way, the syntax isn't the problem with this proposal.