Feature Request: Syntactic Sugar to yield the whole of an IEnumerable directly.

[Brondahl]

See also #378 . This is separating out a syntactic sugar request, from a functional performance enhancement request.
This is NOT a duplicate!
Please consult @CyrusNajmabadi before Closing-As-Duplicate. :)

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

If you are writing a yield Iterator, and you end up with an IEnumerable<T> that represents some sub-set of the values you wish to return, then you are obliged to do this:

foreach (var x in someValuesToBeYielded) { yield return x; }

I (and many others) would like there to be a syntactic sugar to express that (even) more tersely:

return yield foreach someValuesToBeYielded;

This would be solely syntactic sugar - it's literally just a short hand for the explicit foreach version.
(Implementation wise ... we're talking about a Lowering Rewriter)
There would be no changes to performance in any sense.

To support my assertion that this is a syntactic sugar that is desired by C# users:

• There is a heavily viewed and liked SO question asking "how do you do this?". (Answer: you can't.)
• The syntax exists in Python, Javascript, and F#.
• The related-but-subtly-different request of "write a syntax for in recursive calls that fixes a performance issue with the long-hand version" is littered with people coming along and saying "but all I want is the syntactic sugar ... I don't care about improving the performance".

The possibility of this syntax as a pure sugar syntax came up in #378 , which relates to a very similar concept, but concerns an edge case involving recursive calls, in which the the long-hand code about causes quadratic overhead.
Over in that discussion, the argument presented against this syntax (primarily lead by @CyrusNajmabadi) claims that it is at risk of looking like it's doing something clever about performance, when it's not. i.e. that by including the sugar syntax, which is not confusing in the majority of cases, it will make it look like the recursive edge case has in some way been "fixed" so as to not have the quadratic performance cost.
(I think. ... Note that I don't fully understand this aspect ... @CyrusNajmabadi perhaps you could summarise better?)

The quadratic performance is not itself a concern of this discussion (since we're pitching this a sugar syntax, not claiming to change the performance). But the concern that the syntax could mislead some users does seem a legitimate concern.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

(Note: the exact nature of the keyword is not important for this stage of the discussion. I happen to have proposed yield foreach , but it could be yield from, yield all yield many, yield*, yield! ... maybe even just yield with type analysis identifying that it's yielding an IEnumerable in a method that is declared to yield Ts. The particular keyword isn't important for now - this discussion is to establish whether such a syntax is appropriate/desirable.)

[TonyValenti]

I really hope we see this at some point. It has been asked for so.many.times.

[CyrusNajmabadi]

@TonyValenti the above concerns would need to be addressed in order for any movement to be made on this. Right now there is already a viable solution in the language to syntactically yield an entire enumerable. Given that, there will be practically no chance of a terser syntax being taken given the negatives outlined above.

[Korporal]

I've wondered about this too sometimes but I also now wonder if LINQ's SelectMany is sufficient for this?

[CyrusNajmabadi]

@Korporal can you expand on this? How would you use SelectMany to help out here? Thanks!

[foxesknow]

I don't see why we need this feature just to save writing an foreach loop to accomplish the same thing. This sort of pattern is not so common that it needs explicit language support, and adding it is more of a box ticking exercise more than anything else.

[Brondahl]

adding it is more of a box ticking exercise more than anything else

Yeah, definitely a box-ticking exercise and not, say, a response to:

• dozens of users actively requesting it on the other GH issue
• nearly 200 SO Likes on the question asking how to do it
• nearly 50 thousand views on the same SO question.
  /s

To assert that it's a bad idea, or that you personally don't need it, is perfectly reasonable.
To assert that there's been no desire for it expressed by the user-base is utterly absurd.

[sab39]

but concerns an edge case involving recursive calls, in which the the long-hand code about causes quadratic overhead.

I think the miscommunication here is in calling this an "edge case", for two reasons:

1. It doesn't actually only apply to recursive calls - the nasty performance behavior happens every time you use the longhand code, it just compounds the more nested functions are involved, and recursion is the most common way you get lots of nested functions.
2. Recursive code isn't an edge case, it's an incredibly common type of coding for anything that involves hierarchical data structures (ie Json, Xml/Html, any kind of source code, file systems, ...). This feature, if it existed, would be very widely used (that's the whole point of the "there's huge demand for it" argument), and a lot of that usage would surely be in recursive code.

In other words - the current longhand version of this is already a problem and you need to be careful about how you use it. Until that gets fixed, the syntax being inconvenient is a feature because it discourages use of the pattern and encourages looking for other approaches.

The argument isn't "adding syntactic sugar would make it seem like the issue has been fixed", it's "adding syntactic sugar would encourage people to write this kind of code, which they generally shouldn't be doing until the compiler can produce non-awful code for it".

[CyrusNajmabadi]

In any way, someone who's proficient enough to use yielding will certainly know performance drawbacks of that.

I don't believe that's going to be true. Even today, we run into many people who run into this pitfall, even with the explicit foreach. Making it easier to fall into is not something i'm in support of. As mentioned above:

In any way, someone who's proficient enough to use yielding will certainly know performance drawbacks of that.

[CyrusNajmabadi]

P.S. It's just ridiculous to hear this arguments of performance, when I use this yield* from TS on a daily basis,

Sure. The performance hit may be totally fine for you (And some subset of our audience). But we have to consider the entirety of our audience, and we cannot haphazardly impact performance for everyone just because some are ok with it.

And it do not stops neither devs of TS

Right... but these are different communities with different priorities. What is fine for one community may be fine for another. For example, many things taht C# has done for our community would not be right for C++ or Rust. Or, if they wanted to do something similar, they would need to do so in a way that didn't go against the goals of those languages.

it's just an excuse.

Yes. It's the reason and explanation for why this hasn't been able to move forward so far :)

All that we, the language users, should just take into account, is that the language team do things that make interest to them, not to us, the actual users.

We have millions of "actual users". We have thousands of requests for language improvements, and we have to consider the pros/cons for all users. Given limited time and resources, taht necessarily means making judicious choices as to where to allocate those resources. That means things that have problems, or do not seem to provide enough value will not make the cut.

[GrafGenerator]

we have to consider the entirety of our audience
What is fine for one community may be fine for another
making judicious choices

I understand the tradeoff, indeed:) Just to be clear, I don't want to be flaming here, I just wanted a clear and honest reason why:)

many people who run into this pitfall, even with the explicit foreach

Well you can't build a completely safe-to-use language, I'm pretty sure you understand it as well

Just to be on the same page, we're talking here about a pure syntactic sugar which will replace kind of "yield foreach" into "foreach (var x in y) yield return x", and this adds no performance overhead, however it can be more obscure for those who don't go even one step under the cover of the matter. And main point of language team is that potential value of this is just too small given the limited time and more priority features, is that correct?

If that is correct, then would the following be correct as well - if someone will make a PR with this syntactic sugar implemented, the language team will be ready to consider it for release?

[CyrusNajmabadi]

I just wanted a clear and honest reason why:)

I was being honest. :)

Well you can't build a completely safe-to-use language, I'm pretty sure you understand it as well

Yes. but we can also avoid lang features taht actively lead people astray. :)

if someone will make a PR with this syntactic sugar implemented, the language team will be ready to consider it for release?

Absolutely not. See https://github.com/dotnet/csharplang/#milestones. The proposal would need to at least be in the "any time" category for us to consider the PR for release.

For this issue, we would need an actual proposal that we signed off on that we were confident was the right lang change and would be the right thing for the community/ecosystem.

[CyrusNajmabadi]

Just to be on the same page, we're talking here about a pure syntactic sugar which will replace kind of "yield foreach" into "foreach (var x in y) yield return x", and this adds no performance overhead, however it can be more obscure for those who don't go even one step under the cover of the matter. And main point of language team is that potential value of this is just too small given the limited time and more priority features, is that correct?

Yes. Though i would also add: having a simpler way of expressing the exact same thing, with teh same perf overhead, but which looks like it would not have such overhead is something we feel is actively harmful in that we expect that people will see this and now think they can write these iterators without the high computation complexity, which would not be true. We would be open to a nice syntax form if it could both provide recursive iterators without that computation complexity explosion, and without overhead for users whose iterators do not make use of this functionality.

In other words, this should be "pay to play". You should only have a perf hit here if your iterator makes use of this capability. Existing code that does not should not be affected at all.

[sean-bloch]

Hello, I came here from the "I have done a search for similar discussions". I came about looking for a simpler syntax such as yield return each ... , when I encountered code that was hiding implementation details where the IEnumerable may have been a List, and further processing treated my result differently based on the underlying type. In my case, the "simpler" syntax feels like it would make it clearer that it was "all" i was attempting with the code I was writing...

The discussion I almost started until I found the existing ones is below. I understand now from reading that there may be some performance cases, but also sharing while I would use even just the "longer" syntax in a common case:

In short, this would allow shortcutting:

foreach(var i in items)
{
yield return i;
}

with simply:

yield return each items;

likewise, "items" could just as easily be a method result like:

yield return each GetItems();

It's small, but I ran into a situation with an ORM and abstractions of IEnumerables, where I wanted to only treat the IEnumerable as an IEnumerable. I found that a simple yield return helped to "guarantee" this.

The below shows a small sample of how an "external api" might hide some implementation behavior (outputs to console)

IEnumerable<SomeData> YieldReturnEach(IEnumerable<SomeData> items)
{
    // this below would be shortcut simply to:
    // yield return each items;
    foreach (var i in items)
    {
        yield return i;
    }    
}

void Main()
{
    var xApi = new ExternalAPIClass();
   
    //Note that the api surface hides that this IEnumerable is *also* a List...
    IEnumerable<SomeData> fromDataList  /**/ = xApi.aList().GetSomeEnumerableData();
    //The api surface returns this IEnumerable from a source that is *not* a List... 
    IEnumerable<SomeData> fromKeyedData /**/ = xApi.anEnumerable().GetSomeEnumerableData();
    
    

    Console.WriteLine($$"""
    {{nameof(fromDataList)}}
    ===
    One:               {{xApi.APITransformOne(fromDataList)}}
    
    // Despite being visible as an IEnumerable<> from the API surface.. it is treated as a List... 
    Two:               {{xApi.APITransformTwo(fromDataList)}}
    Two(AsEnumerable): {{xApi.APITransformTwo(fromDataList.AsEnumerable())}}
    Two(Cast):         {{xApi.APITransformTwo((IEnumerable<SomeData>)fromDataList)}} 
    Two(FuncResult):   {{xApi.APITransformTwo(((Func<IEnumerable<SomeData>>)(() => { return fromDataList.AsEnumerable(); })).Invoke())}}
    
    // by "wrapping" in a yield return IEnumerable, we can "guarantee" that our IEnumerable<> is treated *only* as an IEnumerable... 
    Two(ReturnEach):   {{xApi.APITransformTwo(YieldReturnEach(fromDataList))}}
    
    {{nameof(fromKeyedData)}}
    ===
    One: {{xApi.APITransformOne(fromKeyedData)}}
    Two: {{xApi.APITransformTwo(fromKeyedData)}}
    """
    );
    
}

public sealed class ExternalAPIClass()
{
    private string doOne(IEnumerable<SomeData> items) /**/ => $"1[ {string.Join(",", items.Select(i => i.d.ToString()))} ]";
    private string doOne(List<SomeData> items)        /**/ => $"1( {string.Join(",", items.Select(i => i.d.ToString()))} )";
    public string APITransformOne(IEnumerable<SomeData> items) => doOne(items);

    //This highlights a potential "hidden implementation problem"
    public string APITransformTwo(IEnumerable<SomeData> items) => items switch
    {
        List<SomeData> /**/=> $"2( {string.Join(",", items.Select(i => i.d.ToString()))} )",
        _              /**/=> $"2[ {string.Join(",", items.Select(i => i.d.ToString()))} ]",
    };
    
    List<SomeData> m_someDataList               /**/ = new([new(1),new(2)]);
    Dictionary<string,SomeData> m_someKeyedData /**/ = new([new("three",new(3)),new("four",new(4))]);
    public IAdherent aList()        => new Adherent_List(m_someDataList);
    public IAdherent anEnumerable() => new Adherent_Enumerable(m_someKeyedData);
}

public interface Id
{
    public int d { get; }
}

public record SomeData(int d) : Id; 

public interface IAdherent
{
    public IEnumerable<SomeData> GetSomeEnumerableData();
}

public class Adherent_List(List<SomeData> someDataList) : IAdherent
{
    public IEnumerable<SomeData> GetSomeEnumerableData() => someDataList;
}

public class Adherent_Enumerable(Dictionary<string,SomeData> someKeyedData) : IAdherent
{
    public IEnumerable<SomeData> GetSomeEnumerableData() => someKeyedData.Select(kv => kv.Value);
}

Expected Output:

fromDataList
===
One:               1[ 1,2 ]
Two:               2( 1,2 )
Two(AsEnumerable): 2( 1,2 )
Two(Cast):         2( 1,2 ) 
Two(FuncResult):   2( 1,2 )

// by "wrapping" in a yield return IEnumerable, we can "guarantee" that our IEnumerable<> is treated *only* as an IEnumerable... 
Two(ReturnEach):   2[ 1,2 ]

fromKeyedData
===
One: 1[ 3,4 ]
Two: 2[ 3,4 ]

Thanks,

[mrwensveen]

I would also like this feature, but I understand the reasons for not implementing this yet. However, I'd like to note that the performance problems with the current syntax is either undocumented, or very hard to find. This makes writing foreach (var x in someValuesToBeYielded) { yield return x; } less a "I know about the performance problems but I need to do this anyway" and more a "Ugh, why is it so ugly" kind of thing.

If and when syntax is implemented, I'd like yield foreach to be the syntax, because of the already existing await foreach syntax.

[TonyValenti]

I think that the performance problems with the current methodology are exactly a strong reason this feature should be implemented.
yield all items; would certainly be no worse performing that the current foreach loop people write and, if at some point down the road, the enumerable implementations are optimized, then things are future ready.