Ensure the selector gets run during Count. #14435

jamesqo · 2016-12-11T20:47:23Z

Select does not change the count of an enumerable, so previously we made an optimization where if Count() was called we would bypass running the selector altogether and iterate directly through the source. This commit undoes that and makes sure we always run the selector if onlyIfCheap is false.

Fixes #13910

cc @JonHanna, @stephentoub, @VSadov

jamesqo · 2016-12-11T20:56:11Z

src/System.Linq/src/System/Linq/Select.cs

+                if (!onlyIfCheap)
+                {
+                    int end = _minIndexInclusive + count;
+                    for (int i = _minIndexInclusive; i != end; ++i)


An issue came up here that I wasn't sure best how to approach:

lazyEnumerable.Select(i => i).Skip(1).Count() runs the selector lazyEnumerable.Count() times, because Select.Skip on a lazy enumerable isn't specially recognized and the selector gets run on the first item.

list.Select(i => i).Skip(1).Count() is specially recognized, however, and it returns a SelectListPartitionIterator which does not run the selector on the first item.

One way to fix this would be to start from 0 instead of _minIndexInclusive here. However, if we do that, we break Skip(1).Select(i => i) which also ends up here; patterns like those should definitely not run the selector on the first item.

Ideally, we would somehow have a way to differentiate if Skip or Select was called first from within the iterator, and start from _minIndexInclusive or 0 accordingly. But then we might need to add an extra field...

cc @JonHanna

I'm inclined to think that we don't care.

A scenario that was called out as important is someone calling Count() on a Select result specifically to trigger side effects in selectors. (Not a sound practice IMO, but that's another matter). Such a use would be stymied by optimisations that skipped the selectors, and so we avoid such optimisation.

A user who skips something has indicated indifference to that thing. As such I'm inclined to think it doesn't matter whether we run n or n-1 selectors. Indeed, I'm happy running 0 in this case and just calculating what the result of Count() would be.

Others may not be as willing to go with quite so observable a difference to .Net4.6 Framework behaviour though. TBH if this was my PR I'd be taking the fastest route but prepared to back down if I failed to convince on that point.

jamesqo · 2016-12-11T20:56:45Z

src/System.Linq/tests/SelectTests.cs

+            Assert.Equal(source.Count(), timesRun);
+        }
+
+        // [Theory]


Disabled currently because the first assert is giving inconsistent results. See comment above

JonHanna · 2016-12-11T20:54:04Z

src/System.Linq/src/System/Linq/Select.cs

@@ -226,6 +255,14 @@ public List<TResult> ToList()

            public int GetCount(bool onlyIfCheap)
            {
+                if (!onlyIfCheap)


The reason for this is obscure and would likely benefit from being commented on. Without context this looks like pointless busy work that should be deleted to improve efficiency.

JonHanna · 2016-12-11T20:56:15Z

src/System.Linq/src/System/Linq/Select.cs

-                return _source.Count;
+                int count = _source.Count;
+
+                if (!onlyIfCheap)


Likewise, your reason for doing this isn't obvious from the code alone, so should be commented on. And likely elsewhere, so I won't call out other cases.

JonHanna · 2016-12-11T23:41:16Z

src/System.Linq/src/System/Linq/Select.cs

+                    {
+                        _selector(item);
+                    }
+                }


IIRC, if we just returned -1 in this case then the calling Count() method would do pretty much the above. I would imagine this would be slightly faster, but only slightly (that is just a guess though). Do we need the extra code here?

@JonHanna During #12703 when I had optimized Where.Select and the issue of running these selectors had come up, I had originally had a EnumerableHelpers.Count function that Enumerable.Count would call after checking for Linq interfaces (just like what ToArray does today). This was the code I had written for the iterators

// Leave it to Count to iterate through us public int GetCount(bool onlyIfCheap) => onlyIfCheap ? -1 : EnumerableHelpers.Count(this);

However, @stephentoub argued against this. See here for context: #12703 (comment) I ended up writing everything inline for GetCount in those iterators. So I just employed the same strategy here.

I would imagine this would be slightly faster, but only slightly

Virtual method calls are pretty expensive; going from 2 -> 3 virtual method calls (MoveNext & Current to MoveNext, Current & MoveNext) should probably be more than half of a 33% difference. I haven't measured either, but I'm not sure if it would be wise to regress perf here regardless.

Fair enough.

JonHanna · 2016-12-11T23:42:03Z

src/System.Linq/src/System/Linq/Select.cs

+                    for (int i = 0; i < count; i++)
+                    {
+                        _selector(_source[i]);
+                    }


As above, could we just return -1 here and let the caller do this?

JonHanna · 2016-12-11T23:42:56Z

I'm passing the buck on the matter you raised, but aside from that, LG2M.

JonHanna · 2016-12-11T23:43:05Z

Test Innerloop CentOS7.1 Release Build and Test

karelz · 2016-12-12T16:07:53Z

Nit: Please change title, this one won't be useful in git history ...

VSadov · 2016-12-27T23:25:15Z

@jamesqo @JonHanna - I think we need an issue to discuss whether Skip is allowed to skip selectors.

To me it is not obvious that Select(s).Skip(10) would be ok to run the same as Skip(10).Select(s)

VSadov · 2016-12-27T23:25:27Z

The change itself - LGTM

VSadov · 2016-12-27T23:32:52Z

Enetered https://github.com/dotnet/corefx/issues/14729 to follow up with the Skip, or more precisely the behavior of Partition.

crokusek · 2017-05-26T02:13:54Z

Please consider weighing on the "side issue" if you think it is a good question.

https://stackoverflow.com/questions/32000607/is-it-bad-practice-to-purposely-rely-on-linq-side-effects

crozone · 2021-05-18T05:17:37Z

Is there any chance of these changes to be rolled back in the future? I came across this PR while researching LINQ performance and it seems a shame that these opimisations still cannot be enabled because of bad code that relies on .Select() side effects.

Is there scope to add an additional LINQ API to force enumeration to give codebases that rely on this behaviour an easy out, while the general case can benefit?

Alternatively, is there any scope for these optimisations to be enabled if LINQ is used with static anonymous functions that are less likely to have side effects?

karelz · 2021-05-18T11:29:34Z

@crozone We do not monitor closed PRs/bug, esp. in dotnet/corefx repo which is now used only for servicing. I would recommend to open a question on dotnet/runtime repo and link back to this PR.

Ensure the selector gets run during Count. Commit migrated from dotnet/corefx@b879dc0

jamesqo added 2 commits December 11, 2016 15:43

Run the selector during Select.Count chains.

96a84cc

Add tests for the new changes.

6787c11

dnfclas added the cla-already-signed label Dec 11, 2016

jamesqo commented Dec 11, 2016

View reviewed changes

JonHanna suggested changes Dec 11, 2016

View reviewed changes

Add clarifying comments.

5d629d1

jamesqo force-pushed the select-count branch from ca5307f to 5d629d1 Compare December 11, 2016 21:26

JonHanna reviewed Dec 11, 2016

View reviewed changes

karelz added the area-System.Linq label Dec 12, 2016

karelz assigned jamesqo, VSadov and OmarTawfik Dec 12, 2016

jamesqo changed the title ~~Fix #13910 and add tests to ensure the selector gets run during Count.~~ Ensure the selector gets run during Count. Dec 19, 2016

VSadov merged commit b879dc0 into dotnet:master Dec 27, 2016

jamesqo deleted the select-count branch December 28, 2016 01:28

karelz modified the milestone: 2.0.0 Jan 3, 2017

picenka21 pushed a commit to picenka21/runtime that referenced this pull request Feb 18, 2022

Merge pull request dotnet/corefx#14435 from jamesqo/select-count

afddcaf

Ensure the selector gets run during Count. Commit migrated from dotnet/corefx@b879dc0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ensure the selector gets run during Count. #14435

Ensure the selector gets run during Count. #14435

jamesqo commented Dec 11, 2016

jamesqo Dec 11, 2016 •

edited

Loading

JonHanna Dec 11, 2016

jamesqo Dec 11, 2016

JonHanna Dec 11, 2016

JonHanna Dec 11, 2016

JonHanna Dec 11, 2016

jamesqo Dec 12, 2016 •

edited

Loading

JonHanna Dec 12, 2016

JonHanna Dec 11, 2016

JonHanna commented Dec 11, 2016

JonHanna commented Dec 11, 2016

karelz commented Dec 12, 2016

VSadov commented Dec 27, 2016

VSadov commented Dec 27, 2016

VSadov commented Dec 27, 2016

crokusek commented May 26, 2017

crozone commented May 18, 2021

karelz commented May 18, 2021

Ensure the selector gets run during Count. #14435

Ensure the selector gets run during Count. #14435

Conversation

jamesqo commented Dec 11, 2016

jamesqo Dec 11, 2016 • edited Loading

Choose a reason for hiding this comment

JonHanna Dec 11, 2016

Choose a reason for hiding this comment

jamesqo Dec 11, 2016

Choose a reason for hiding this comment

JonHanna Dec 11, 2016

Choose a reason for hiding this comment

JonHanna Dec 11, 2016

Choose a reason for hiding this comment

JonHanna Dec 11, 2016

Choose a reason for hiding this comment

jamesqo Dec 12, 2016 • edited Loading

Choose a reason for hiding this comment

JonHanna Dec 12, 2016

Choose a reason for hiding this comment

JonHanna Dec 11, 2016

Choose a reason for hiding this comment

JonHanna commented Dec 11, 2016

JonHanna commented Dec 11, 2016

karelz commented Dec 12, 2016

VSadov commented Dec 27, 2016

VSadov commented Dec 27, 2016

VSadov commented Dec 27, 2016

crokusek commented May 26, 2017

crozone commented May 18, 2021

karelz commented May 18, 2021

jamesqo Dec 11, 2016 •

edited

Loading

jamesqo Dec 12, 2016 •

edited

Loading