Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Collection<T> and ObservableCollection<T> do not support ranges #10752

Open
robertmclaws opened this Issue Aug 13, 2016 · 150 comments

Comments

@robertmclaws
Copy link

robertmclaws commented Aug 13, 2016

Update 10/04/2018

@ianhays and I discussed this and we agree to add this 6 APIs for now:

    // Adds a range to the end of the collection.
    // Raises CollectionChanged (NotifyCollectionChangedAction.Add)
    public void AddRange(IEnumerable<T> collection) => InsertItemsRange(0, collection);

    // Inserts a range
    // Raises CollectionChanged (NotifyCollectionChangedAction.Add)
    public void InsertRange(int index, IEnumerable<T> collection) => InsertItemsRange(index, collection);

    // Removes a range.
    // Raises CollectionChanged (NotifyCollectionChangedAction.Remove)
    public void RemoveRange(int index, int count) => RemoveItemsRange(index, count);

    // Will allow to replace a range with fewer, equal, or more items.
    // Raises CollectionChanged (NotifyCollectionChangedAction.Replace)
    public void ReplaceRange(int index, int count, IEnumerable<T> collection)
    {
         RemoveItemsRange(index, count);
         InsertItemsRange(index, collection);
    }

    #region virtual methods
    protected virtual void InsertItemsRange(int index, IEnumerable<T> collection);
    protected virtual void RemoveItemsRange(int index, int count);
    #endregion

As those are the most commonly used across collection types and the Predicate ones can be achieved through Linq and seem like edge cases.

To answer @terrajobst questions:

Should the methods be virtual? If no, why not? If yes, how does eventing work and how do derived types work?

Yes, we would like to introduce 2 protected virtual methods to stick with the current pattern that we follow with other Insert/Remove apis to give people hability to add their custom removals (like filtering items on a certain condition).

Should some of these methods be pushed down to Collection?

Yes, and then ObservableCollection could just call the base implementation and then trigger the necessary events.

Let's keep the final speclet at the top for easier search

Speclet (Updated 9/23/2016)

Scope

Modernize Collection<T> and ObservableCollection<T> by allowing them to handle operations against multiple items simultaneously.

Rationale

The ObservableCollection is a critical collection when it comes to XAML-based development, though it can also be useful when building API client libraries as well. Because it implements INotifyPropertyChanged and INotifyCollectionChanged, nearly every XAML app in existence uses some form of this collection to bind a set of objects against UI.

However, this class has some shortcomings. Namely, it cannot currently handle adding or removing multiple objects in a single call. Because of that, it also cannot manipulate the collection in such a way that the PropertyChanged events are raised at the very end of the operation.

Consider the following situation:

  • You have a XAML app that accesses an API.
  • That API call returns 25 objects that need to be bound to the UI.
  • In order to get the data displayed into the UI, you likely have to cycle through the results, and add them one at a time to the ObservableCollection.
  • This has the side-effect of firing the CollectionChanged event 25 times. If you are also using that event to do other processing on incoming items, then those events are firing 25 times too. This can get very expensive, very quickly.
  • Additionally, that event will have ChangedItems Lists that will only ever have 0 or 1 objects in them. That is... not ideal.

This behavior is unnecessary, especially considering that NotifyCollectionChangedEventArgs already has the components necessary to handle firing the event once for multiple items, but that capability is presently not being used at all.

Implementing this properly would allow for better performance in these types of apps, and would negate the need for the plethora of replacements out there (here, here, and here, for example).

Usage

Given the above scenario as an example, usage would look like this pseudocode:

    var observable = new ObservableCollection<SomeObject>();
    var client = new HttpClient();
    var result = client.GetStringAsync("http://someapi.com/someobject");
    var results = JsonConvert.DeserializeObject<SomeObject>(result);
    observable.AddRange(results);

Implementation

This is not the complete implementation, because other *Range functionality would need to be implemented as well. You can see the start of this work in PR #10751

    // Adds a range to the end of the collection.
    // Raises CollectionChanged (NotifyCollectionChangedAction.Add)
    public void AddRange(IEnumerable<T> collection)

    // Inserts a range
    // Raises CollectionChanged (NotifyCollectionChangedAction.Add)
    public void InsertRange(int index, IEnumerable<T> collection);

    // Removes a range.
    // Raises CollectionChanged (NotifyCollectionChangedAction.Remove)
    public void RemoveRange(int index, int count);

    // Will allow to replace a range with fewer, equal, or more items.
    // Raises CollectionChanged (NotifyCollectionChangedAction.Replace)
    public void ReplaceRange(int index, int count, IEnumerable<T> collection);

    // Removes any item that matches the search criteria.
    // Raises CollectionChanged (NotifyCollectionChangedAction.Remove)
    // RWM: Excluded for now, will see if possible to add back in after implementation and testing.
    // public int RemoveAll(Predicate<T> match);

Obstacles

Doing this properly, and having the methods intuitively named, could potentially have the side effect of breaking existing classes that inherit from ObservableCollection to solve this problem. A good way to test this would be to make the change, compile something like Template10 against this new assembly, and see if it breaks.


So the ObservableCollection is one of the cornerstones of software development, not just in Windows, but on the web. One issue that comes up constantly is that, while the OnCollectionChanged event has a structure and constructors that support signaling the change for multiple items being added, the ObservableCollection does not have a method to support this.

If you look at the web as an example, Knockout has a way to be able to add multiple items to the collection, but not signal the change until the very end. The ObservableCollection needs the same functionality, but does not have it.

If you look at other extension methods to solve this problem, like the one in Template10, they let you add multiple items, but do not solve the signaling problem. That's because the ObservableCollection.InsertItem() method overrides Collection.InsertItem(), and all of the other methods are private. So the only way to fix this properly is in the ObservableCollection itself.

I'm proposing an "AddRange" function that accepts an existing collection as input, optionally clears the collection before adding, and then throws the OnCollectionChanging event AFTER all the objects have been added. I have already implemented this in a PR #10751 so you can see what the implementation would look like.

I look forward to your feedback. Thanks!

@joshfree joshfree added this to the 1.2.0 milestone Aug 16, 2016

@robertmclaws

This comment has been minimized.

Copy link
Author

robertmclaws commented Sep 13, 2016

@joshfree @Priya91 Since I already have a PR that addresses this issue, is there any way this could be moved up to 1.1?

@LanceMcCarthy

This comment has been minimized.

Copy link

LanceMcCarthy commented Sep 13, 2016

While you're in there adding an AddRange() method, can you throw an OnPropertyChanged() into the Count property's setter? Thanks :)

@thomaslevesque

This comment has been minimized.

Copy link
Collaborator

thomaslevesque commented Sep 13, 2016

A long time ago I had implemented a RangeObservableCollection with AddRange, RemoveRange, InsertRange, ReplaceRange and RemoveAll. But it turned out that the WPF binding system didn't support CollectionChanged notifications with multiple items (I seem to remember it has been fixed since then, but I'm not sure).

@joshfree

This comment has been minimized.

Copy link
Member

joshfree commented Sep 13, 2016

@Priya91 can you help shepherd this through the API review process http://aka.ms/apireview with @robertmclaws ?

/cc @terrajobst

@Priya91

This comment has been minimized.

Copy link
Member

Priya91 commented Sep 13, 2016

@Priya91 can you help shepherd this through the API review process http://aka.ms/apireview with

Sure.

@Priya91

This comment has been minimized.

Copy link
Member

Priya91 commented Sep 13, 2016

@robertmclaws Can you create an api speclet on this issue, outling the api syntax, like this. Mainly interested in usage scenarios

@svick

This comment has been minimized.

Copy link
Contributor

svick commented Sep 14, 2016

@robertmclaws

Doing this properly, and having the methods intuitively named, could potentially have the side effect of breaking existing classes that inherit from ObservableCollection to solve this problem.

In what situation could it be a breaking change? The only issue I can think of is that it would cause a warning that tells you to use new if you meant to hide a base class member, which would be actually an error with warnings as errors enabled. Is this what you meant? Or is there another case I'm missing?

@robertmclaws

This comment has been minimized.

Copy link
Author

robertmclaws commented Sep 14, 2016

@svick Could possibly be a runtime problem. If you just upgraded the framework w/o recompiling, I'm not sure exactly how the runtime execution would react. We'd need to test it just to make sure.

@svick

This comment has been minimized.

Copy link
Contributor

svick commented Sep 14, 2016

@robertmclaws I think that could only be a problem if you don't recompile, but you do upgrade a library with the custom type inheriting from ObservableCollection<T>, which removed its version of AddRange() in the new version. But that would be the fault of that library.

Otherwise, adding a new member won't affect how old binaries behave.

@Priya91

This comment has been minimized.

Copy link
Member

Priya91 commented Sep 14, 2016

+1 The api sounds good to me. For manipulating multiple items , along with AddRange, does it provide value to add, InsertRange, RemoveRange, GetRange for the specified usage scenarios?

cc @terrajobst

@robertmclaws

This comment has been minimized.

Copy link
Author

robertmclaws commented Sep 16, 2016

@svick You are probably right. I personally would want to test the behavior just to be sure we're not breaking anyone... otherwise this would move to a 2.0 release item.

@Priya91 I'm not sure if a GetRange() would be necessary, but InsertRange() and RemoveRange() would be, along with ReplaceRange(), and possible a Clear() method if one is not currently available.

So if we're comfortable with the API, what's the next step? :)

@Priya91

This comment has been minimized.

Copy link
Member

Priya91 commented Sep 16, 2016

Clear is already available. We still haven't gotten the shape of apis to add, if RemoveRange and InsertRange are to be added, then we need these apis added to the speclet. And then we'll mark api-ready-for-review, to be discussed in the next api-review meeting either on tuesday or friday.

@robertmclaws

This comment has been minimized.

Copy link
Author

robertmclaws commented Sep 16, 2016

OK, I made changes to the speclet. Note that the parameters might change for the actual implementation, but those are what makes the most sense at this particular second. Please LMK if I need to do anything else. Thanks!

@Priya91

This comment has been minimized.

Copy link
Member

Priya91 commented Sep 16, 2016

RemoveRange(int index, int count) instead of RemoveRange(ICollection) ? How does RemoveRange behave when the ICollection elements are duplicated in ObservableCollection

@Priya91

This comment has been minimized.

Copy link
Member

Priya91 commented Sep 16, 2016

count instead of endIndex..

public void ReplaceRange(IEnumerable<T> collection, int startIndex, int count)
@Priya91

This comment has been minimized.

Copy link
Member

Priya91 commented Sep 16, 2016

public void AddRange(IEnumerable<T> collection, bool clearFirst = false) { }
public void InsertRange(IEnumerable<T> collection, int startIndex) { }
public void RemoveRange(int startIndex, int count) { }
public void ReplaceRange(IEnumerable<T> collection, int startIndex, int count) { }
@thomaslevesque

This comment has been minimized.

Copy link
Collaborator

thomaslevesque commented Sep 16, 2016

Basically the signatures should be the same as in List<T>.

I don't think the clearFirst parameter in AddRange is useful, and anyway optional parameters should be avoided in public APIs.

A RemoveAll method would be useful all well, for consistency with List<T>:

public int RemoveAll(Predicate<T> match)
@robertmclaws

This comment has been minimized.

Copy link
Author

robertmclaws commented Sep 16, 2016

I think RemoveRange(IEnumerable<T> collection) should remain. It would cycle through collection, call IndexOf(item) and then call RemoveAt(index). Duplicates of the same item would also be removed.

@thomaslevesque I have the clearFirst parameter in there specifically because it IS useful, as in I'm using it in production code right now. Consider in UWP apps when you are resetting a UI... if you call Clear() first, it will fire another CollectionChanged event, which is not always desirable.

I'm not against a RemoveAll function.

@thomaslevesque

This comment has been minimized.

Copy link
Collaborator

thomaslevesque commented Sep 16, 2016

Also, the index parameter usually comes first in existing APIs, so InsertRange, RemoveRange and ReplaceRange should be updated accordingly.

And I don't think ReplaceRange needs a count parameter; what should the method do if the count parameter doesn't much the number of items in the replacement collection?

Here's the API as I see it:

public void AddRange(IEnumerable<T> collection) { }
public void InsertRange(int index, IEnumerable<T> collection) { }
public void RemoveRange(int index, int count) { }
public void ReplaceRange(int index, IEnumerable<T> collection) { }
public int RemoveAll(Predicate<T> match)
@thomaslevesque

This comment has been minimized.

Copy link
Collaborator

thomaslevesque commented Sep 16, 2016

@thomaslevesque I have the clearFirst parameter in there specifically because it IS useful, as in I'm using it in production code right now. Consider in UWP apps when you are resetting a UI... if you call Clear() first, it will fire another CollectionChanged event, which is not always desirable.

I'm not sold on it, but hey, it's your proposal, not mine 😉. At the very least, I think it should a separate overload, rather than an optional parameter.

@thomaslevesque

This comment has been minimized.

Copy link
Collaborator

thomaslevesque commented Sep 16, 2016

@thomaslevesque I have the clearFirst parameter in there specifically because it IS useful, as in I'm using it in production code right now. Consider in UWP apps when you are resetting a UI... if you call Clear() first, it will fire another CollectionChanged event, which is not always desirable.

This makes me think... there are lots of possible combination of changes you might want to do on the collection without triggering events for each one. So instead of trying to think of each case and introduce a new method for each, perhaps we should lean toward a more generic solution. Something like this:

using (collection.DeferCollectionChangedNotifications())
{
    collection.Add(...);  // no event raised
    collection.Add(...); // no event raised
    // ...
} // event raised here for all changes
@robertmclaws

This comment has been minimized.

Copy link
Author

robertmclaws commented Sep 16, 2016

@thomaslevesque Overload vs optional parameter makes no practical difference to the end user. It's just splitting hairs. Having overloads just adds unnecessary lines of code.

ReplaceRange with a count would remove all items in the given range, and then insert the new items at that point. The counts not matching would be irrelevant.

If the index comes first in existing APIs, then I'm fine with this:

public void AddRange(IEnumerable<T> collection, clearFirst bool = false) { }
public void InsertRange(int index, IEnumerable<T> collection) { }
public void RemoveRange(int index, int count) { }
public void ReplaceRange(int index, int count, IEnumerable<T> collection) { }
public int RemoveAll(Predicate<T> match)
@thomaslevesque

This comment has been minimized.

Copy link
Collaborator

thomaslevesque commented Sep 16, 2016

@thomaslevesque Overload vs optional parameter makes no practical difference to the end user. It's just splitting hairs.

It's not. Optional parameter can cause very real issues when used in public APIs. Read this blog post by @Haacked for details.

@weitzhandler

This comment has been minimized.

Copy link
Collaborator

weitzhandler commented Oct 9, 2018

I'd be happy to be assigned too.

@safern

This comment has been minimized.

Copy link
Member

safern commented Oct 9, 2018

All yours 😄

@weitzhandler

This comment has been minimized.

Copy link
Collaborator

weitzhandler commented Oct 9, 2018

Tx. I'd need a bit of help tho.

Under what solution is Collection.cs?
Looks like it's part of Lib, I'm not sure where to get started.

Please have a look here.

@safern

This comment has been minimized.

Copy link
Member

safern commented Oct 9, 2018

@safern

This comment has been minimized.

Copy link
Member

safern commented Oct 9, 2018

Should I first open a separate PR in the coreclr repo for the changes in Collection?

Yes, you can open a simultaneous PR in corefx to expose the APIs in the reference assemblies, add tests and also add the implementation for ObservableCollection. However the CI builds will fail until the coreCLR PR is merged and corefx consumes a new version of the coreclr package.

After adding the APIs in Collection implementation in order to test your changes in CoreFX, once you added the APIs to the reference assembly for Collection you will want to follow this steps to build CoreFX wit a custom version of coreclr: https://github.com/dotnet/corefx/blob/d9193a1bd70eee4320f8dcdd4e237ee9acf689e9/Documentation/building/advanced-inner-loop-testing.md#compile-corefx-with-self-compiled-coreclr-binaries

Please let me know here or in gitter if you need help from my side 😄

@weitzhandler

This comment has been minimized.

Copy link
Collaborator

weitzhandler commented Oct 9, 2018

Where are the tests for Collection<T> located?

@safern

This comment has been minimized.

@alexsorokoletov

This comment has been minimized.

Copy link
Contributor

alexsorokoletov commented Oct 20, 2018

Finally some justice for the observable collections :) Wonder though how it would work in real-life existing derived classes with AddRange

@adrientetar

This comment has been minimized.

Copy link

adrientetar commented Nov 9, 2018

FWIW, two other methods I'm missing in IList/ICollection are GetRange and Reverse. The bare-bones interfaces provided mean I have to either give up on returning an interface in my properties or give up the nice extra methods...

@weitzhandler

This comment has been minimized.

Copy link
Collaborator

weitzhandler commented Jan 23, 2019

@safern Is there a proper guide on adding tests for System.Private.CoreLib (they're added in corefx), so that it runs against the clr on my machine?
I'm reading this but I'm not certain this is the right one, can you please confirm or elaborate?

@safern

This comment has been minimized.

@ChaseFlorell

This comment has been minimized.

Copy link

ChaseFlorell commented Feb 26, 2019

Unless I'm thick and missing something super obvious, this appears to be landing in NetCore as opposed to NetStandard. Does it need to be implemented in both, will this cover NetStandard?

@danmosemsft

This comment has been minimized.

Copy link
Member

danmosemsft commented Feb 27, 2019

@ChaseFlorell this repo is for.NET Core as you mention. The dotnet/standard repo is the place where the standard is defined and requests should go in In issues there

@weitzhandler

This comment has been minimized.

Copy link
Collaborator

weitzhandler commented Feb 27, 2019

Because the Collection is in Private Lib in CLR, I find it really hard and time consuming to set up the tests for my changes in the CLR. Gave it a few shots but didn't have much of success, need to find a free afternoon to get it done 💔

@safern

This comment has been minimized.

Copy link
Member

safern commented Feb 27, 2019

Gave it a few shots but didn't have much of success, need to find a free afternoon to get it done 💔

Please let me know or ping me on gitter if you get stuck at some point.

@ahoefling

This comment has been minimized.

Copy link
Contributor

ahoefling commented Mar 4, 2019

Before I realized that the mono corefx implementation is forked from here I got AddRange and InsertRange working locally. Today I spent some time porting it to the main repository to see if I could get it to work with the xUnit tests.

@weitzhandler I would be happy to combine our efforts so we can get this wrapped up as I am able to get the CoreCLR build working with my xUnit tests.

My Current Implementation:
Collection (only):

  • AddRange & xUnit Tests
  • InsertRange & xUnit Tests

Update

I ended up taking this all the way to the point where I can submit a PR to complete this. Looking forward to everyone's feedback

@Grauenwolf

This comment has been minimized.

Copy link

Grauenwolf commented Mar 5, 2019

Do we have an open ticket for adding AddRange to the concurrent collections?

@terrajobst

This comment has been minimized.

Copy link
Member

terrajobst commented Mar 8, 2019

@safern is this on track on being done soonish? If so, we could still add it to .NET Standard.

@safern

This comment has been minimized.

Copy link
Member

safern commented Mar 8, 2019

Awesome. Yeah PRs are our in CoreCLR and CoreFX. Will try to get them in ASAP

@ahoefling

This comment has been minimized.

Copy link
Contributor

ahoefling commented Mar 8, 2019

@safern @terrajobst What is the process to getting this into the Standard from here? I went ahead before I started dev on this and created an issue to track this in the mono project as well. mono/mono#13265.

I am happy to continue working on anything additional that needs to be done on the other projects to help get this in. This is really going to be a great addition to .NET Standard

@sharpe5

This comment has been minimized.

Copy link

sharpe5 commented Mar 29, 2019

WPF developer from a large exchange-listed company here.

It's incredible how much of a difference a proper .AddRange() function will make to the speed of ObservableCollection. Adding items one-by-one with a CollectionChanged event after every add is incredibly slow. Adding lots of items with one CollectionChanged at the end is orders of magnitude faster.

tl;dr AddRange() makes the difference between a huge grid with 10,000 items being fast, fluent and responsive vs. slow, clunky and unresponsive (to the point of being unusable).

So this PR gets my +1 vote. Of all the improvements to grid speed, this is by far the most important.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.