New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support filtered Include #1833

Open
0xdeafcafe opened this Issue Mar 16, 2015 · 115 comments

Comments

Projects
None yet
@0xdeafcafe
Contributor

0xdeafcafe commented Mar 16, 2015

We keep this issue to track specifying filters inline when you specify eager loading in a query. However there are many scenarios that can be satisfied with global query filters:

Please learn about global query filters

We are seeing that a large portion of the scenarios customers want this feature for can already be better addressed using global query filters. Please try to understand that feature before you add your vote to this issue.

We are keeping this issue open only to represent the ability to specify filters on Include on a per-query basis, which we understand can be convenient on some cases.

Original issue:

Doing a .Where() inside a .Include() is allowed (by the syntax, it fails in execution). However, if you then follow that up with a .ThenInclude(), like so:

.Include(t => t.Ratings.Where(r => !r.IsDeleted))
.ThenInclude(r => r.User)

You get the following error:

'IEnumerable' does not contain a definition for 'User' and no extension method 'User' accepting a first argument of type 'IEnumerable' could be found (are you missing a using directive or an assembly reference?)

Is this something you're aware of and going to allow in a future version, or am I just doing something wrong and It's already supported?

@popcatalin81

This comment has been minimized.

popcatalin81 commented Mar 16, 2015

Historically this was not supported by EF, However this would be a nice feature to have. We had to implement something similar ourselves in the our code base using Linq tree rewriting. It hope the EF team considers this.

@0xdeafcafe

This comment has been minimized.

Contributor

0xdeafcafe commented Mar 16, 2015

Yeah. I noticed discussion of this in the .Include() design meeting docs. It looked as if it was road-mapped and going to be implemented. Although that could all just be a dreadful fantasy and I'm completely wrong.

@rowanmiller rowanmiller added this to the Backlog milestone Mar 23, 2015

@rowanmiller

This comment has been minimized.

Member

rowanmiller commented Mar 23, 2015

Moving to back log as a new feature (we do want to support filtered include but it is a lower priority than some other features that were available in EF6).

@rowanmiller rowanmiller changed the title from Can't do a Where statement inside an Include, if it's followed by another Include to Support filtered Include Mar 23, 2015

@rowanmiller

This comment has been minimized.

Member

rowanmiller commented Mar 23, 2015

Opened new issue for better exception message #1883

@mikes-gh

This comment has been minimized.

mikes-gh commented Dec 12, 2015

Don't underestimate the value of filtered include. It was a top request in ef 6.
I came across this requirement on my first ef 7 project.
I'd take that over lazy loading any day.

@rendmath

This comment has been minimized.

rendmath commented Dec 30, 2015

Any non-trivial development using the Entity Framework will encounter this limitation.

It also has consequences on other technologies likes the OData protocol, because there is really no point to adding filtered $expands to the protocol if EF does not support them.

For those who are not (yet) familiar with the internals of EF Core, could you point us to the parts of the Entity Framework that would be impacted in order support this ?

@joshmouch

This comment has been minimized.

joshmouch commented Jan 21, 2016

I too would love this feature.

@gdoron

This comment has been minimized.

gdoron commented Jul 25, 2016

@rowanmiller so what is the current way of using left join with a where clause \ on clause?
I currently query for the entire relationship and filter on the client.
Is there a better way of doing it?

var story = await _context.Stories
    .Include(x => x.Paragraphs)
    .Include(x => x.User)
    .Include(x => x.Tips)
    .Include(x => x.Images)
    .Include(x => x.Comments)
    .ThenInclude(x => x.User)
    .SingleOrDefaultAsync(x => x.Id == storyId);
if (story == null)
    return null;

story.Comments = story.Comments.Where(x => !x.IsDeleted).ToList();
story.Images = story.Images.Where(x => !x.IsDeleted).ToList();
story.Paragraphs = story.Paragraphs.Where(x => !x.IsDeleted).ToList();
story.Tips = story.Tips.Where(x => !x.IsDeleted).ToList();
return story;

It's a mess... and bad performance-wise.

@lucacestola

This comment has been minimized.

lucacestola commented Jul 25, 2016

Will Filtered Loading be supported at configuration level, in order to have one to many navigation property filtered upstream? Can this feature be someway useful to avoid bad configurations with circular reference like this?

@gdoron

This comment has been minimized.

gdoron commented Jul 25, 2016

@lucacestola I'm not sure why you think it's related to conditional querying of a navigation property.

@lucacestola

This comment has been minimized.

lucacestola commented Jul 25, 2016

@gdoron because it could be also applyed to one to many relationships and would be implicit.

Actually, with EF 6, I was not yet able to find a good solution for the relation type like the one at the posted link, without using a circular FK and, at the meantime, have the possibility to use an expression like this: parent.MainChild.ChildProperty.

I know that this kind of relations depends on a very custom logic so there is no simple way to address such a need, and I was hoping that, configuring the way relationships are loaded could almost partially address the issue.

@Bartmax

This comment has been minimized.

Bartmax commented Jul 25, 2016

I just have the exactly same need as @gdoron (and not surprising, with IsDeleted fields as well)
is there any way to left join w/ where at db level?

@armartinez

This comment has been minimized.

armartinez commented Sep 5, 2016

I'm also in the @gdoron scenario, but I return a collection instead of a single record so I'm doing a foreach on the collection and something like
story.Comments = story.Comments.Where(x => !x.IsDeleted).ToList();
on each record.
I've tried this http://stackoverflow.com/questions/4720573/linq-to-entities-how-to-filter-on-child-entities/4724297#4724297 but it doesn't seem to work on EF Core. Is there a better way to do this?

@arielcloud

This comment has been minimized.

arielcloud commented Sep 5, 2016

You can query stories without including comments, then query separately comments and Explicit loading them: https://docs.efproject.net/en/latest/querying/related-data.html

@armartinez

This comment has been minimized.

armartinez commented Sep 5, 2016

Looks interesting, but I need to filter the first query to only return a record if there are any results in the specific navigation property, so I need to have the include or change the result of the first query when the explicit loading doesn't return anything which I think it's worse.

@atrauzzi

This comment has been minimized.

atrauzzi commented Oct 6, 2016

Yikes, don't know why this is sitting idle in a backlog. Should be considered for any upcoming version as per what the comments from @mikes-gh and @rendmath imply. Easy to overlook this one.

@gdoron

This comment has been minimized.

gdoron commented Oct 6, 2016

@atrauzzi well, this feature was and still is idle for years in EF, so I'm afraid ... 😢
Not sure why it's not prioritized.

@HappyNomad

This comment has been minimized.

HappyNomad commented Nov 1, 2016

Yes, this feature request has been around a while. I found it posted on UserVoice since 2010. It's crucial for allowing me to properly load my complex data model, so I hope it's soon given higher priority.

@gdoron

This comment has been minimized.

gdoron commented Nov 21, 2016

@rowanmiller @divega Can you please share with us when if ever this is going to be implemented?
We designed our data structure in a way that is best practice regarding DB design, but heavily rely on this feature.

Consider this simplified scheme:

public class Post
{
    public int Id { get; set; }
    public string Text { get; set; }
    public List<PostImage> Images { get; set; }
}

public class PostImage
{
    public int Id { get; set; }
    public bool IsDeleted { get; set; }
    public Post Post { get; set; }
    public int PostId { get; set; }
    public string Url { get; set; }
    public string CdnUrl { get; set; }
    public ImageType Type { get; set; }
}

public enum ImageType
{
    Normal = 0,
    Primary = 1,
    Header = 2,
    Profile = 3
}

Now lets say I want to query 10 posts for my blog homepage with their single Primary image.
Currently, there is no way of doing it.
I will have to query for 10 posts with ALL of their images (even the deleted ones!) and only on the client filter out the useless data.

As our application is getting more sophisticated and gets more traffic, this is becoming a real and significant pain and we need a solution for this.

Is it going to have the same luck as the feature request on EF which was and still is idle for 6 years?

We really need an answer, there are solutions like denormalize our data model but that's rarely a good idea.

Thanks!

@rowanmiller

This comment has been minimized.

Member

rowanmiller commented Nov 21, 2016

@gdoron we do want to do this, but it's not at the top of the list. Our Roadmap will give you an idea of the things that are going to be worked on next. You will see that this is listed under "High Priority", but just not the "Critical O/RM" list.

@gdoron

This comment has been minimized.

gdoron commented Nov 22, 2016

@rowanmiller I'm sure everyone has different points of view and needs but here are my two cents:
Most of the things that are missing have somewhat reasonable workarounds.

e.g.
Lazy load - simply Include upfront all your data.
View mapping- Manually create a migration.
SP Mapping - Use some technique as with View.
etc.

But Filtered Include has 0 reasonable workarounds.
The only workaround is writing raw SQL but in many cases you need it for almost all of your queries, so that's not an option or else why use an ORM to begin with.

So reiterating what @mikes-gh wrote months ago:

Don't underestimate the value of filtered include. It was a top request in ef 6.
I came across this requirement on my first ef 7 project.
I'd take that over lazy loading any day.

I had already 3 projects on EF Core, and it was a requirement and a pain in ALL of them.

@rowanmiller

This comment has been minimized.

Member

rowanmiller commented Nov 22, 2016

Just to be clear, the items on the roadmap are the very top of the 100s of feature requests we have sitting on the backlog. So it's inclusion on the list means that it is one of our top priorities. The split between "Critical" and "High Priority" is always subjective. The current split is based on our imperfect aggregation of the feedback from all our customers.

It's not as clean as true filtered loading support, but you can do something like this to do filtered loading. It will run two queries, but that is what EF Core would do under the covers anyway, when you load a collection.

var blogs = db.Blogs
    .Where(b => b.Rating> 3)
    .ToList();

var blogIds = blogs.Select(b => b.BlogId).ToList();

db.Posts.Where(p => blogsIds.Contains(p.BlogId))
    .Where(p => p.Rating > 3)
    .Load();
@gdoron

This comment has been minimized.

gdoron commented Nov 22, 2016

@rowanmiller

It will run two queries, but that is what EF Core would do under the covers anyway, when you load a collection.

I thought it's a temporary limitation that you're working on fixing.
Are you telling me it is by design?
Because it's not just two queries, it's two round trips to the DB.

Anyway, it might be acceptable for one collection loading, but when you have 10 included entities (we already have 6-7 in some places,), that means 11 round trips or querying the DB with 10 connections concurrently.

I might got you wrong, but if I didn't... Houston we have a problem.

@mhosman

This comment has been minimized.

mhosman commented Feb 2, 2018

@anpete Do you mean to initialize the value in the PermissionType Class? (sorry my confusion but I can't see where to initialize that specific value for that specific query in the controller).

@anpete

This comment has been minimized.

anpete commented Feb 2, 2018

The normal way is to have the value initialized when creating the context:

var context = new MyContext(permissionType: 1);

In this case, the value is "set" for the lifetime of that specific context instance. Other instances can have difference values.

You should also be able to add a PermissionType property to your context.

var context = new MyContext();
context.PermissionType = 1;
var q = context.Users.Include(u => u.Permissions).ToList();

This is less common but should allow you to change the value on the fly.

@mhosman

This comment has been minimized.

mhosman commented Feb 2, 2018

Great @anpete! Thank you very much for your help!

@mhosman

This comment has been minimized.

mhosman commented Feb 2, 2018

@anpete Just one last question. How to access _permissionType attribute from a Self-contained type configuration file?

@anpete

This comment has been minimized.

anpete commented Feb 2, 2018

@smitpatel Passing in the context to the entity configuration works, right?

@smitpatel

This comment has been minimized.

Contributor

smitpatel commented Feb 2, 2018

No. See issue #10301

@mhosman

This comment has been minimized.

mhosman commented Feb 2, 2018

Yes, passing the context to a file with this example code:

class CustomerConfiguration : IEntityTypeConfiguration<Customer>
{
  public void Configure(EntityTypeBuilder<Customer> builder)
  {
     builder.HasKey(c => c.AlternateKey);
     builder.Property(c => c.Name).HasMaxLength(200);
   }
}
@anpete

This comment has been minimized.

anpete commented Feb 2, 2018

Thanks @smitpatel

@mhosman For now you will need to define the filters in OnModelCreating directly

@weitzhandler

This comment has been minimized.

Contributor

weitzhandler commented Mar 7, 2018

How about:

.Include(f => f.Thumbnails
  .Where(t => t.ThumbnailSize < maxThumbSize)
  .OrderBy(t => t.Rank)
  .Take(3))
.ThenInclude(...)
@stap123

This comment has been minimized.

stap123 commented Mar 13, 2018

Just want to throw a +1 onto this, would be excellent to have this feature I commonly hit this limitation.

@n123r

This comment has been minimized.

n123r commented Apr 21, 2018

Also adding a +1 onto this feature. The ability to filter on include on a per query basis would be an excellent addition to EF Core

@conterio

This comment has been minimized.

conterio commented Apr 24, 2018

It's been a couple years now! Work on this feature please!!!!! Query on Includes. I'm going to Ignite this year, have it done by then or else. Or else what? Or else i'll just have to keep waiting.

@orobert91

This comment has been minimized.

orobert91 commented May 25, 2018

I also upvote this feature, and yes I know about global query filters and it doesn't solve any of my use cases.

@SonicGD

This comment has been minimized.

SonicGD commented Jun 14, 2018

+1 to this feature. Global filters doesn't solve my problems, i need to set condition on per-query basis

@ajbeaven

This comment has been minimized.

ajbeaven commented Jun 20, 2018

Can we lock new comments on this issue? I'm keen to hear progress on this but don't want to be spammed by inane +1 comments.

If you wish to throw support behind this issue, thumbs up the OP and subscribe to notifications. The fourth comment here says that this will happen at some point, so either be patient or create a pull request.

@HairyPorker

This comment has been minimized.

HairyPorker commented Jun 22, 2018

to OP, my current workaround is to use LINQ join

@jmatter

This comment has been minimized.

jmatter commented Jun 27, 2018

I use the following workaround:

var entity = context.Table.First();
context.Entry(entity)
                        .Collection(parent => parent.myChildTable)
                        .Query()
                        .Where(child => child.amount > 100)
                        .ToList();

The collection of myChildTable inside the parent entity then gets populated with the filtered entities.

@KuzDeveloper

This comment has been minimized.

KuzDeveloper commented Jul 12, 2018

@jmatter How do you use this approach, when you want to retrieve list of items instead of just the first one, and also, when the child table also has children? So, like:

class A { List ListOfB; bool IsDeleted; int JustANumber; }
class B { List ListOfC; bool IsDeleted; int SomethingElse; }
class C { ... }

What I would normally do is this:

public IEnumerable GetFilteredListOfA(int parameter)
{
IEnumerable result = MyDbContext.ClassA.Where(a => !a.IsDeleted && JustANumber >= parameter).Include(a => a.ListOfB.Where(b => !b.IsDeleted && SomethingElse >= parameter).Include(b => b.ListOFC....);

Right now I need to use the .Select and create new typed instances, but the code is quite long and I had to break my domain driven approach as e.g. the Id field now cannot be set to private, else all my items will have the Id of 0 instead of the real one. So, with Select it looks like this:

IEnumerable result = MyDbContext.ClassA.Where(a => !a.IsDeleted && JustANumber >= parameter)
.Select(a => new ClassA()
{
Id = a.Id,
ListOfB = a.ListOfB.Where(...).Select(b => new ClassB() { Id = b.Id, ListOFC = ...
}

@YandyZaldivar

This comment has been minimized.

YandyZaldivar commented Jul 25, 2018

Consider the scenario where instead of:
query.Include(p => p.Property)
we have:
query.Include("Xxx.Yyy.Zzz")
only the chained property names.

How are we going to add the Where, OrderBy, Take and Skip filters?

We can try to create some Expression for our property name:

static Expression<Func<T, object>> ToLambda<T>(string propertyName)
{
	var parameter = Expression.Parameter(typeof(T));
	var property = Expression.Property(parameter, propertyName);
	var propAsObject = Expression.Convert(property, typeof(object));

	return Expression.Lambda<Func<T, object>>(propAsObject, parameter);
}

or even something like for nested properties:
var body = propertyName.Split('.').Aggregate<string, Expression>(param, Expression.PropertyOrField);

But this seems an obscure path to me.

@YandyZaldivar

This comment has been minimized.

YandyZaldivar commented Jul 25, 2018

https://docs.microsoft.com/en-us/ef/core/querying/related-data

Tip
Entity Framework Core will automatically fix-up navigation properties to any other entities that were previously loaded into the context instance. So even if you don't explicitly include the data for a navigation property, the property may still be populated if some or all of the related entities were previously loaded.

What if we want to include only what we filter, and if we filter nothing for a specific query, include nothing?

Take into account that data obtained is potentially for being serialized over the network (to client apps), so it is not a good idea end up sending what we don't need.

@Der-Kraken

This comment has been minimized.

Der-Kraken commented Sep 19, 2018

I do not understand why 'global query filters' should help me in case of filtering a navigation collection. The query depends from case to case.
I tried so much ways to filter included collections because i never thougth that this functionality not exists.
The EntityFramework Core is realy a cool thing!

@KuzDeveloper

This comment has been minimized.

KuzDeveloper commented Sep 19, 2018

I did not say it is easy to achieve this, but in its current state EF Core is not production ready. We used it for one of our microservices and it is causing a lot of performance issues to load too many unnecessary records when we already have 6-7 different filters available. Other than this, it would be an excellent tool.

@GertArnold

This comment has been minimized.

GertArnold commented Sep 23, 2018

Anybody in favor of filtered includes should realize that it requires some dilemmas to be solved:

Should a partially loaded collection be marked as loaded?

Now that lazy loading is possible in EF core it's relevant to know whether a collection is loaded or not. A collection navigation property is marked as loaded when -

  • It is Included
  • It is loaded lazily
  • The Load statement (not relevant for now)

A collection is not marked as loaded when -

  • It is populated (either partially or fully) by relationship fixup

With lazy loading, when we have one entity, entity1, and we access entity.Children, EF will query the children from the database when the collection is not marked as loaded.

So what to do with a partially Include-ed collection? When it's not marked as loaded its content will be overwritten when lazy loading is triggered. Most developers will not expect that: they filtered the Include with a purpose.
However, when it's marked as loaded lazy loading will never populate the entire collection and developers who count on that to happen will be puzzled. Also, when additional child entities get attached to the context otherwise, it will be impossible to tell which were part of the loaded collection.

What should the behavior be with repeated Include statements?

Suppose we have this statement:

var entities1 = context.Entities.Include(e => e.Children.Where(c => c.Active));

Later on in the same scope, this:

var entities2 = context.Entities.Include(e => e.Children.Where(c => !c.Active));

What do we expect an Entity's Children collection to contain now? The first statement attached the active children to the context, the second statement attached the inactive children. Lazy loading aside, relationship fixup commands that each Entity contains all of its children, so entities1 and entities2 aren't essentially different any more, although a developer might expect them to be.

Or should the last Include statement be leading? But then, in view of relationship fixup, what to do with the attached active children? Detach them? OK, but what if they have changes? We can't simply ditch these changes, can we? So keep the change tracker aware of these changes (somehow) while still disposing the entity objects? I'm afraid that's would be a major departure of the current architecture.

I can't speak for the EF team, but I think they are aware of these dilemma's, which may explain their wariness of implementing this feature.

@HairyPorker

This comment has been minimized.

HairyPorker commented Oct 2, 2018

@GertArnold I am not sure I understand the dilemma.
In your sample,
The first statement should return all the entities including the children in which active == true;
The second statement should return all the entities including the children in which active == false;

At least this is what my use case most of the time, that EF unable to support filtering on the Include / children level; of course we could always workaround with Select or other awesome solutions that mentioned in this thread :)

@GertArnold

This comment has been minimized.

GertArnold commented Oct 2, 2018

@HairyPorker

The second statement should return all the entities including the children in which active == false;

But it doesn't, that's my point. Sure, the SQL query will return only inactive children. But right after the statement all children are tracked, so all Children collections of all entities will be populated fully if relationship fixup keeps working the way it does now. Keep in mind that entities1 and entities2 contain the same parent instances, because the statements run in the same scope, i.e. using one and the same context instance.

@YandyZaldivar

This comment has been minimized.

YandyZaldivar commented Oct 2, 2018

@GertArnold even that being the case, the ability to create such filtered query is a great plus for many scenarios. Internal performance improvements could achieved later with the corresponding clarification, one example is the recent introduction of GroupBy translation in 2.1.

@Der-Kraken

This comment has been minimized.

Der-Kraken commented Oct 4, 2018

@GertArnold That is an interesting point of view and I think that could be a problem in a long living context instance (not a technical problem but more an organizational).

But I understand the DataContext as an UnitOfWork. An UnitOfWork is created for one task. After that it get's disposed. During one task I do not believe that the scenario you describe is relevant.

To response directly to the situation you described: I would expect that the last query wins (so it replaces the previous ones).

@salaros

This comment has been minimized.

salaros commented Nov 16, 2018

Currently I have can't filter the translations of children elements, so a complex result like this

return await _db.Products
                .Include(p => p.Documents)
                    .ThenInclude(a => a.Translations)
                .Include(p => p.ProductCategories)
                    .ThenInclude(pc => pc.Category)
                        .ThenInclude(c => c.Translations)
                .Include(p => p.ProductProperties)
                    .ThenInclude(pp => pp.Property)
                        .ThenInclude(c => c.Translations)
                .ToArrayAsync();

would look like this

// It's just en example. don't use similar code in real apps
var lang = context.Request.Headers["Accept-Language"];
return await _db.Products
                .Include(p => p.Documents)
                    .ThenInclude(a => a.Translations, t => lang.Equals(t.Language))
                .Include(p => p.ProductCategories)
                    .ThenInclude(pc => pc.Category)
                        .ThenInclude(c => c.Translations, t => lang.Equals(t.Language))
                .Include(p => p.ProductProperties)
                    .ThenInclude(pp => pp.Property)
                        .ThenInclude(prop => prop.Translations, t => lang.Equals(t.Language))
                .ToArrayAsync();
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment