New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Discussion on many-to-Many relationships (without CLR class for join table) #1368

Open
0xdeafcafe opened this Issue Jan 8, 2015 · 240 comments

Comments

Projects
None yet
@0xdeafcafe
Copy link
Contributor

0xdeafcafe commented Jan 8, 2015

Note:

  • Issue #10508 has been created to track actual work on many-to-many relationships. The new issue is locked so that it can be subscribed to for updates on the implementation without noise.
  • This older issue has been re-purposed to be discussion about many-to-many relationships in EF Core. It will not (for the time being) be locked because it is important to us that we don’t shut down channels of communication with the community.

With regard to feedback, I think it is worth reiterating some comments made a few months ago. We read and consider all feedback (and try to make the right choices based on it) no matter how it is delivered; that is, whether it is provided politely and with respect or not. Yelling at us or others with different opinions at best makes you feel good and us/them feel bad. It doesn’t achieve anything concrete. We are doing our best to prioritize features and implement them in the best way we can. We really appreciate your feedback and it makes a big difference on shaping the product. We personally appreciate it more when feedback is delivered in a respectful way, but please don’t stop providing constructive feedback.


Original issue:

I'm assuming there isn't a way to do Many-to-Many relationships in EF Core yet, as I've found nothing mentioning how to do it. Are there any examples on getting round this? Or am I just completely missing it, and it's actually there after all.

@bricelam

This comment has been minimized.

Copy link
Member

bricelam commented Jan 8, 2015

The workaround is to map the join table to an entity.

class Product
{
    public int Id { get; set; }
    public ICollection<Categorization> Categorizations { get; set; }
}

// NOTE: Entity key should be (ProductId, CategoryId)
class Categorization
{
    public int ProductId { get; set; }
    public Product Product { get; set; }

    public int CategoryId { get; set; }
    public Category Category { get; set; }
}

class Category
{
    public int Id { get; set; }
    public ICollection<Categorization> Categorizations { get; set; }
}

To navigate, use a Select:

// product.Categories
var categories = product.Categorizations.Select(c => c.Category);
@0xdeafcafe

This comment has been minimized.

Copy link
Contributor

0xdeafcafe commented Jan 8, 2015

Thanks!

@0xdeafcafe 0xdeafcafe closed this Jan 8, 2015

@tonysneed

This comment has been minimized.

Copy link
Contributor

tonysneed commented Jan 20, 2015

@bricelam,

Are there plans to support many-to-many without mapping the intersection table?

Also, will there be support for marking related entities as added or deleted to indicate they should be removed from the relationship and not the table? With EF6 and prior the only way to do this was to dip down to the ObjectContext API and work with independent associations. Thanks!

@bricelam

This comment has been minimized.

Copy link
Member

bricelam commented Jan 20, 2015

Yes, many-to-many associations will eventually be enabled via shadow state entities (#749) and skip-level navigation properties (not sure if there's a bug for that; I'll re-open this one).

I'm not sure I follow your second question, but @divega or @ajcvickers will know if it'll be covered by our "graph behavior" improvements.

@bricelam bricelam reopened this Jan 20, 2015

@divega

This comment has been minimized.

Copy link
Member

divega commented Jan 20, 2015

@tonysneed Re your second question, not sure I understand it either, but here is some data that might help:

We should have similar default behavior in EF Core as in previous versions of EF for the general case of removing an entity from a collection navigation property: this will cause the relationship to be marked as removed, not necessarily the entity.

Removing the relationship is promoted to an actual removal of the entity for identifying relationship only, i.e. when the primary key of the dependent entity contains the primary key of the principal entity and hence the dependent cannot be orphaned or re-parented. I believe we will have similar behavior for that too, although we have talked about deferring to the SaveChanges to detect if there are orphans entities in that state rather than trying to delete them immediately. @ajcvickers can elaborate/correct me on that.

If your question is about having API that allows to control the state of relationships without manipulating the collection navigation property directly on the entity, as described in https://entityframework.codeplex.com/workitem/271, then yes, I believe that would be something nice to have, however we haven't prioritized it. Your feedback would be helpful.

Does this help answer your question?

@tonysneed

This comment has been minimized.

Copy link
Contributor

tonysneed commented Jan 20, 2015

@bricelam, thanks for answering my first question. I'll be interested to learn more about implementing many-to-many via shadow state entities.

@divega, I'll clarify my second question. Let's say I have two entity classes that have a many-to-many relationship, for example, Employee and Territory from the Northwind sample database. If I wanted to add or remove a Territory from Employee.Territories, in a disconnected fashion using EF6, I would not be able to do so by setting EntityState on the Territory to Added or Deleted. Instead, I would call ObjectStateManager.ChangeRelationshipState, specifying the property name. Since ObjectContext is going away in EF Core, I'm just wondering how changing relationship state will work for entities in many-to-many relationships when performed in a disconnected manner (for example, from within a Web API controller). Hopefully this helps clarify my question a bit. Thanks!

@tonysneed

This comment has been minimized.

Copy link
Contributor

tonysneed commented Jan 21, 2015

Alight, I think I can clear everything up now that I've had a chance to try out a few things. The way to add or remove entities from a relationship in a disconnected fashion in vCurrent is not very straightforward or consistent when it comes to many-to-many relations. Stating that it is not possible without resorting to the OSM was incorrect. It can be done via the DbContext API but it's awkward. And what I'm wondering is if the API for EF Core could be improved to allow direct manipulation of the relationship, similar to what could be done via the OSM (albeit without needing to specify the navigation property name)?

For those following this conversation, here is what the OSM API looks like in vCurrent:

// Attach territory then set relationship state to added
context.Territories.Attach(territory);
var osm = ((IObjectContextAdapter)context).ObjectContext.ObjectStateManager;
osm.ChangeRelationshipState(employee, territory, "Territories", EntityState.Added);

So let's start with the vCurrent behavior of adding and removing entities from a m-m relation. Assuming that Employees and Territories have a m-m relationship, if I add a territory to an employee, then the territory will be marked as Added and EF will attempt to add it to the Territories table. If I want to just add the relationship to an existing territory, then I need to explicitly mark it as Unchanged, which makes sense but isn't all that intuitive. It's also inconsistent with the behavior for removing entities from a relationship, where removing a territory from an employee will not mark it as Deleted, without the need to explicitly mark it as Unchanged.

It seems to me that the remove behavior is a bit more straightforward. Adding a territory to the Territories collection of an employee could also leave it in an Unchanged state but result in adding it to the m-m relation. Then explicitly marking the entity as Added or Deleted would mean that you wanted to add to or remove from the Territories table as well as the m-m relation. This is the behavior that I would recommend for EF Core.

So this leads to the question of an API that would let you deal with the relationship directly. My question (which is hopefully clear by now) is whether could be a way in EF Core to change the state of a relationship. Following the proposal from CodePlex issue 271, it could look something like this:

context.Entry(employee).Member(e => e.Territories).Add(territory);

Even though may look somewhat intimidating, the important thing would be the ability to use the API from within the callback for AttachGraph (#1248), which is an Action. But for this to work, there would have to be a way to infer how the current entity in the graph is related to its parent (1-1, M-1, M-M). (This is what I do in my Trackable Entities project.) So we might need to add an overload of AttachGraph as follows:

void AttachGraph(object root, Action<EntityEntry, DbMemberEntry, EntityEntry> callback)

I'm probably getting this wrong, but we would need a parameter that represents how the current entity being iterated is related to its parent. (Here DbMemberEntry represents the navigation property and the second EntityEntry represents the parent of the current entity.) That way, we could figure out what kind of relationship it is and use the correct approach to attach the entity. Please let me know if I'm making sense here. :)

Cheers,
Tony

@rowanmiller rowanmiller changed the title Many-to-Many relationships Many-to-Many relationships (without CLR class for join table) Jan 23, 2015

@rowanmiller rowanmiller added this to the Backlog milestone Jan 23, 2015

@ctolkien

This comment has been minimized.

Copy link

ctolkien commented Jul 27, 2015

Just chiming in on this issue. Many-to-many is the one scenario that is blocking us from EF Core adoption. I realise that you can use intermediate mapping classes, but that requires significant changes to how we'd like to model things.

I note this is still tagged for the backlog. Is this feature on the roadmap for v1?

@damccull

This comment has been minimized.

Copy link

damccull commented Jul 27, 2015

Sooo....gimme. Many-To-Many is probably the only thing left that I NEED in order to do what I want...I WANT more features, but many-to-many is all I NEED...probably.

@popcatalin81

This comment has been minimized.

Copy link

popcatalin81 commented Jul 28, 2015

@bricelam, @rowanmiller Will this feature make into the RTM? I did not see it in the list of upcoming feature in the latest blog post. For me this is the only remaining feature preventing a full migration from EF6 to EF Core.

@rowanmiller

This comment has been minimized.

Copy link
Member

rowanmiller commented Jul 29, 2015

@popcatalin81 you can have a many-to-many relationship but you need to include an entity that represents the join table. This is how it will be for our first stable release. In the future we will enable shadow state entities (entities that don't have a CLR type) and then you will be able to model it without a CLR class to represent the join table.

@lajones

This comment has been minimized.

Copy link
Member

lajones commented Oct 6, 2015

Note: see also #3352 where the user has a requirement to support a join table which defines the combination of the 2 foreign keys as unique, but does not provide a primary key - which makes defining an entity type for it impossible.

@bragma

This comment has been minimized.

Copy link

bragma commented Nov 3, 2015

Regarding the original suggestion with the CLR class for join table, how should the "Categorization" class be used? Is it application's duty to create and delete instances of those class directly?
Something like...

var product = new Product();
var category = new Category();
var categorization = new Categorization()
{
Product = product,
Category = category
};

dbContext.Add(product);
dbContext.Add(category);
dbContext.Add(categorization);
dbContext.SaveChanges();

And to remove a relationship preserving product and category:

dbContext.Remove(categorization);
dbContext.SaveChanges();
@mjrousos

This comment has been minimized.

Copy link
Member

mjrousos commented Nov 19, 2015

I'll add a +1 for wanting many-to-many relationships without CLR objects for the join table. I have a website using EF6 that I'm trying to move forward to EF Core and it would save some hassle changing the model on the CLR side to be able to use the direct many-to-many modeling option that EF6 had.

@HerbertBodner

This comment has been minimized.

Copy link

HerbertBodner commented Dec 25, 2015

+1 for wanting "many-to-many relationships without CLR objects for the join table" from my side as well.

@sthiakos

This comment has been minimized.

Copy link

sthiakos commented Dec 25, 2015

What's interesting is when a Many-To-Many relationship has additional attributes/columns (e.g. DateOfAssociation, Priority, etc.) it becomes an observed entity. I can see that consistency would drive towards creating an intermediate Entity and I'm comfortable with the current implementation.

With that, requiring that we implement an intermediate Many-To-Many entity forces our object model to conform to database implementation, whereas hiding this detail leaves it in the hands of the ORM; not requiring an explicit intermediate makes the relationship seem more natural and represents the majority of Many-To-Many relationships.

@sven-n

This comment has been minimized.

Copy link

sven-n commented Jan 8, 2016

@sthiakos Well, it depends. Most of the data models and their relations I saw, do not need additional attributes/columns. In this cases the additional entity classes would just pollute the data model with unnecessary bloat. And depending on the size of a project, upgrading from EF6 could be a tremendously big task.

So I'm also voting for a solution without additional entity classes, and looking forward for the shadow state entities.

@bragma

This comment has been minimized.

Copy link

bragma commented Jan 12, 2016

i'm not sure I understand how this feature request would work. Do you mean that no entity class is required but a table is still created and managed automatically by Entity?
I'm asking because after some initial confusion in creating navigations, I'd say that handling many-to-many mapping entities manually is not that bad...

@raffaeler

This comment has been minimized.

Copy link

raffaeler commented Nov 28, 2018

@chaim1221 this can work if you are creating a new app.
If instead you are migrating an existing one where the model is public and cannot be changed, you are out of business.

Before or later this feature will need to be implemented. Given the length of this thread and the comments I am getting in conferences as a speaker I can say that the priority should be higher.

@chaim1221

This comment has been minimized.

Copy link

chaim1221 commented Nov 28, 2018

I hope I am not "out of business" because that is exactly what I am doing. I may have to go back and make the join tables as a separate migration in my MVC 4.5.1 project before I can apply the new schema, but I don't think it should choke too much after that?

Now you've given me pause. I'll post my results.

@ajcvickers

This comment has been minimized.

Copy link
Member

ajcvickers commented Nov 28, 2018

Just to chime in here on a couple of things. First, we have had a plan for many-to-many from the beginning--see this comment: #1368 (comment). So hopefully we won't need to re-architecture much. However, many-to-many is a high-level behavior that builds on lower-level blocks, and it is these lower level blocks that we have been making progress towards.

Second, we generally consider this and TPT to be the highest priority big general-purpose features that are not yet to implemented. That being said, there are things like a robust and rich query pipeline where we still need to spend significant resources, there are some breaking changes we need to make in 3.0 since its a major version bump, and there are also strategic plays like the Cosmos provider which are a priority.

@nmocruz

This comment has been minimized.

Copy link

nmocruz commented Dec 7, 2018

There's any estimate for this? before end 2Q 2019 ??

@vibeeshan025

This comment has been minimized.

Copy link

vibeeshan025 commented Dec 27, 2018

Will it get released with EF Core 3.0

@VinnieThePooh

This comment has been minimized.

Copy link

VinnieThePooh commented Jan 8, 2019

January 2019 is outside. Any estimates for this feature?
It would be nice to configure seamlessly (or almost) many-to-many for the same table.

@bricelam

This comment has been minimized.

Copy link
Member

bricelam commented Jan 9, 2019

Support for property bag entities (listed on the 3.0 roadmap) is a stepping stone toward many-to-many support. Property bag entities will replace what EF6 called an independent associations in the state manager.

@popcatalin81

This comment has been minimized.

Copy link

popcatalin81 commented Jan 11, 2019

property bag entities (listed on the 3.0 roadmap) is a stepping stone toward many-to-many support

@bricelam Are you pulling our leg? You can easily add the many to many entities as shadow state, without property bags. Shadow state works with virtual getters and setters does not need actual properties on a CLR class, so you already have your bags ...

@bfsmithATL

This comment has been minimized.

Copy link

bfsmithATL commented Jan 11, 2019

So, this paragraph is in the latest edition of MSDN Magazine (see below.) In the context of this very old thread, is this the way the only way we can get many-many relationships without a join class? Worse, does this mean we should not bother with EFCore at all, since the fully featured & very mature EF6 is being ported to run on .NET Core?
renderedimage

@Cito

This comment has been minimized.

Copy link

Cito commented Jan 11, 2019

This is also online here: What's Coming in .NET Core 3.0

@bricelam

This comment has been minimized.

Copy link
Member

bricelam commented Jan 11, 2019

@popcatalin81 I too fail to see the difference between shadow entities and property-bag entities, but I’ve been told they’re different and better. 🙃

@ajcvickers

This comment has been minimized.

Copy link
Member

ajcvickers commented Jan 11, 2019

@bfsmithATL If EF6 has the features and characteristics that you want, then waiting for it is a reasonable option. But keep in mind that EF6 will not get any significant new features.

@ajcvickers

This comment has been minimized.

Copy link
Member

ajcvickers commented Jan 11, 2019

@bricelam @popcatalin81 EF Core currently supports shadow properties, but they must be on an entity type represented by a unique CLR type. So each join table could be represented by a CLR type (e.g. ProductFeatures, ProductCategories, and so on) which then had shadow properties. This is really no different from the current pattern for many-to-many since it still requires a CLR type for each join table.

What we originally planned here was to have an entity type not backed by any CLR type. This is the shadow entity types feature, which is not currently implemented (except in a very limited form for the model snapshot.)

However, really what we need is an entity type that is not backed by any unique CLR type. That is, if we can re-use the same CLR type for all join tables, then that would work and be easier to implement and manage. An obvious candidate for this is Dictionary<string, object>, but this really reduces down to any CLR type with a string to object indexer property. This is a general feature that we refer to as property bag entity types.

Those observing closely might also notice that weak entity types, which are currently only used for owned entity types with multiple owners, is essentially "an entity type that is not backed by any unique CLR type." So property bag entity types are a matter of enabling indexer properties (which is not hard) and an expansion of weak types to cover more general cases (which is quite hard).

After this, the other big missing piece for many-to-many is skip-level navigation properties. That is, the ability to have a navigation property pointing from one entity type to another even when they are not directly related, but instead are only related through some other middle entity--in this case the join table mapping.

@raffaeler

This comment has been minimized.

Copy link

raffaeler commented Jan 11, 2019

@ajcvickers I am interested in all the details about the way you will implement this feature.

Could you please expand this point please?

After this, the other big missing piece for many-to-many is skip-level navigation properties. That is, the ability to have a navigation property pointing from one entity type to another even when they are not directly related, but instead are only related through some other middle entity--in this case the join table mapping.

@bfsmithATL

This comment has been minimized.

Copy link

bfsmithATL commented Jan 11, 2019

@ajcvickers Thx for the info, I sincerely appreciate it. One question for you (and the community), in your statement:

But keep in mind that EF6 will not get any significant new features.

Do you mean that the ported version EF6 will no longer receive new updates/features (e.g. the port won't be maintained in parallel to EF6 moving forward), or that EF6 is going to be sunsetted in favor of future development of EF Core?

@ajcvickers

This comment has been minimized.

Copy link
Member

ajcvickers commented Jan 11, 2019

@raffaeler For example, taking the example at the top of this issue:

class Product
{
    public int Id { get; set; }
    public ICollection<Categorization> Categorizations { get; set; }
}

class Categorization
{
    public int ProductId { get; set; }
    public Product Product { get; set; }

    public int CategoryId { get; set; }
    public Category Category { get; set; }
}

class Category
{
    public int Id { get; set; }
    public ICollection<Categorization> Categorizations { get; set; }
}

That's what works now. However, to avoid this dependency on the join table, we need something like:

class Product
{
    public int Id { get; set; }
    public ICollection<Category> Categories { get; set; }
}

class Categorization
{
    public int ProductId { get; set; }
    public int CategoryId { get; set; }
}

class Category
{
    public int Id { get; set; }
    public ICollection<Product> Products { get; set; }
}

That is, the Categories navigation property to refers directly to the Category entity (and vice versa), even though there is no direct relationship between the two. In other words, it is "skipping over" the join table entity.

How this is implemented is not decided. Possibly by re-writing to a chain of shadow navigation properties defined by convention on the underlying entities.

@ajcvickers

This comment has been minimized.

Copy link
Member

ajcvickers commented Jan 11, 2019

@bfsmithATL Yes. That is already the case, and has been for the last 4 or 5 years.

@ajcvickers

This comment has been minimized.

Copy link
Member

ajcvickers commented Jan 11, 2019

@ajcvickers

This comment has been minimized.

Copy link
Member

ajcvickers commented Jan 11, 2019

@bfsmithATL Sorry for the spam, but I realized I only answered part of your question. The .NET Core EF6 will continue in parallel to .NET Framework support. It will be the same NuGet package(s). See aspnet/EntityFramework6#271

@bfsmithATL

This comment has been minimized.

Copy link

bfsmithATL commented Jan 11, 2019

@ajcvickers No problem. I see your point, much more activity in the EF Core repo than EF6, but is that a function of EF6 being more mature (less activity), whereas EF Core is up & coming? I'm not trying to split hairs here, but really just want to be sure I pick the library that will have the most longevity. So with that in mind, I'll stay with EF Core and keep gritting my teeth with waiting on the true many-many support :-)

@raffaeler

This comment has been minimized.

Copy link

raffaeler commented Jan 11, 2019

@ajcvickers I am aware of how they are currently implemented as I already developed a 'workaround' consisting in a custom collection redirecting the many to many through a non-public intermediate entity, and an expression visitor rewriting the member access + call (include, theninclude) to redirect the queries.
Reason for this: can't change the public model
Reason for not being happy: it is very invasive and is hard to put in production (too long to explain).

I am asking because I would like to know if there can be any other way to workaround the custom collection. Again the need is to avoid modifying the current public model which doesn't have the intermediate entity. Time is a huge constraint and I need to run.

@ajcvickers

This comment has been minimized.

Copy link
Member

ajcvickers commented Jan 11, 2019

@bfsmithATL It's not due to that. EF6 is not receiving any significant investment. EF Core currently has four engineers spending the majority of their time on it. In general, EF6 doesn't have any dedicated engineering resources. One EF Core engineer is currently spending time away from EF Core to port EF6 to .NET Core. Once that is done, they will be back on EF Core.

@ajcvickers

This comment has been minimized.

Copy link
Member

ajcvickers commented Jan 11, 2019

@raffaeler The skip-level navigation properties will not look any different from other navigation properties to the application. They do not need a custom implementation of the collection. The difference is in how EF interacts with them.

@raffaeler

This comment has been minimized.

Copy link

raffaeler commented Jan 12, 2019

@ajcvickers afaik there are no skip-level navigation properties now. Is there any workaround for my custom collection that I can implement in 2.1? (2.1 is important because it is an LTS)

@ajcvickers

This comment has been minimized.

Copy link
Member

ajcvickers commented Jan 12, 2019

@raffaeler Everything I know about workarounds is in this thread, or in my blog posts referenced from this thread.

@DM2489

This comment has been minimized.

Copy link

DM2489 commented Jan 17, 2019

I'm really confused by this. With mapping the joining table, don't we end up with a list of relationships that then contain a list of the mapped object's we're after?

In essence, an extra layer is added...

@raffaeler

This comment has been minimized.

Copy link

raffaeler commented Jan 17, 2019

@DM2489 If you can change the model by adding the intermediate entity and "force" your users to express the queries in terms of the intermeditate entities, the current implementation works.
In all the other cases you just have a broken toy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment