Flexible mapping to CLR types and members (Custom O/C Mapping) #240

rowanmiller · 2014-05-22T20:45:09Z

This is a grouping of related issues. Feel free to vote (👍) for this issue to indicate that this is an area that you think we should spend time on, but consider also voting for individual issues for things you consider especially important.

This an "epic" issue for the theme of improvements to mapping of different CLR constructs to standard database concepts. Specific pieces of work will be tracked by linked issues.

Done in 7.0

Done in 8.0

Planned for 9.0

Use EF8-style primitive collections in the Cosmos provider #25364

Backlog

Although we have had support for POCOs through a few major versions, using EF for persistence still places some undesired constraints on the design choices developers can make on their domain objects.

For example, ~~EF Core still requires that mapped CLR types can be instantiated through constructors without parameters~~ (this is only true for navigation properties now), that they contain both property getters and setters for each mapped scalar and reference property, and that all mapped collection navigation properties are exposed as properties that are of a type that implements ICollection<T>.

Note that POCO proxy functionality introduces additional constraints, e.g. lazy loading requires that property getters are virtual and change tracking proxies (not yet supported in EF Core) require that all properties are virtual at the same time that properties mapped to collections are effectively declared as ICollection<T>.

EF Core also requires each entity type in the model to be mapped to a distinct CLR type, which makes some dynamic model scenarios (#2282) harder.

Moreover, scalar properties on objects have to be of a small set of recognized types in order to be mapped (in EF Core the set of types supported nativly by the provider).

Richer constructs such as collections of scalars or collections of complex types, ordered collections, inheritance in complex types, collection manipulation methods, factory methods, and immutable objects are not supported.

This issue tracks the removal of those constrains as a whole and serves as a parent issue for some individual features that we will track independently:

An old hope

One of many interesting design strategies of EF as an O/RMs was that every EF model defined an abstract data model that is separate from the object model. When using EF in regular O/RM scenarios there are in fact three layered models involved: the storage/database model (often referred to as the S-space model), the conceptual/abstract data model (often referred to as the C-space) and the actual object model (also known as the O-space model). EF also makes use of two mapping specifications: the translation of data access operations between the C-space model and the S-space model is the most well-known and where most of the mapping capabilities of EF focus. The mapping between the C-space and the O-space model is less widely known and in fact since only trivial 1:1 mappings are supported.

There are many historical motivations for this underlying design, including the desire to provide a complete “weakly typed“ programming interface (often referred to as a “value layer”) that could be used in scenarios outside the traditional O/RM space, such as runtime metadata driven user interfaces and tools, database reporting, etc. EntityClient was intended to be such weakly typed API, but it had considerable restrictions (e.g. it was read-only) and some usability issues (e.g. it was designed after an ADO.NET provider without a public DbDataAdapter and with a quite complicated connection string format) on the first version of EF. Since its popularity never took off, the investments necessary to unleash the “value layer” were never completed. In the meanwhile, EF role in solving the traditional O/RM scenario of mapping strongly typed objects to database tables and functions became more and more relevant since EF4, so the lion share of the investments have gone into making EF easier to use in those scenarios, which in many cases meant burying the separation between the abstract data model and the object model even deeper under layers of new API.

In practice the object model can be considered for most intents and purposes a straight realization of what is in C-space. The most commonly used APIs on EF take advantage of this similarity and just conflates C-space and O-space seamlessly, abstracting away the fact that there is a layer of indirection. In other words, although this differentiation between C-space and O-space is real and very explicit in the implementation, it is purposely not evident in most of the EF APIs and usage patterns. A clear example of this is Code First. To a great degree, the usability of the Code First API relies on users being able to ignore the existence of an abstract data model.

While sharing the same meta-model for conceptual models has made it easier to expose EF models through things like OData services, the majority of EF users has never needed to know that there is a difference between the conceptual and object models of their applications.

Regardless of the past motivations for creating this extra layer of indirection, there is potential in the decoupling of the conceptual level (at which most of EF’s mapping, query and update pipeline operates) and the objects level. The basic idea is that by breaking the current status quo of 1:1 O-C mapping, it should be possible to extend EF to support much richer object mapping without having to touch at all the guts of the system.

The text was updated successfully, but these errors were encountered:

rowanmiller · 2014-11-10T18:59:20Z

Here is a good example of this feature (good case to verify against once we implement it) - #1009

divega · 2015-09-01T18:39:26Z

@rowanmiller Looking at this it seems that the bug I filed yesterday is more specific about property access while this is the all-encompassing flexible mapping. Would you mind if I just add #2968 as a child, similar to how #857 is a child specific to collection patterns?

rowanmiller · 2015-09-01T18:40:30Z

@divega yep that works

divega · 2015-09-01T18:42:05Z

Otherwise I can merge it all here...

divega · 2015-09-01T21:01:44Z

Made some edits to the original text of the issue to add background and additional links to related issues.

biqas · 2016-03-08T20:35:20Z

Hi,
the following example is for Illustration how maybe interface definition could be used to map EF structures.

public class BloggingContext : DbContext
{
    // Type mapping for materialization
    public DbSet<IBlog, Blog> Blogs { get; set; }

    // Materialization can create type on the fly because no specific type was given. (Generators or emitting)
    public DbSet<IPost> Posts { get; set; }

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<IBlog>()
            .Property(b => b.Url);

        // Optional: provide type creation factory.
        modelBuilder.Entity<IBlog>(() => new Blog());

        // Optional: provide collection behavior if non is provided then use default or emitt or generate.
        modelBuilder.Collection<IList<Any>>((IList<Any> collection, Any item) => collection.Add(item));
    }
}

// Any represents all kind of types which can be used in a collection.
class Any { }

public class DbSet<T, TEntity> where TEntity : class, T { }

public interface IBlog
{
    int BlogId { get; set; }

    string Url { get; set; }

    IList<IPost> Posts { get; set; }
}

public class Blog : IBlog
{
    public int BlogId { get; set; }

    public string Url { get; set; }

    public IList<IPost> Posts { get; set; }
}

public interface IPost
{
    int PostId { get; set; }

    string Title { get; set; }

    string Content { get; set; }

    int BlogId { get; set; }

    IBlog Blog { get; set; }
}

public class Post : IPost
{
    public int PostId { get; set; }

    public string Title { get; set; }

    public string Content { get; set; }

    public int BlogId { get; set; }

    public IBlog Blog { get; set; }
}

I know there are lot of other scenarios, so if someone has specific questions how for example x,y and z would fit in this kind of mapping, please ask, I will try to provide examples and explanations.

bjorn-ali-goransson · 2016-07-30T23:35:07Z

Actually, todays code base is quite close to being able to handle this. It just needs to be 'laxed in some parts. I mean the IClrPropertyGetter/IClrPropertySetter infrastructure is already there.

I got stuck at the type check in EntityType.AddProperty, I dunno if it would have worked otherwise (probably not...)

We need to be able to specify, when adding a Property:

Clr type (not needed in practice as PropertyInfo.SetValue takes object)
Resulting primitive backing type (for DB provider)
~~Getter~~ Serializer (converts from Clr type to Resulting primitive type)
~~Setter~~ Parser (converts from Resulting primitive type to Clr type)

And the type check mentioned above needs to not happen if(Flexible).

Here's a (hopefully somewhat) functioning example of what I'd like to achieve:

public class Context : DbContext
{
    private static readonly IServiceProvider _serviceProvider = new ServiceCollection()
        .AddEntityFrameworkSqlServer()
        .AddSingleton<ICoreConventionSetBuilder, MyCoreConventionSetBuilder>()
        .BuildServiceProvider();

    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
    {
        optionsBuilder
            .UseInternalServiceProvider(_serviceProvider)
            .UseSqlServer(@"Data Source=.\SQLEXPRESS2014;Initial Catalog=EfCoreTest;Integrated Security=True;");
    }

    private class MyCoreConventionSetBuilder : CoreConventionSetBuilder
    {
        public override ConventionSet CreateConventionSet()
        {
            var value = base.CreateConventionSet();
            value.EntityTypeAddedConventions.Add(new MyPropertyDiscoveryConvention());
            return value;
        }
    }

    private class MyPropertyDiscoveryConvention : IEntityTypeConvention
    {
        public InternalEntityTypeBuilder Apply(InternalEntityTypeBuilder entityTypeBuilder)
        {
            entityTypeBuilder.Metadata.AddProperty(
                "Ancestors",
                propertyType: typeof(string),
                flexible: true,
                serialize: JsonConvert.Serialize,
                parse: value => JsonConvert.Parse<List<int>>((string)value)
            );

            return entityTypeBuilder;
        }
    }

    public DbSet<Page> Pages { get; set; }

    public class Page
    {
        public int Id { get; set; }
        public List<int> Ancestors { get; set; }
    }
}

bjorn-ali-goransson · 2016-08-05T13:34:47Z

@rowanmiller any thoughts or progress on this? I really think that just relaxing the checks a bit on this (and maybe generalizing the structure of Property a little) would get us 80% there.

( @divega )

Also, FWIW, I now think that we would be better off with a new AddFlexibleProperty method rather than further overloading the AddProperty.

Parts of issues #3342, #240, #10509, #3797 The main things here are: - Support for injecting values into parameterized entity constructors - Property values are injected if the parameter type and name matches - The current DbContext as DbContext or a derived DbContext type - A service from the internal or external service provider - A delegate to a method of a service - The IEntityType for the entity - Use of the above to inject lazy loading capabilities into entities For lazy loading, either the ILazyLoader service can be injected directly, or a delegate can be injected if the entity class cannot take a dependency on the EF assembly--see the examples below. Currently all constructor injection is done by convention. Remaining work includes: - API/attributes to configure the constructor binding - Allow factory to be used instead of using the constructor directly. (Functional already, but no API or convention to configure it.) - Allow property injection for services - Configuration of which entities/properties should be lazy loaded and which should not ### Examples In this example EF will use the private constructor passing in values from the database when creating entity instances. (Note that it is assumed that _blogId has been configured as the key.) ```C# public class Blog { private int _blogId; // This constructor used by EF Core private Blog( int blogId, string title, int? monthlyRevenue) { _blogId = blogId; Title = title; MonthlyRevenue = monthlyRevenue; } public Blog( string title, int? monthlyRevenue = null) : this(0, title, monthlyRevenue) { } public string Title { get; } public int? MonthlyRevenue { get; set; } } ``` In this example, EF will inject the ILazyLoader instance, which is then used to enable lazy-loading on navigation properties. Note that the navigation properties must have backing fields and all access by EF will go through the backing fields to prevent EF triggering lazy loading itself. ```C# public class LazyBlog { private readonly ILazyLoader _loader; private ICollection<LazyPost> _lazyPosts = new List<LazyPost>(); public LazyBlog() { } private LazyBlog(ILazyLoader loader) { _loader = loader; } public int Id { get; set; } public ICollection<LazyPost> LazyPosts => _loader.Load(this, ref _lazyPosts); } public class LazyPost { private readonly ILazyLoader _loader; private LazyBlog _lazyBlog; public LazyPost() { } private LazyPost(ILazyLoader loader) { _loader = loader; } public int Id { get; set; } public LazyBlog LazyBlog { get => _loader.Load(this, ref _lazyBlog); set => _lazyBlog = value; } } ``` This example is the same as the last example, except EF is matching the delegate type and parameter name and injecting a delegate for the ILazyLoader.Load method so that the entity class does not need to reference the EF assembly. A small extension method can be included in the entity assembly to make it a bit easier to use the delegate. ```C# public class LazyPocoBlog { private readonly Action<object, string> _loader; private ICollection<LazyPocoPost> _lazyPocoPosts = new List<LazyPocoPost>(); public LazyPocoBlog() { } private LazyPocoBlog(Action<object, string> lazyLoader) { _loader = lazyLoader; } public int Id { get; set; } public ICollection<LazyPocoPost> LazyPocoPosts => _loader.Load(this, ref _lazyPocoPosts); } public class LazyPocoPost { private readonly Action<object, string> _loader; private LazyPocoBlog _lazyPocoBlog; public LazyPocoPost() { } private LazyPocoPost(Action<object, string> lazyLoader) { _loader = lazyLoader; } public int Id { get; set; } public LazyPocoBlog LazyPocoBlog { get => _loader.Load(this, ref _lazyPocoBlog); set => _lazyPocoBlog = value; } } public static class TestPocoLoadingExtensions { public static TRelated Load<TRelated>( this Action<object, string> loader, object entity, ref TRelated navigationField, [CallerMemberName] string navigationName = null) where TRelated : class { loader?.Invoke(entity, navigationName); return navigationField; } } ```

smarts · 2019-03-08T22:00:18Z

Which one of the issues in this issue's description handles support for unsigned integer types?

ajcvickers · 2019-03-08T22:20:52Z

@smarts Unsigned integer types are supported as of EF Core 2.1.

smarts · 2019-03-10T21:44:01Z

@ajcvickers is there documentation on how to use this feature? My team recently tried to use them and was getting an exception that EF Core was expecting Int32 instead of UInt32. Do you know if this feature works for shadow properties too? Also, does it work for any DB integer type?
Our specific use-cases are SQL Server and PostgreSQL

ajcvickers · 2019-03-11T19:40:59Z

@smarts Some databases natively support storing unsigned types, in which case they should behave like any other property. Some databases, like SQL Server, don't supported unsigned types, but EF should automatically perform a value conversion if you have such a type in your model. In this case the value is stored in the database as a bigger signed type, by default.

If you have an entity type with unsigned properties and you are seeing errors, then please file a new issue with a small, runnable project/solution of complete code listing that demonstrates the behavior you are seeing.

smarts · 2019-03-28T18:01:51Z

@ajcvickers sorry for the delay in responding.

In this case the value is stored in the database as a bigger signed type, by default.

Does that mean that using UInt32 for an entity's property will only work if the DB type for the corresponding column supports the full range of values for UInt32? I.e., we want to use INT in the DB and UInt32 on the entity.
While I understand that this conversion is generally bad, in the specific case of [generated] IDs the risk becomes negligible. Admittedly, this is based on my minimal experience with SQL DBs, so I apologize if this assumption is incorrect.
Is it possible to allow this generally-risky conversion for the negligible-risk scenario of IDs via some custom EF Convention?

roji · 2019-03-29T13:43:16Z

@smarts take a look at value converters - you can set up lossy value conversions, but EF will not that automatically for you (as it does with uint->long). However, lossy value conversions may get a bit tricky in some scenarios, and are best avoided if you have an easy lossless alternative (such as long).

markusschaber · 2019-03-29T14:24:32Z

@roji As far as I can see, conversion between UInt32 and Int32 is lossless if you use "C-Style" casts. Negative int32 values are mapped to "high" unsigned values and back, one just should be careful when interpreting the converted values (e. G. ordering is different).
SQL Server, PostgreSQL and MySQL provide "int" as a 32 bit signed data type, so trying to store an uint32 there via C-Style cast looks feasible. (I didn't check other DBs).

roji · 2019-03-29T14:36:06Z

@markusschaber you're technically right: if your value converter does a simple cast in C#, then uint values which are larger than int.MaxValue will show up as negatives. While this is technically lossless and would work, as you say there are some tricky caveats.

Given that databases typically have a larger bigint type which can contain all uint values, it generally really makes sense to convert to that - trying to save 4 bytes isn't going to be justifiable in most cases.

smarts · 2019-03-29T19:49:23Z

@markusschaber i'm not sure if you noticed, but i'm specifically talking about auto-generated IDs in the database. Using long for the entity's property type has the same problem (allowing negative values), which isn't what I want. Also, as far as large values… since the value only ever comes from the DB (i.e., not set by any C# code outside of EF Core reading DB values), it will never be larger than Int32.MaxValue (because INT is the DB type).

ajcvickers · 2019-03-29T20:48:42Z

Just to close the loop on this. When deciding how to convert automatically, we look at:

Is is lossy?
Does it preserve semantics on the database--most notably ordering?

The second point is why we convert uint to long, and not uint to int, since for the latter, if the uint does have a value that can't "fit" it will still be stored, but it will be interpreted as negative by the database and hence will have a different ordering.

That being said, it's safe to convert uint to int if you know that the domain space for your uints can fit in an int, or if a specific ordering on the database is not needed. One way to do it is:

modelBuilder.Entity<Foo>().Property(e => e.MyUInt).HasConversion<int>();

Finally, there are currently some limitations for conversion on keys: see #11597

smitpatel · 2019-09-03T16:44:46Z

@ajcvickers - What exactly needs-cleanup label is?

voroninp · 2020-10-14T19:45:26Z

Proposed for 6.0

Does it mean not earlier than November 2021?

roji · 2020-10-15T06:14:24Z

@voroninp yes, 6.0 would mean November 2021. Previews would make this available much earlier, though.

atrauzzi · 2020-10-31T04:27:53Z

I'm just here to say I've been waiting 6 years and counting for polymorphic relations using discriminators. I think it's more than made a case for itself from the digging I've been doing here.

Something along the lines of:

Integration
IIntegratable
User : IIIntegratable
Organization : IIntegratable

Therefore, the table integration in the database gets a integratable_type and integratable_id column, then I could go:

integration.Integratable...; // (returns a IIntegratable that can be passed through pattern matching to detect concrete type)
// and of course:
user.Integrations...; // (returns a list of `Integration` instances)

rowanmiller added this to the 1.0.0 milestone May 22, 2014

rowanmiller mentioned this issue Oct 22, 2014

ICollection for related entities is too permissive. #857

Closed

rowanmiller mentioned this issue Nov 10, 2014

Extensibility for custom date and time value types #1009

Closed

rowanmiller modified the milestones: Backlog, 7.0.0 Dec 1, 2014

rowanmiller added the type-enhancement label Dec 1, 2014

rowanmiller mentioned this issue Jul 14, 2015

Roadmap Questions #2566

Closed

rowanmiller mentioned this issue Sep 1, 2015

Support for Extension Properties #2955

Closed

divega changed the title ~~Flexible mapping to CLR types/properties/methods/etc. (O/C Mapping)~~ Flexible object mapping to CLR types and members (Custom O/C Mapping) Sep 1, 2015

divega changed the title ~~Flexible object mapping to CLR types and members (Custom O/C Mapping)~~ Flexible mapping to CLR types and members (Custom O/C Mapping) Sep 1, 2015

anpete mentioned this issue Sep 9, 2015

The entity type 'System.Uri' requires a key to be defined #3062

Closed

rowanmiller mentioned this issue Sep 28, 2015

Can I use EF Core with immutable object? #3129

Closed

divega mentioned this issue Oct 8, 2015

Support DI for contructors of POCO #3342

Open

rowanmiller mentioned this issue Dec 2, 2015

Support materialization hook #3943

Closed

rowanmiller mentioned this issue Dec 28, 2015

The impedance mismatch between EF and domain models: what is Entity Framework's answer? #4185

Closed

rowanmiller mentioned this issue Jan 19, 2016

Type converters #4333

Closed

rowanmiller mentioned this issue Feb 29, 2016

TPH does not support immutable discriminator properties #4650

Closed

rowanmiller mentioned this issue Mar 8, 2016

Passing in QueryModel from re-linq #4417

Closed

rowanmiller mentioned this issue Mar 25, 2016

DateTime.Kind comes back as Unspecified #4711

Closed

rowanmiller mentioned this issue Apr 25, 2016

Question: Are non-parameterless constructors and initializers supported by this latest .net core version from EF? #5124

Closed

rowanmiller mentioned this issue Jul 15, 2016

Add-Migration error: The key {'TempId'} contains properties in shadow state and is referenced by a relationship #5984

Closed

soycabanillas mentioned this issue Jul 31, 2016

Map JSON values stored in database to EF properties #4021

Closed

1 task

ajcvickers added the needs-cleanup label Aug 30, 2019

ajcvickers mentioned this issue Oct 10, 2019

Developers can customize type mapping #18306

Closed

ajcvickers mentioned this issue Mar 15, 2020

EF Pain Points / Feature Discussion #20249

Closed

ajcvickers mentioned this issue Sep 11, 2020

Support Maybe patterns for navigations #22504

Open

ajcvickers modified the milestones: Backlog, Epics Oct 11, 2020

ajcvickers added the area-o/c-mapping label Oct 11, 2020

atrauzzi mentioned this issue Nov 1, 2020

Support Polymorphic Relationships #7623

Open

AndriySvyryd removed the needs-cleanup label Jan 26, 2022

roji mentioned this issue May 16, 2022

Navigation property between two models with "columns in common" without a join/pivot table #27979

Closed

ajcvickers added the composite-issue A grouping of multiple related issues into one issue label Dec 6, 2022

ajcvickers modified the milestones: Epics, Backlog Dec 7, 2022

roji mentioned this issue Feb 10, 2024

Ephemeral/Shadow navigation properties or other mechanism of automatic joins. #32658

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flexible mapping to CLR types and members (Custom O/C Mapping) #240

Flexible mapping to CLR types and members (Custom O/C Mapping) #240

rowanmiller commented May 22, 2014 •

edited by AndriySvyryd

rowanmiller commented Nov 10, 2014

divega commented Sep 1, 2015

rowanmiller commented Sep 1, 2015

divega commented Sep 1, 2015

divega commented Sep 1, 2015

biqas commented Mar 8, 2016

bjorn-ali-goransson commented Jul 30, 2016 •

edited

bjorn-ali-goransson commented Aug 5, 2016

smarts commented Mar 8, 2019

ajcvickers commented Mar 8, 2019

smarts commented Mar 10, 2019

ajcvickers commented Mar 11, 2019

smarts commented Mar 28, 2019 •

edited

roji commented Mar 29, 2019

markusschaber commented Mar 29, 2019 •

edited

roji commented Mar 29, 2019

smarts commented Mar 29, 2019

ajcvickers commented Mar 29, 2019

smitpatel commented Sep 3, 2019

voroninp commented Oct 14, 2020

roji commented Oct 15, 2020

atrauzzi commented Oct 31, 2020

Flexible mapping to CLR types and members (Custom O/C Mapping) #240

Flexible mapping to CLR types and members (Custom O/C Mapping) #240

Comments

rowanmiller commented May 22, 2014 • edited by AndriySvyryd

Done in 7.0

Done in 8.0

Planned for 9.0

Backlog

An old hope

rowanmiller commented Nov 10, 2014

divega commented Sep 1, 2015

rowanmiller commented Sep 1, 2015

divega commented Sep 1, 2015

divega commented Sep 1, 2015

biqas commented Mar 8, 2016

bjorn-ali-goransson commented Jul 30, 2016 • edited

bjorn-ali-goransson commented Aug 5, 2016

smarts commented Mar 8, 2019

ajcvickers commented Mar 8, 2019

smarts commented Mar 10, 2019

ajcvickers commented Mar 11, 2019

smarts commented Mar 28, 2019 • edited

roji commented Mar 29, 2019

markusschaber commented Mar 29, 2019 • edited

roji commented Mar 29, 2019

smarts commented Mar 29, 2019

ajcvickers commented Mar 29, 2019

smitpatel commented Sep 3, 2019

voroninp commented Oct 14, 2020

roji commented Oct 15, 2020

atrauzzi commented Oct 31, 2020

rowanmiller commented May 22, 2014 •

edited by AndriySvyryd

bjorn-ali-goransson commented Jul 30, 2016 •

edited

smarts commented Mar 28, 2019 •

edited

markusschaber commented Mar 29, 2019 •

edited