New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for custom type mapping and data store to CLR type conversions #242

Closed
rowanmiller opened this Issue May 22, 2014 · 62 comments

Comments

Projects
None yet
@rowanmiller
Member

rowanmiller commented May 22, 2014

There is a continuum of scenarios that can be supported here:

  • Allow for simple hard coded conversions between types that are related in a well-know manner to types that are supported. E.g.:
    • char can map to the database exactly as a string of size 1 (see #8656)
    • byte can map to the database exactly the same as a byte[]
    • signed or unsigned small integers can fit in the nearest wider signed or unsigned integer
  • Allow providers to supply their own additional type mappings for types they don't support, e.g. if a database engine doesn't have native support for bool, it can decide to use a small integer representation
  • Allow specific well known scenarios that are commonly demanded, like mapping enum types to strings
  • Allow for conversions to be performed on the server (vs. only on the client) for cases in which there isn't a viable CLR representation for the server type - see #10434 and #10861
  • Allow for full user customization - Not sure what this means

They all probably require extending the reach of the type mapper to be able to participate of the generation of:

  • Parameters
  • Expressions for value access in materialization
  • Expressions for equality and inequality expressions - see #10265
    1. Equality on the server should be equivalent to equality on the client as long as the conversion is deterministic and bijective.
  • Value comparisons and sorting - see #10265
    1. Comparisons and sorting could equivalent as long as the conversion is a monotonic function (although reversed if it is decreasing)
    2. For non order-preserving conversions, we may need to differentiate the scenarios in which we introduce sorting just to get the order of two or more results to be deterministic so we can zip them together, vs. explicit ordering requested by user, which we may need to evaluate on the client after the conversion is performed.
  • Operator and function translation - see #10434
@mj1856

This comment has been minimized.

Show comment
Hide comment
@mj1856

mj1856 Nov 7, 2014

Just checking - Is this the work item tracking this uservoice item? Thanks.

mj1856 commented Nov 7, 2014

Just checking - Is this the work item tracking this uservoice item? Thanks.

@mj1856

This comment has been minimized.

Show comment
Hide comment
@mj1856

mj1856 Jul 12, 2015

Any movement on this? Or hint of direction, proposed API, anything? Thanks.

mj1856 commented Jul 12, 2015

Any movement on this? Or hint of direction, proposed API, anything? Thanks.

@rowanmiller

This comment has been minimized.

Show comment
Hide comment
@rowanmiller

rowanmiller Jul 15, 2015

Member

@mj1856 nothing yet. We know we want to do this (and that has influenced how things are architected internally) but we're currently planning to work on lighting this feature up after our initial RTM of EF7.

Member

rowanmiller commented Jul 15, 2015

@mj1856 nothing yet. We know we want to do this (and that has influenced how things are architected internally) but we're currently planning to work on lighting this feature up after our initial RTM of EF7.

@mj1856

This comment has been minimized.

Show comment
Hide comment
@mj1856

mj1856 commented Jul 15, 2015

Thanks!

@smitpatel

This comment has been minimized.

Show comment
Hide comment
@smitpatel

smitpatel Sep 4, 2015

Contributor

Details:
At present we are making changes to our property discovery so that all the types supported by the provider can be mapped as primitive properties. In the addition to provider supported types, user may want to use some other type as a primitive by providing a custom typemapping through which it can be mapped to one of the known types of the provider and subsequently be used as a primitive property.
Also see #2588

Contributor

smitpatel commented Sep 4, 2015

Details:
At present we are making changes to our property discovery so that all the types supported by the provider can be mapped as primitive properties. In the addition to provider supported types, user may want to use some other type as a primitive by providing a custom typemapping through which it can be mapped to one of the known types of the provider and subsequently be used as a primitive property.
Also see #2588

@smitpatel smitpatel changed the title from Flexible data store type to CLR type conversions to Add support for custom typemapping for data store type to CLR type conversions Sep 4, 2015

@divega divega changed the title from Add support for custom typemapping for data store type to CLR type conversions to Add support for custom typemapping for data store type to CLR type conversions (aka custom TypeMapping) Sep 4, 2015

@divega divega changed the title from Add support for custom typemapping for data store type to CLR type conversions (aka custom TypeMapping) to Add support for custom typemapping for data store type to CLR type conversions Sep 4, 2015

@divega divega changed the title from Add support for custom typemapping for data store type to CLR type conversions to Add support for custom type mapping for data store to CLR type conversions Oct 1, 2015

@ilmax

This comment has been minimized.

Show comment
Hide comment
@ilmax

ilmax Oct 8, 2015

Contributor

Does this allow me to map a custom type to whatever filed in the storage? e.g. create a custom type for entity.Id (which only wraps an int) instead of using int or map enum to char?

Contributor

ilmax commented Oct 8, 2015

Does this allow me to map a custom type to whatever filed in the storage? e.g. create a custom type for entity.Id (which only wraps an int) instead of using int or map enum to char?

@Rseding91

This comment has been minimized.

Show comment
Hide comment
@Rseding91

Rseding91 Oct 27, 2015

This is exactly what I was recently looking to do. It would be an extremely useful to have this.

Rseding91 commented Oct 27, 2015

This is exactly what I was recently looking to do. It would be an extremely useful to have this.

@Marusyk

This comment has been minimized.

Show comment
Hide comment
@Marusyk

Marusyk Nov 21, 2015

Contributor

I need the same possibility as mentioned by @ilmax:

Does this allow me to map a custom type to whatever filed in the storage? e.g. create a custom type
for entity.Id (which only wraps an int) instead of using int or map enum to char?

Does this allow me to do that?

Contributor

Marusyk commented Nov 21, 2015

I need the same possibility as mentioned by @ilmax:

Does this allow me to map a custom type to whatever filed in the storage? e.g. create a custom type
for entity.Id (which only wraps an int) instead of using int or map enum to char?

Does this allow me to do that?

@ajcvickers

This comment has been minimized.

Show comment
Hide comment
@ajcvickers

ajcvickers Jan 24, 2018

Member

@TsengSR There is no built-in support for the EnumMember attribute at this time. However, you can write your own value converter. For example, using code for conversion from https://stackoverflow.com/questions/10418651/using-enummemberattribute-and-doing-automatic-string-conversions

public class MyEnumConverter<TEnum> : ValueConverter<TEnum, string>
    where TEnum : struct
{
    public static MyEnumConverter<TEnum> Instance = new MyEnumConverter<TEnum>();

    private MyEnumConverter()
        : base(v => ToEnumString(v), v => ToEnum(v))
    {
    }

    private static string ToEnumString(TEnum type)
    {
        var enumType = typeof(TEnum);
        var name = Enum.GetName(enumType, type);
        var enumMemberAttribute = ((EnumMemberAttribute[])enumType.GetField(name).GetCustomAttributes(typeof(EnumMemberAttribute), true)).Single();
        return enumMemberAttribute.Value;
    }

    private static TEnum ToEnum(string str)
    {
        var enumType = typeof(TEnum);
        foreach (var name in Enum.GetNames(enumType))
        {
            var enumMemberAttribute = ((EnumMemberAttribute[])enumType.GetField(name).GetCustomAttributes(typeof(EnumMemberAttribute), true)).Single();
            if (enumMemberAttribute.Value == str)
            {
                return (TEnum)Enum.Parse(enumType, name);
            }
        }

        return default;
    }
}
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    modelBuilder.Entity<Blogg>().Property(e => e.Foo)
        .HasConversion(MyEnumConverter<DocumentTypes>.Instance);
}
Member

ajcvickers commented Jan 24, 2018

@TsengSR There is no built-in support for the EnumMember attribute at this time. However, you can write your own value converter. For example, using code for conversion from https://stackoverflow.com/questions/10418651/using-enummemberattribute-and-doing-automatic-string-conversions

public class MyEnumConverter<TEnum> : ValueConverter<TEnum, string>
    where TEnum : struct
{
    public static MyEnumConverter<TEnum> Instance = new MyEnumConverter<TEnum>();

    private MyEnumConverter()
        : base(v => ToEnumString(v), v => ToEnum(v))
    {
    }

    private static string ToEnumString(TEnum type)
    {
        var enumType = typeof(TEnum);
        var name = Enum.GetName(enumType, type);
        var enumMemberAttribute = ((EnumMemberAttribute[])enumType.GetField(name).GetCustomAttributes(typeof(EnumMemberAttribute), true)).Single();
        return enumMemberAttribute.Value;
    }

    private static TEnum ToEnum(string str)
    {
        var enumType = typeof(TEnum);
        foreach (var name in Enum.GetNames(enumType))
        {
            var enumMemberAttribute = ((EnumMemberAttribute[])enumType.GetField(name).GetCustomAttributes(typeof(EnumMemberAttribute), true)).Single();
            if (enumMemberAttribute.Value == str)
            {
                return (TEnum)Enum.Parse(enumType, name);
            }
        }

        return default;
    }
}
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    modelBuilder.Entity<Blogg>().Property(e => e.Foo)
        .HasConversion(MyEnumConverter<DocumentTypes>.Instance);
}
@glucaci

This comment has been minimized.

Show comment
Hide comment
@glucaci

glucaci Jan 24, 2018

Contributor

@ajcvickers I'm suggesting that when using .HasConvertion() the target property will be automatically ignored.

By using only .HasConvertion()

modelBuilder.Entity<User>().Property(e => e.Email).HasConversion(EmailConvertor.Instance);

without .Ignore() it

modelBuilder.Entity<User>().Ignore(e => e.Email);

will end up with an exception. ...or am I wrong with this?

Contributor

glucaci commented Jan 24, 2018

@ajcvickers I'm suggesting that when using .HasConvertion() the target property will be automatically ignored.

By using only .HasConvertion()

modelBuilder.Entity<User>().Property(e => e.Email).HasConversion(EmailConvertor.Instance);

without .Ignore() it

modelBuilder.Entity<User>().Ignore(e => e.Email);

will end up with an exception. ...or am I wrong with this?

@ajcvickers

This comment has been minimized.

Show comment
Hide comment
@ajcvickers

ajcvickers Jan 24, 2018

Member

@glucaci I don't really understand why you would want to ignore a property that has a conversion. If the property is ignored, then what is the conversion for?

Member

ajcvickers commented Jan 24, 2018

@glucaci I don't really understand why you would want to ignore a property that has a conversion. If the property is ignored, then what is the conversion for?

@glucaci

This comment has been minimized.

Show comment
Hide comment
@glucaci

glucaci Jan 24, 2018

Contributor

Having the following model

public class User
{
   public User(Email email)
   {
       Id = Guid.NewGuid();
       Email = email;
   }

   public Guid Id { get; private set; }
   public Email Email { get; private set; }
}

public class Email
{
   private readonly string _value;
   private Email(string value)
   {
       _value = value;
   }

   public static Email Create(string value)
   {
       // some validation
       return new Email(value);
   }

   public static implicit operator string(Email email) => email._value;
}

and the configuration

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    modelBuilder.Entity<User>(builder =>
    {
        builder.HasKey(x => x.Id);
        builder.Property(x => x.Email).HasConversion(
            new ValueConverter<Email, string>(email => email, value => Email.Create(value)));
    });
}

it will throw
System.InvalidOperationException: 'Cannot call Property for the property 'Email' on entity type 'Customer' because it is configured as a navigation property. Property can only be used to configure scalar properties.'

EFCore : 2.1.0-preview1 (local build, the nuget package from nightly build is not finding some dependencies)

Contributor

glucaci commented Jan 24, 2018

Having the following model

public class User
{
   public User(Email email)
   {
       Id = Guid.NewGuid();
       Email = email;
   }

   public Guid Id { get; private set; }
   public Email Email { get; private set; }
}

public class Email
{
   private readonly string _value;
   private Email(string value)
   {
       _value = value;
   }

   public static Email Create(string value)
   {
       // some validation
       return new Email(value);
   }

   public static implicit operator string(Email email) => email._value;
}

and the configuration

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    modelBuilder.Entity<User>(builder =>
    {
        builder.HasKey(x => x.Id);
        builder.Property(x => x.Email).HasConversion(
            new ValueConverter<Email, string>(email => email, value => Email.Create(value)));
    });
}

it will throw
System.InvalidOperationException: 'Cannot call Property for the property 'Email' on entity type 'Customer' because it is configured as a navigation property. Property can only be used to configure scalar properties.'

EFCore : 2.1.0-preview1 (local build, the nuget package from nightly build is not finding some dependencies)

@ajcvickers

This comment has been minimized.

Show comment
Hide comment
@ajcvickers

ajcvickers Jan 25, 2018

Member

@glucaci Looks like a bug--can you file a new issue for it?

Member

ajcvickers commented Jan 25, 2018

@glucaci Looks like a bug--can you file a new issue for it?

@glucaci

This comment has been minimized.

Show comment
Hide comment
@glucaci

glucaci Jan 25, 2018

Contributor

Sure, I've just created one #10765

I don't know how the conversion should work internally but I realize that by ignoring the property everything is working (column created, saving to db, queering the db), but it seams that it's not the proper way 😏

Contributor

glucaci commented Jan 25, 2018

Sure, I've just created one #10765

I don't know how the conversion should work internally but I realize that by ignoring the property everything is working (column created, saving to db, queering the db), but it seams that it's not the proper way 😏

@ajcvickers

This comment has been minimized.

Show comment
Hide comment
@ajcvickers

ajcvickers Jan 26, 2018

Member

Note for triage: Additional work to be done here:

  • Allowing query translation to use type-mapping/conversion info.
  • Mechanism/API to specify a default conversion for any property of a given type in the model.
Member

ajcvickers commented Jan 26, 2018

Note for triage: Additional work to be done here:

  • Allowing query translation to use type-mapping/conversion info.
  • Mechanism/API to specify a default conversion for any property of a given type in the model.
@ajcvickers

This comment has been minimized.

Show comment
Hide comment
@ajcvickers

ajcvickers Jan 27, 2018

Member

Closing this as done for 2.1. Additional related work is tracked by #10784, #10434 #10265

Member

ajcvickers commented Jan 27, 2018

Closing this as done for 2.1. Additional related work is tracked by #10784, #10434 #10265

@tystol

This comment has been minimized.

Show comment
Hide comment
@tystol

tystol Jan 31, 2018

When this was first created back in 2014 (based on a uservoice suggestion from 2012) the intent was obviously for it to be a feature of legacy EF(ie.1-6). So while it is great that EFCore supports this, where does this leave the vast majority of us with existing apps on EF6 (that can't update until EFCore is at feature parity with EF6 - ie. GROUP BY!)

Is there any plans to backport this to EF6?

tystol commented Jan 31, 2018

When this was first created back in 2014 (based on a uservoice suggestion from 2012) the intent was obviously for it to be a feature of legacy EF(ie.1-6). So while it is great that EFCore supports this, where does this leave the vast majority of us with existing apps on EF6 (that can't update until EFCore is at feature parity with EF6 - ie. GROUP BY!)

Is there any plans to backport this to EF6?

@ajcvickers

This comment has been minimized.

Show comment
Hide comment
@ajcvickers

ajcvickers Jan 31, 2018

Member

@tystol This issue is not something that our team is planning to address in the EF6.x code base. This does not mean that we would not consider a community contribution to address this issue. However, the nature of the EF6 code and the EDM type system makes it non-trivial to implement. Being able to do things like this in a reasonable way is one of the reasons for EF Core being a new codebase that is not a front for EDM.

Member

ajcvickers commented Jan 31, 2018

@tystol This issue is not something that our team is planning to address in the EF6.x code base. This does not mean that we would not consider a community contribution to address this issue. However, the nature of the EF6 code and the EDM type system makes it non-trivial to implement. Being able to do things like this in a reasonable way is one of the reasons for EF Core being a new codebase that is not a front for EDM.

@essmd

This comment has been minimized.

Show comment
Hide comment
@essmd

essmd Apr 5, 2018

As already mentioned from @Marusyk and @ilmax:

I need the same possibility as mentioned by @ilmax:

Does this allow me to map a custom type to whatever filed in the storage? e.g. create a custom type
for entity.Id (which only wraps an int) instead of using int or map enum to char?

Does this allow me to do that?

I already tried ValueConverters and those are awesome features! But today i tried to use a custom type for Identities (primary and foreign keys). In DDD where you want to express your values with more context - an int (primary key and foreign key) is just an int and its value could be anything.

It would be so much value for EF if it would be possible to consider value conversations for primary keys and foreign keys. Is something like this planned or will this issue resolve parts of this?

public class CustomerId
{
    public int Value { get; }

    public CustomerId(int value)
    {
        if (value <= 0) throw new ArgumentOutOfRangeException(nameof(value));
        Value = value;
    }
}

public class Customer
{
    public CustomerId Id { get; private set; }
    public CustomerId ReferredById { get; private set; }
}

public class CustomerEntityConfiguration : IEntityTypeConfiguration<Customer>
{
    public void Configure(EntityTypeBuilder<Customer> builder)
    {
        builder.ToTable("Customers");
        builder.HasKey(x => x.Id);

        // works already fine and maps client type CustomerId to storage type int and reverse
        builder.Property(x => x.Id).HasConversation(id => id.Value, value => new CustomerId(value);

        // not possible (Exception) because "Id" must be an int to support value generation
        // but since there is an value conversation, it should use the storage type which is/results in an int?
        builder.Property(x => x.Id).UseSqlServerIdentityColumn();
    }
}

essmd commented Apr 5, 2018

As already mentioned from @Marusyk and @ilmax:

I need the same possibility as mentioned by @ilmax:

Does this allow me to map a custom type to whatever filed in the storage? e.g. create a custom type
for entity.Id (which only wraps an int) instead of using int or map enum to char?

Does this allow me to do that?

I already tried ValueConverters and those are awesome features! But today i tried to use a custom type for Identities (primary and foreign keys). In DDD where you want to express your values with more context - an int (primary key and foreign key) is just an int and its value could be anything.

It would be so much value for EF if it would be possible to consider value conversations for primary keys and foreign keys. Is something like this planned or will this issue resolve parts of this?

public class CustomerId
{
    public int Value { get; }

    public CustomerId(int value)
    {
        if (value <= 0) throw new ArgumentOutOfRangeException(nameof(value));
        Value = value;
    }
}

public class Customer
{
    public CustomerId Id { get; private set; }
    public CustomerId ReferredById { get; private set; }
}

public class CustomerEntityConfiguration : IEntityTypeConfiguration<Customer>
{
    public void Configure(EntityTypeBuilder<Customer> builder)
    {
        builder.ToTable("Customers");
        builder.HasKey(x => x.Id);

        // works already fine and maps client type CustomerId to storage type int and reverse
        builder.Property(x => x.Id).HasConversation(id => id.Value, value => new CustomerId(value);

        // not possible (Exception) because "Id" must be an int to support value generation
        // but since there is an value conversation, it should use the storage type which is/results in an int?
        builder.Property(x => x.Id).UseSqlServerIdentityColumn();
    }
}
@ajcvickers

This comment has been minimized.

Show comment
Hide comment
@ajcvickers

ajcvickers Apr 6, 2018

Member

@essmd To clarify, the only part of this that isn't working for you is that the key cannot be store-generated?

Member

ajcvickers commented Apr 6, 2018

@essmd To clarify, the only part of this that isn't working for you is that the key cannot be store-generated?

@essmd

This comment has been minimized.

Show comment
Hide comment
@essmd

essmd Apr 7, 2018

@ajcvickers Correct! As i can see here in the code, converters are not allowed.

My expectation from this setup was, that the add migration would create an migration with an INT IDENTITY column, not throwing an exception.

Later when SaveChangesAsync is called, EF executes the INSERT INTO sql query and get the identity/id back from store (INT) and when it comes to map the INT from store to the property, it will use the configured conversation.

I tested again with foreign key too, and its also not possible to map foreign keys with properties using custom types and configured converters:

public class Order
{
    public CustomerId CustomerId { get; private set; }
}

public class OrderEntityConfiguration : IEntityTypeConfiguration<Order>
{
    public void Configure(EntityTypeBuilder<Order> builder)
    {
        builder.ToTable("Orders");

        // works fine and maps client type CustomerId to storage type int and reverse
        builder.Property(x => x.CustomerId)
            .HasConversation(id => id.Value, value => new CustomerId(value);

        // not working
        builder.HasOne<Customer>()
            .WithMany()
            .HasForeignKey(x => x.CustomerId);
    }
}

essmd commented Apr 7, 2018

@ajcvickers Correct! As i can see here in the code, converters are not allowed.

My expectation from this setup was, that the add migration would create an migration with an INT IDENTITY column, not throwing an exception.

Later when SaveChangesAsync is called, EF executes the INSERT INTO sql query and get the identity/id back from store (INT) and when it comes to map the INT from store to the property, it will use the configured conversation.

I tested again with foreign key too, and its also not possible to map foreign keys with properties using custom types and configured converters:

public class Order
{
    public CustomerId CustomerId { get; private set; }
}

public class OrderEntityConfiguration : IEntityTypeConfiguration<Order>
{
    public void Configure(EntityTypeBuilder<Order> builder)
    {
        builder.ToTable("Orders");

        // works fine and maps client type CustomerId to storage type int and reverse
        builder.Property(x => x.CustomerId)
            .HasConversation(id => id.Value, value => new CustomerId(value);

        // not working
        builder.HasOne<Customer>()
            .WithMany()
            .HasForeignKey(x => x.CustomerId);
    }
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment