Support for .net Core and EF core?

[Eneuman]

Hi!

Will there be support for .Net Core and Entity Framework 7?
If so, when? :)

[jaredmoo]

Hi, thanks for the request! We have no plans to implement this in the near future, but the code is open-source so if you'd like to give it a try yourself I'd be happy to work with you on getting it merged. :)

[andrelmp]

i tryed, but is pretty impossible now, .net core RC1 doesnt have IDataReader and IDataRecord interfaces also SqlDataReader.GetSchemaTable() is missing.

I will look into again when RC2 comes out

[abatishchev]

Any chance you could take a look at EF Core 1.0 please? Also I'd rename the title, from EF7 to EF Core.

[pristinesource]

https://github.com/pristinesource/ElasticDatabaseToolsNetCore
https://www.nuget.org/packages/ElasticScaleNetCore.Client/1.0.0

[jaredmoo]

Awesome, thanks @pristinesource ! Our build situation is a little weird but I'll see if I can slip the .xproj etc that you created into our codebase so we don't end up with 2 parallel codebases.

[pristinesource]

Awesome! There were certainly a lot more changes than just the project file. I will do 2 commits, one reverting all changes and another reapplying all changes to you can see everything I did in one commit. The way Net Core projects work though, I had to create a whole new project and copy the source files to it instead of modifying the existing project. I will add another comment here after I make those 2 commits with a link to the diff.

[pristinesource]

@jaredmoo, I have created a single commit that has all of the changes. I realized that a huge part of it was the namespace refactor, but if you ignore that, you can see all of the work I did to get this library to work with .Net Core. I hope this helps and I would LOVE to see an official .Net Core version soon :)

pristinesource/ElasticDatabaseToolsNetCore@4d4917f

[jaredmoo]

Hey, I'm working through the changes and it all makes sense except for the change to MultiShardDataReader.Close() to make it not override. Do you understand what's going on here? DbDataReader has virtual void Close() defined, I don't understand why it can't be overridden.

[pristinesource]

@jaredmoo: That is very strange, when I try to add "override" back to the method declaration in VS2015 it gives the error that there is no valid method to override. When I look at the API docs for the DbDataReader it does say that there is a Close() method and it is virtual https://docs.microsoft.com/en-us/dotnet/core/api/system.data.common.dbdatareader#System_Data_Common_DbDataReader_Close, so I am not sure what the discrepancy is on my end. Are you seeing any similar behavior when you open my project directly?

[jaredmoo]

Yes I see the same thing. What's also weird is that when I press F12 on DbDataReader, VS shows me the metadata with no .Close method. So I think that somehow DbDataReader class that is being referenced is not the same as the source that I just linked. Very baffling.

[jaredmoo]

Also these errors makes no sense. DbDataReader inherits IDataReader, which contains .Close().

[pristinesource]

@jaredmoo: It has confused me a lot also even with other stuff in .Net Core. Take for example if you attempt to open the referenced DLL (the comment at the top of F12) in ILSpy it only gives the structure (which matches F12 output and that is correct because it is a reference DLL), but I cannot find the real library that matches the reference DLL. I am not sure what the true referenced assembly is in this case, if it is the corefx library then it does have the Close() method (according to ILSpy), but that is not what VS2015 and the compiler see/use.

[jaredmoo]

Hmm yeah maybe the reference assembly that VS is using is not correct? I'll try contacting .net core folks. btw my working branch is https://github.com/jaredmoo/elastic-db-tools/tree/NetCore

[pristinesource]

@jaredmoo: Awesome! Then maybe all of the calls to *.Close() can be uncommented too ;)

[jaredmoo]

https://github.com/dotnet/corefx/issues/11949

[jaredmoo]

Apparently Close() should be replaced with Dispose(), see dotnet/corefx#11949. Since there are a lot of issues in MSQ, I totally disabled MSQ for now. The good news is that SMM now builds successfully in https://github.com/jaredmoo/elastic-db-tools/tree/NetCore . Next step is to get the SMM tests to build so that we can find out if these compile fixes actually produced an assembly that works good :)

[jaredmoo]

Ok, unit tests are going to be messy due to usage of fakes. One more reason for me to dislike fakes :)

That's about as far as I can get today, and unfortunately I don't expect I will be able to get back to this for a little while. If you want to make changes on top of https://github.com/jaredmoo/elastic-db-tools/tree/NetCore please go ahead and I'll be happy to review. Just try to make each individual commit small so that it's reviewable.

Cheers

[pristinesource]

@jaredmoo: How do you feel about me switching to xUnit for unit tests while I am updating them to work with the new changes for Net Core?

[jaredmoo]

That would be fine, but if possible would you be able to do that on a separate branch based on master (separate from net core changes)? So the branching is like

master -> net core     /-> net core with tests
      \-> xunit      -/

The reason that I ask is that it would be nice to be able to merge xunit into master independently of net core.

Also the latest xunit version that is available on my build servers is 2.1.0, will that work?

[pristinesource]

Absolutely! I will do that :)

[jaredmoo]

Awesome, thanks!

[jaredmoo]

Removed fakes dependency: #143

[JohnGalt1717]

Are there any samples or guidance on how to setup constructors and dependency injection etc. in .NET core yet?

[grajdeanserghei]

Hi!

Are there any updates on this issue?

@jaredmoo I can see that https://github.com/jaredmoo/elastic-db-tools contains NetCore branch and two pull requests related to it created by @pristinesource. Why it is not merged in this repository and not included in NuGet package? Do you need any help with that?

Are there any plans to support this library? It haven't been updated since October.

[erick-thompson]

I was able to get the core branch up and running, but I ran into an issue when creating the ShardMapManager. The call

var manager = ShardMapManagerFactory.GetSqlShardMapManager("", ShardMapManagerLoadPolicy.Eager);
results in a Linq exception "Sequence contains no elements"

System.InvalidOperationException occurred
HResult=0x80131509
Message=Sequence contains no elements
Source=System.Linq
StackTrace:
at System.Linq.Enumerable.Single[TSource](IEnumerable1 source) at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.GetShardMapManagerGlobalOperation.DoGlobalExecute(IStoreTransactionScope ts) in P:\external\elastic-db-tools\Src\ElasticScale.Client\ShardManagement\StoreOperations\ShardMapManagerFactory\GetShardMapManagerGlobalOperation.cs:line 51 at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.StoreOperationGlobal.<Do>b__10_0() in P:\external\elastic-db-tools\Src\ElasticScale.Client\ShardManagement\StoreOperations\Base\StoreOperationGlobal.cs:line 83 at Microsoft.Azure.SqlDatabase.ElasticScale.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func1 func) in P:\external\elastic-db-tools\Src\ElasticScale.Client\ElasticScale.Common\TransientFaultHandling\Implementation\RetryPolicy.cs:line 216
at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.StoreOperationGlobal.Do() in P:\external\elastic-db-tools\Src\ElasticScale.Client\ShardManagement\StoreOperations\Base\StoreOperationGlobal.cs:line 73
at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.ShardMapManagerFactory.GetSqlShardMapManager(String connectionString, ShardMapManagerLoadPolicy loadPolicy, RetryBehavior retryBehavior, EventHandler1 retryEventHandler, Boolean throwOnFailure) in P:\external\elastic-db-tools\Src\ElasticScale.Client\ShardManagement\ShardMapManager\ShardMapManagerFactory.cs:line 580 at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.ShardMapManagerFactory.GetSqlShardMapManager(String connectionString, ShardMapManagerLoadPolicy loadPolicy, RetryBehavior retryBehavior, EventHandler1 retryEventHandler) in P:\external\elastic-db-tools\Src\ElasticScale.Client\ShardManagement\ShardMapManager\ShardMapManagerFactory.cs:line 508
at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.ShardMapManagerFactory.GetSqlShardMapManager(String connectionString, ShardMapManagerLoadPolicy loadPolicy, RetryBehavior retryBehavior) in P:\external\elastic-db-tools\Src\ElasticScale.Client\ShardManagement\ShardMapManager\ShardMapManagerFactory.cs:line 463
at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.ShardMapManagerFactory.GetSqlShardMapManager(String connectionString, ShardMapManagerLoadPolicy loadPolicy) in P:\external\elastic-db-tools\Src\ElasticScale.Client\ShardManagement\ShardMapManager\ShardMapManagerFactory.cs:line 442
at SharedServices.QuantumShardMapManager.GetShardMap() in P:\quantumeventservice\SharedServices\Repositories\QuantumShardMapManager.cs:line 23
at SharedServices.Tests.Repositories.QuantumShardMapManagerTests.Test1() in P:\quantumeventservice\SharedServices.Tests\Repositories\QuantumShardMapManagerTests.cs:line 17

It looks like it is coming out of the SqlUtils class. I see that the SQL command is embedded in ReadOnlyScripts.resx. I'm going to see if I can get it working locally, but I assume that any of the SQL inside the resources will have similar issues.

[jaredmoo]

I've been having a similar problem with the resources but I thought it was fixed. Let me take another look.

[jaredmoo]

The issue is dotnet/msbuild#308 . The resources build correctly in VS but not using dotnet core SDK. Investigating fixes/workarounds.

[erick-thompson]

Nice find! It there a particular reason why interned strings aren't an option? Or are you using the locales for various flavors of SQL?

[jaredmoo]

Resx was used for developer convenience, but it's not very convenient now that resx build isn't working. I just pushed an update that switches to hardcoded strings.

[erick-thompson]

Awesome, thanks. I'll pull you changes and rebuild.

[GreenAirplane]

We are starting a new big project and we would like to use elastic tools instead of our own tool. So, my question is: @jaredmoo, do you need any community help in order to speed up the release of .net standard support?

[jaredmoo]

Just try it out and let me know if you have any issues, also it would be great to hear if everything worked :) . The only thing left to do is signing and publishing an official release, which unfortunately can't be done by community.

[GreenAirplane]

Just wanted to provide an update. We've created a "proof of concept" application and looks like it works for us! We are using Elastic Tools with EF Core and so far so good.

Btw, there was one time when some stored procedures were not created in a shard db (e.g. spBulkOperationShardMappingsLocal). However, I was not able to reproduce it again. Also, I've found issue #147 with a similar description, so I don't think that this case is related to .net standard support.

[GreenAirplane]

One more thing, it might help some other people who decide to use pre-release version.

We had a strange issue when this lib didn't work in a new solution. It just crashed with "Could not load file or assembly" error. It looks like there is an issue in VS 2017 and .net core how they work with libs from local disk. The easy way to fix is to generate a local NuGet package and add it to the new solution. Works like a miracle..

[TomBeckett]

Any update on when this may be signed / published as an official release?

[drewauman]

Can you let us know when this is going to be signed and published? Looking forward to it!

[jaredmoo]

Thanks for the feedback and patience everyone. It is now published on nuget.
https://github.com/Azure/elastic-db-tools/releases/tag/v2.0.0-preview2
https://www.nuget.org/packages/Microsoft.Azure.SqlDatabase.ElasticScale.Client/2.0.0-preview2

[rob-mckenna]

Are any Entity Framework Core samples available?

I'm using 2.0.0-preview2 from Nuget but getting stuck on Database.SetInitializer<MyDbContext>(...) in both the SetInitializerForConnection and CreateDDRConnection methods. This is because the samples aren't using EF Core so System.Data.Entity is not available. And in EF Core it appears the equivalent is DatabaseFacade, although they do not line up in terms if initialization.

For example, calling 'Database.SetInitializer<MyDbContext>(null);' from CreateDDRConnection results in a "An object reference is required for the non-static field, method, or property 'DbContext.Database'" error.

[jaredmoo]

Sorry I haven't gotten a chance to try with EF core :(

[bbqchickenrobot]

In case anyone was wondering I whipped a quick EFCore sample to use with preview-2.0.0 build.... it's not perfect and/or tested - but if you wanted to understand the concept I created a Gist here:

Azure Elastic Scale Client and EFCore 2.1

Let me know how it works. Hope that helps....

[drewauman]

Could anyone send a full working sample of sharding with EF 7 Core? I have to admit, I have been banging on this thing off and on for way too long. Would be very helpful if someone has a working sample. Would be a huge help here ... I want multiple Azure database in an elastic pool, multiple tenants per database with the ability to filter at the database level and to do split/merge to move tenants from database to database as needed. Help! :) Thank you ahead of time, I appreciate the help. Feel free to send it directly to me at drewauman [at] outlook.com if you could. Thanks!!

[bbqchickenrobot]

@drewauman - I provided a full working sample. Create your DbContext from your database, create your shards and then you can use my gist to talk with it. I don't have a working sample as this would require to create a database, configure sharding (this is done within azure sql), etc....

Assuming you are using this tool + Entity Framework Core + Entity Migrations you may want to check this out to help you get started pretty quickly creating a database:

Entity Migrations w/ Elastic Scale Client

[drewauman]

Thank BBQ .... I have got this working several times in the past using Entity Framework. It's Entity Framework "Core" that I have had a hard time finding documentation and/or examples for. I have my solution setup, I just seem to be having issues creating the schema the first time in the shardlet DBs .... it is driving me crazy! ha ha and it's probably something simple that is causing it. That's why I was asking for a working version (I was just looking for the code or solution).

The downloadable example that is provided in the link that you sent is entity framework 6 it appears, not core. Thoughts? Thanks for your time sir!

[drewauman]

I can't seem to figure out how to add migrations for the first time or kick off migrations for the first time and then subsequent times into the shard database. I am in a Entity Framework Core Code First setup. I need to initialize migrations for the first time and then migrate automatically every time new migrations are added. Thoughts?

[moawadallah]

is there any documentation for using it with EntityFrameworkCore?

[SychevIgor]

Thank you for everything you did!
It will be nice to have a version without "preview" in the end
https://github.com/Azure/elastic-db-tools/tags here we can find only preview-2 that was shipped in February 2018... 8th month ago.

[vigouredelaruse]

with the imminent move to a write once run almost anywhere .net 5 situation, what is the current assessment of the relevance of porting this library to .net core as per https://docs.microsoft.com/en-us/dotnet/core/porting/ ?

obviously ideally both the .net framework and .net core versions should pass the same unit tests

i fear further time languishing with .net framework dependencies merely leaves this open source effort subject to the predations of a branched commercial effort that supports .net core

to kick off any potential further discussion here is a screenshot of the output of the portability analyzer tool mentioned - i've attached the source xlsheet and a pdf export of same for those who don't download xl spreadsheets from strangers on the github

ApiPortAnalysis.xlsx
ApiPortAnalysis.pdf

please advise