New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ESENT database provider #4423

Closed
Opiumtm opened this Issue Jan 28, 2016 · 61 comments

Comments

Projects
None yet
10 participants
@Opiumtm

Opiumtm commented Jan 28, 2016

Provide Entity Framework 7 provider for a ESENT NOSQL database.
This database is officially supported on UWP platform, but now there is only ISAM imperative access to ESENT DB in C#.

ESENT is proven as very fast, stable and reliable NOSQL database, and it is supported in UWP apps.
It's a good alternative to SQLite on UWP platform. But ESENT lacks any high-level API, it have only low-level imperative C++ style APIs for C#.

@ErikEJ

This comment has been minimized.

Contributor

ErikEJ commented Jan 28, 2016

Could you link to the current C# api?

@Opiumtm

This comment has been minimized.

Opiumtm commented Jan 28, 2016

Current ESENT C# api:

https://managedesent.codeplex.com/

@Opiumtm

This comment has been minimized.

Opiumtm commented Jan 28, 2016

And here documentation of ESENT C# on MSDN:
https://msdn.microsoft.com/en-us/library/dn375980(v=exchg.10).aspx

@Opiumtm

This comment has been minimized.

Opiumtm commented Feb 4, 2016

ErikEJ, could you answer if EF Core team planning to support ESENT DB in future? To be honest, I'm not satisfied with SQLite on UWP platform. I have used SQLite many times in a past and it certainliy not a best option. But today UWP platform have not any other options, only SQLite DB. It's very weird, because UWP platform (and Microsoft Windows as well) natively support ESENT embeddable database, known for decades as a good, stable, fast and reliable database engine. The only problem with ESENT its out of date APIs, lack of support from any major data access frameworks. In the server or desktop worlds it's not a problem - there is many other supported database engines, SQL or NoSQL. But on the UWP platform officially supported database engines is scarce. SQLite is recommended by Microsoft as a primary solution. And it's very weird! Microsoft have it's own well tested and proven reliable embeddable database engine, ESENT. And Microsoft is totally silent about ESENT. It's officially supported on UWP, but to productively use ESENT in UWP you must be a die hard C++ developer, or at least have this hardcore experience in the past. Newbie developers have not any chances to use ESENT in their UWP applications, because APIs is very hardcore and aged, no sane C# developer would use ESENT when there is much easy to use SQLite option. But, I must write it again, SQLite is not a best option technically speaking. ESENT is a much more technically advanced database engine. For an embeddable database engine ESENT is certainly a piece of art.ESENT supports snapshot isolation (feature supported not by all server-scale full blown SQL databases!) for a transactions, multi-value index keys, pre-joined tables, an much more... And same time ESENT have very small memory and CPU footprint, it's truly embeddable database, suited for use even on a smartphones (and internally Microsoft use ESENT DB in many parts of Windows OS itself, ESENT was an internal OS related project made publically available some time ago).

@ErikEJ

This comment has been minimized.

Contributor

ErikEJ commented Feb 4, 2016

@Opiumtm I do work for the EF team, so I am not aware of any plans. One major issue I see with ESENT is that it does not have a query language and a query processor as in a RDBMs system. But I think it would be a good candidate for a useful NoSQL provider, and I would maybe be interested in taking a stab at it, but the main issue with that is, that there are no NoSQL EF providers implemented based on the current codebase.
The plans as I understand them are to revive the Azure Table Storage provider, which exposes a key/value storage quite similar to the PersistentDictionary.

@Opiumtm

This comment has been minimized.

Opiumtm commented Feb 4, 2016

@ErikEJ

major issue I see with ESENT is that it does not have a query language and a query processor as in a RDBMs system

It will not be a problem because ESENT does not have its own query language, but it anyway a relational DB and ESENT API supports querying, indexes, tables, joins via its own ISAM imperative style (open table, have an open handle, add filters and indexes, enumerate filtered records via method calls and so on) API. It does not differ fundamentally from any relational SQL database (ESENT is a perfect example of a relational database) and ESENT APIs can be mapped to a LINQ language without a SQL intermediate language. In fact, this direct mapping to a LINQ queries will be much faster, because it will be mapped directly, without intermediate text based language, without query processing overhead.

So, ESENT is a relational NoSQL database (NoSQL in a sense that it does not support SQL queries, but support relational queries and operations directly via low level API). ESENT is well suited for a direct LINQ-to-"low level relational API" mapping!

@ErikEJ

This comment has been minimized.

Contributor

ErikEJ commented Feb 4, 2016

@Opiumtm Sounds great!

@Opiumtm

This comment has been minimized.

Opiumtm commented Feb 4, 2016

@ErikEJ yes!
And think about ESENT is an embeddable database, designed to use in a resource constrained environments (but it as well used at an enterprise level too - just think about a fact that ESENT used as a data storage solution in such products as Miscrosoft Exchage Server and Active Directory!).
Its embeddable nature matches well an opportunity to direct map its relational API to LINQ query language without any intermediate layers. It will be a perfect solution for an UWP universal platform, it will make (honestly speaking) mediocre SQLite out of competition easily.

@Opiumtm

This comment has been minimized.

Opiumtm commented Feb 4, 2016

@ErikEJ
Also I personally have some experience in ESENT C# use on an UWP platform (and some past experience long time ago in a native C++). Major negative factor for its use is not a technical issues (it works good technically) but an out of date low level API and lack of support from any data access frameworks. It require a big amount of code to use in production. When you familiar with ESENT hardcore low level API it is not a problem to write a workable code, but a problem with a big amount of a boilerplate error prone code still persists, or you must invent your own homebrew framework for a personal ESENT use.

@rowanmiller

This comment has been minimized.

Member

rowanmiller commented Feb 4, 2016

@Opiumtm this is interesting because SQLite is very limited in what it supports, and this flows on to a bunch of limitations in EF functionality (see http://docs.efproject.net/en/latest/providers/sqlite/limitations.html). I have never looked at ESENT, do you think it would support a number of the things that SQLite does not?

@Opiumtm

This comment has been minimized.

Opiumtm commented Feb 5, 2016

@rowanmiller Not sure, what you mean as "schemas". If you mean a queryable database structure scheme - it's supported. If you mean "schema" as a SQL logical construct - it's not supported by ESENT because ESENT is not a SQL database, ESENT does not support access rights (it is supposed to do by an application layer). ESENT is a relationanal transactional (it supports snapshot isolation) database, but it does not have any text based universal query language, all queries, sorts and joins are performed via direct ESENT API calls, ISAM style. So, ESENT is perfectly suited for a direct mapping of relational API method calls into a LINQ queries without any text based query language intermediate layer. ESENT is fundamentally designed for this approach as a fast small footprint embeddable database engine - omit any intermediate logical layers. ESENT is a perfect candidate for a LINQ query language support.. LINQ by design is not a text based language but a direct mapping to a method calls represented in a compiled code or in an abstract query tree (in case of query expressions). To be honest - SQL is an useless excess entity here. SQL is not needed at all. LINQ fundamentally is a declarative style method mapping to a relational operators - and ESENT supports exactly this, relational operators without useless (for an embeddable database) intermediate text based language.

@Opiumtm

This comment has been minimized.

Opiumtm commented Feb 10, 2016

@ErikEJ @rowanmiller guys, could you share more info about planning (or not planning...) a support for ESENT DB in EF Core?

@ErikEJ

This comment has been minimized.

Contributor

ErikEJ commented Feb 10, 2016

@Opiumtm I do not work for the team, I am just working on the SQL Server Compact provider. I might consider having a look at this when the Azure Table Storage provider has been released, after RTM of 1,0.
I do however see limited scope in the Esent provider, as it is only available for desktop and UWP tablet apps, not Windows Phone (or any other platforms). One advantage of sqlite is that it is available on all platforms

@Opiumtm

This comment has been minimized.

Opiumtm commented Feb 10, 2016

@ErikEJ You are not correct. ESENT is officially supported on Windows 10 Mobile for ARM and I already use ESENT in one my UWP application which targets both desktop and mobile. So, Windows 10 for mobile support ESENT and it's tested.

@ErikEJ

This comment has been minimized.

Contributor

ErikEJ commented Feb 10, 2016

@Opiumtm That is great, did not realize that!

@Opiumtm

This comment has been minimized.

Opiumtm commented Feb 10, 2016

@ErikEJ I successfully deployed application that use ESENT on a Lumia 920 smartphone and it works nice. ESENT supported on smartphones as of Windows 10 version of Windows Phone (now Mobile).

@rowanmiller

This comment has been minimized.

Member

rowanmiller commented Feb 10, 2016

@Opiumtm this probably isn't a provider that our team would develop, we typically work with database vendors or third parties to develop the vast majority of providers. This allows us to push forward with new features in the core of EF.

@Opiumtm

This comment has been minimized.

Opiumtm commented Feb 10, 2016

@rowanmiller And it's weird because ESENT is a native Microsoft embeddable database known for decades for its speed and reliability, it's an official part of classic native WinAPI (and part of UWP API as well). I do not understand why ESENT is not considered as a first-class database provider by a Microsoft "recommended API" team. As I described above, ESENT is perfectly suited for LINQ. If EF Core does not suited for ESENT - it's a major issue. Maybe modern cross-platform minded developers does not care about this brilliant Windows classics, but for an aged veteran windows developer like me it looks like a betrayal and spit directly in a soul. Sorry for an emotions, but it's exactly what I feel. And it certainly not a wise decision. You must not discard things that works perfectly and promises strong technical advantages. ESENT simply is far better than SQLite and its lack of SQL query language support is not a weakness for an embeddable database suited for even a smartphone use. SQL language is not needed at all when you can directly map ESENT relational API to LINQ queries. SQLite is a mediocre and limited database that suffer from a performance issues when database size is slightly larger. ESENT on the other hand is very well known for its speed and scalability. And ESENT supports snapshot isolation for a transactions, have normal strong-typed data model instead of SQLite lame weak-typed data model. And it all we got in a embeddable database. It's certainly a CRIME to does not support ESENT for an embedded database scenario for an UWP applications!

@ErikEJ

This comment has been minimized.

Contributor

ErikEJ commented Feb 10, 2016

@Opiumtm I am sure everyone would be happy for a open source contribution to enable this provider

@Opiumtm

This comment has been minimized.

Opiumtm commented Feb 10, 2016

@rowanmiller Well, and at EF 3-6 era I understands why ESENT is not supported by Entity Framework. In the past EF was a strictly SQL language based framework. As ESENT does not support SQL language queries it was not a subject for an EF support. But now it is claimed that Entity Framework Core does not tightly binded to a SQL language and it will be NoSQL providers for an EF. When I read this on a technical news feed I think - time have come. I think Microsoft would certainly support its own ESENT as a part of a first-class EF providers, especially when there is claimed that EF Core is suited for an UWP platform too. SQLite only support for an UWP platform is a lame. I understand that EF goal is primarily to support enterprise-scale server scenarios involving Microsoft SQL Server, PostgreSQL, Oracle and so on. But please, does not forget an embedded and client-oriented world! Until now there was very few options for a client-oriented .NET developers. SQLite is a well known cross-platform client oriented embeddable database, but certainly is not a better one. It will be nice for a UWP developer to have an alternatives to mediocre and not best at all SQLite database (which personally I consider a lame and mostly unprofessional database engine flourishing on the client-side because of lack of an adequate and well-known alternatives, and ESENT is certainly an adequate alternative for a Windows Platform and it here for a decades). And after all ESENT is a Microsoft native database and must be respected at least by a Microsoft staff. It certainly deserves more respect. As I see, knowledgeable 3-rd party developers have more respect for an ESENT DB than Microsoft themselves.

@Opiumtm

This comment has been minimized.

Opiumtm commented Feb 10, 2016

@ErikEJ It's bad that ESENT DB have not much attention in a developer circles. Known not by all, even not well known by a many Microsoft teams. It was a shocking fact for me that ESENT is not well known even to a Entity Framework Core team! It looks like a sadomasochistic perversion when Microsoft officially recommends SQLite for a client UWP applications and totally silent about anyway officially supported ESENT. What? Microsoft staff are totally replaced by an iOS hipsters or what?

@divega

This comment has been minimized.

Member

divega commented Feb 10, 2016

@Opiumtm ESENT is actually one among a few data stores that were discussed when we started prototyping non-relational providers at the very beginning of the development of EF Core. Personally I agree that an EF Core provider for it would be a very good addition and potentially a rich and unique source of challenges that could help us improve our designs.

Plans and priorities can change at any time, however as @rowanmiller explained we currently intend to only focus on building providers for a very small set of stores. For the rest we prefer to partner with people from the OSS community, from database vendors and other teams at Microsoft that already have the necessary expertise on the specific data store technologies and that are willing to go through the exercise of learning how to create an EF Core provider with us.

I have been following this thread with attention hoping that you would be interested in contributing to that effort. As @ErikEJ mentioned, after EF Core 1.0 RTM and after we have made more progress on some of the non-relational providers using the current codebase would be a much better time to start with an effort like this.

@Opiumtm

This comment has been minimized.

Opiumtm commented Feb 10, 2016

@divega

ESENT is actually one among a few data stores that were discussed when we started prototyping non-relational providers at the very beginning of the development of EF Core

Well, ESENT is a relational transactional ACID-compliant database provider. "SQL language" is not equal to a "relational", it's just a relational database ANSI standard for a text-mode API and a declarative programming language. Just point to this again because there is some theoretical misunderstanding on this thread.

however as @rowanmiller explained we currently intend to only focus on building providers for a very small set of stores

For a server enterprise-level tier ESENT provider certainly is not a top priority. But it's vital for a mobile client-oriented UWP platform. And EF Core is designed for an UWP platform too. So, it's very different set of real life requirements and challenges on a different platforms. As Joel Spolsky said in his famous Law of a leaky abstractions article, it's an exact situation. Abstraction of "support of a most needed data providers" leaks when we see that on a UWP client-side platform there is only mediocre and arguably lame SQLite provider is supported and there is no clear plans for much better alternatives like ESENT (which itself is violently kicks a leaky "SQL language" abstraction, that certainly leaks on a mobile and embedded platforms and become such abomination as SQLite database engine is).

I have been following this thread with attention hoping that you would be interested in contributing to that effort

Personally I thinked about this. But as for now I am already writing a some open-source client oriented project (not on a GitHub, on a CodePlex). Maybe later I can do some contributions to a ESENT DB provider, and in near future I plan to upgrade one of my Windows Phone applications to a Windows 10 universal UWP platform and migrate it from a SQLite to ESENT (which is supported starting with UWP Windows 10 and was not supported on a Windows Phone 8.1 runtime). This certainly will give me a live measurements of an ESENT usage in a wild. As for SQLite, I already got a serious performance degradation issue with it when application is actively used and data in a database updated many times after a each-day usage pattern and grows to a small but many indexed records. In my somewhat distant developement past for a native desktop Win32 platform I already got this SQLite issue when it used for a client-side database cache. It works well and fine first time but as time pass and cache grow (not to terabytes or even not to a gigabytes, I must say!) SQLite seriously suffer from an enormous performance degradation to a point of a total uselessnes. Maybe I need more SQLite expertise and must invoke some magical SQL commands to optimize and defragment SQLite database... But it certainly a discussed above point to a law of leaky abstractions. SQLite abstraction of a "full featured SQL database on a smartphone and embedded system" is certainly leaks. SQL query processing on an embedded system is have its overhead, is not complete, is not optimal (SQLite certainly does not have such a powerful query optimizer as Microsoft SQL Server or PostgreSQL have) and come with many compromises and limitations - and it's known that SQL query language itself is very abstract and good performance is up to a database query optimizer, not a SQL query itself which does not represent much of a detail. It's how abstractions leaks and it is not a SQLite code base issue, it's an issue with a fundamentally lame abstraction of a "full featured SQL language support on an embedded system" which naturally leaks. LINQ on the other hand is more specific and less abstract than SQL query language and it naturally represents a relational operators call tree. LINQ may be optimized by a programmer in a more straightforward way than SQL and it is optimized by a developers when queries are made not to a relational data sources but to a XML in-memory datasources or a to a in-memory object tree and this LINQ query mechanics is not hard to understand.

After all, I am not against some contributions (at least theoretical) to an embedded scenarios of EF Core including ESENT.

@divega

This comment has been minimized.

Member

divega commented Feb 10, 2016

ESENT is a relational transactional ACID-compliant database provider

Thanks for clarifying that. I had read you mention that before, so sorry if I used the term inaccurately anyway 😄 BTW, when I say "non-relational" in the context of EF Core providers I am actually using it at a shortcut to refer to a category of providers that won't inherit from our base relational provider. The latter makes lots of assumptions about SQL being supported, an ADO.NET provider being used to execute commands, etc.

Maybe later I can do some contributions to a ESENT DB provider, and in near future I plan to upgrade one of my Windows Phone applications to a Windows 10 universal UWP platform and migrate it from a SQLite to ESENT ... This certainly will give me a live measurements of an ESENT usage in a wild.

Cool! We also need to do some groundwork to enable this kind of non-relational provider again on the EF Core 1.0 codebase. I hope the timing will work well and to hear back from you.

@Opiumtm

This comment has been minimized.

Opiumtm commented Feb 10, 2016

@divega if it's not too late before release consider rename your inside EF Core terminology from a "relational" to a more correct "SQL based" :-) It may be a big benefit for a better theoretical understanding of a EF Core codebase and architecture. "SQL based" in practical terms means only that LINQ queries is translated into a SQL intermediate language and SQL representation of a query is a central point of a "SQL based" provider, it's all about generation of such a SQL representation for any raltional query and operation, even for a database tables construction at a run time. "Relational" is a more general term and does not require SQL language (or any standartized intermediate text based language at all). Tables, indexes, rows, primary keys, relational operators ("select" or "join" for example) - it's a "relational" and it's supported by many non-SQL databases, especially by any ISAM database engines (on top of which SQL database abstraction can be naturally built, as it did by MySQL database on top of MyISAM storage engine). So, any classical ISAM engine is relational but not SQL.And as ISAM engine can be upgraded into SQL engine naturally. Historically most well known SQL databases (starting with IBM DB2) are indeed built on top of classical ISAM or some ISAM variation - it is correct to generalize "relational" term into an ISAM data source providers, which are anyway historically a foundation for SQL databases, on top of which SQL query processor and SQL query optimizer are built.

@Opiumtm

This comment has been minimized.

Opiumtm commented Feb 10, 2016

@divega So, "NoSQL" term is largely incorrectly used buzz-word and much abused in a modern developer circles. Non-relational databases are hierarchical databases (every day used OS file system is a perfect example of a hierarchical database and most FS are built on top of typical database B+ tree data structure and typical database journaling for data integrity - just think about it), document oriented databases, "key-value" databases and so on. It's all examples of a non-relational DB. "NoSQL" term as it used by a most developers is based on a false assumption that "SQL" an "relational" terms are equal and it drives most discussion into a theoretical misunderstanding and confusion. They are not equal. SQL databases are most used implementation of relational databases, but "SQL" is a subset of "relational", not equivalent.

@rowanmiller rowanmiller added this to the Backlog milestone Feb 12, 2016

@rowanmiller rowanmiller changed the title from EF7 for ESENT database to ESENT database provider Feb 12, 2016

@Opiumtm

This comment has been minimized.

Opiumtm commented Feb 25, 2016

@10Dev I think there is never too late to recover. Yes, I am very sceptical about future of Microsoft developer platform. Not because I like this skepticism. I'm certainly not like it. It's because I can not deny it anymore. Yes, rapid abandonment of own technology is not help to make things clear. There is no point to invest in a new platform/technology when its future is utterly uncertain. No matter how brave and determined Microsoft looks at developer conferences when speak about some technology. As we have learned already, bravado on a scheduled developer conference essentially mean nothing. Last years Microsoft is too accustomed to abandon things even before it fully mature. And most terrifying of all Microsoft is too eager to accept "actual developer trends". Not dictate such trends, merely accept. For example, I see JSON as very weird format for a project config format in VS 2015 .NET Core projects. JSON essentially is a JavaScript bastard child, widely (mis)used because so much people are Html/JavaScript monkey coders (just name things honestly) and they are too well accustomed with such a ad hoc format as JSON is. Well, it looks good in a browser, but in a VS solution folder it looks like a homeless miserable at a aristocracy banquet. It's just a wrong place for him.
It's just one example, but it good in illustration of how Microsoft essentially shifted from a major influential force in a developer world to a merely "accepting the trend" position even within its own environment. It is utterly a passive position. Well, any extremes are harmful. But today we see a very passive Microsoft, passive even on its own platform. Microsoft is so passive it could abandon any technology any time because some girlishly looking hipsters say "this technology is out of trend pheeeee".

@10Dev

This comment has been minimized.

10Dev commented Feb 25, 2016

Opiumtm, I enjoy considering your thoughts and hopefully with the huge wealth of focused code level issues on the EF list, the team will be kind enough to tolerate a few tangental comments related to this rather key issue.

I always feel a bit weird when people refer to a large company with many thousands of people as if it is some sort of pseudo-person with its own thoughts and desires... I'd rather think of Microsoft as a kind of process flow diagram for a nuclear power plant.

Another point is that people in that "plant" have access to huge amounts of metric data about sales and usage although I suspect some of the large blunders come from how such data is compressed and "sanitized" to upper management. But they have a lot of market data that nobody outside the organization has acess to and it is part of their often perplexing decision making.

It is possible that what you are seeing as "passive" is an attempt to be "inclusive" and there is also the horrible list of communications and marketing firms they have used over the last decade. Most of the Silverlight mess which did huge mindset damage was just horrible communication. (with a possible dash of typical large corporation absurdity - rumour has it that Balmer killed Silverlight as part of a secret deal in CEO to CEO meeting he had with Adobe becuase he saw the Silverlight effort as a "Flash Compete" and didn't realize that a large numer of enterprises were planning to adopt the technology for in-house application deployment via the OBE option) and then they disbanded the WPF team and it just looked like a complete slash and burn to the outside world.

But all that wonderful technology is still available, the WPF team is operational again and XAML has been embedded into the Windows API to yield both performance benefits and universal availability to other languages and the things you can get done in the Microsoft ecosystem is better than it ever has been (well, except the EF UWP thing) and somehow even now the positive steps are not being well communicated. It's not "pasive", just failure to communicate.

The EF team has developed an excellent world leading product that has also been under-promoted. At the least they should have a dedicated web site leading out with "As a developer is your time worth anything at all to you? The EF Team gives you SuperPowers in a Bottle"

@Opiumtm

This comment has been minimized.

Opiumtm commented Feb 25, 2016

@10Dev I agree with you on that now Microsoft doing many truly wonderful things. For example, .NET Core project and .NET foundation activity as whole are things that make .NET what it intended to be from the start - truly cross platform not in a cross-OS meaning, cross platform in a meaning that it can run on a wide spectrum of different hardware platforms, existing and future.
But, all right and wonderful long-term movements does not answer a sharp questions of present day. On present day there is very much amount of uncertainty, unfilled gaps, lack of public relation (for example, there is no understandable roadmap to Microsoft platform and UWP future), lack of integration and lack of backward compatibility with a lot of .NET classics and lack of feeling of that Microsoft is truly dedicated and put enough resource to make developers certain in their future. After all, our future is tightly coupled with a future of Microsoft development platform. It's our wages, work places, business opportunities, our sales.

@rowanmiller

This comment has been minimized.

Member

rowanmiller commented Feb 25, 2016

I'm going to wrap this one up with a few comments and then step away from the conversation for a while.

We hear you on the fact that SQLite has some serious limitations that do surface heavily in the EF workflow. We are not ruling out the idea of table-rebuilds (#329) as a way to solve these - we're kicking around some ideas at the moment about how to achieve it without too much hidden magic.

We also hear you that SQLite is not a good fit for everyone - the concerns you raise are valid. We do want to have other providers available for UWP. It's just a matter of timing - I know you disagree with this, but our focus for 1.0.0 is SQLite.

Somewhere you had asked about calling out exactly what is supported in UWP rather than just saying "EF Core works on UWP" (I can't find it now though). This is a fair point and until we have other providers we can adjust the message to be more like "EF Core can be used in UWP with SQLite for local data access (note some limitations of SQLite)".

As a minor idea, if it doesn't degrade your workflow, it would be sensitive to your "customers" if you could re-tag this issue to "bug" (or nothing) since having a working EF on UWP just can't be seen as an "enhancement" to the outside world. In that context, "up-for-grabs" is like a slap in the face and "under review" might be a better choice?

Here is how we use the tags, and I think this is consistent with generally accepted use:

  • An enhancement is a feature that has not yet been worked on.
  • A bug is a piece of functionality that is already implemented but does not behave as expected.
  • up-for-grabs doesn't have any impact on priority or whether we will do the work on our team, it just means we are flagging the issue as something that is a good candidate for someone outside of the EF team to work on.
@rowanmiller

This comment has been minimized.

Member

rowanmiller commented Feb 25, 2016

And for my last personal comment on this thread too…

So, excuse me if I was too straight and not too politically correct on some points.

I am all for direct speech and addressing things head on rather than skirting around issues. That said, being direct and to the point is different than name-calling and using minority/less-privileged people groups in a derogatory manner.

When you must fight bravely you accept yourself as loser and leave yourself on a mercy of victor. As history tell us, victory is never come with a mercy. It always come with death of a loser, fast and painless death if you are lucky.

Mercy may not be a common tactic to win a war… but it sure goes a long way to preventing one 😉.

@10Dev

This comment has been minimized.

10Dev commented Feb 25, 2016

I would guess from the style of speech that English is not Opiumtm's native language. On the interenet, good idea to think about the information content and leave the rest of it to "lost in translation"

I can't agree that taking something that is not working and making it work is a "feature" although in some abstract kind of Kafkaesque way I can see the intent.

And although it feels like quibbling, from your previous description of how work on providers might easily lead to required changes in the core project (in the current state of affairs) then it only follows that "up-for-grabs" should not even be a practical option at this point.

In any case, I for one appreciate the feedback the EF has provided and to the extent that they are individuals completely appreciate their efforts on our behalf and to the extent that they are "oracles" of the entity known as MIcrosoft, there is the sense of some sort of management roadbloack in the flow chart of the organization that has impeded severely the delivery of an EF solution into the WInRT environment for 3 years now. Taken from a broader perspective this is clearly a dysfunctional situation that has caused some damage abstractly to the reputation of Microsoft and practically in the form of the cost of lost opportunity. Reading between the lines, this is completely beyond the control of the EF team to deal with and they are doing the best they can with the resources they have. They probably feel as much pain about the issue as we do on the outside.

Opiumtm: I think the EF team has provided as much guidence as they can and from that I would guess it will be about 6 months before we see something workable for production use and even that feels optimistic. It is what it is. Everything else about the UWP ecosystem seems very positive as all the previous roadblocks such as needing "sideload keys" to deploy apps have been removed. The new Composition API removes the previous DX interop roadblocks and other perf roadblocks that were previously hampering large complex enterprise WPF applications. Of course none of this has been communicated properly and that apears to be an eternal "feature" of Microsoft. Keep an eye on my UWP Tools List and I will try to document all of the data options for UWP but nothing can compare to EF.

@Opiumtm

This comment has been minimized.

Opiumtm commented Feb 25, 2016

@10Dev You are right, English is not my native language, I am from Russia. And political correctness here does not have much impact on a society, jokes about minorities quite popular even among minorities themselves (but outright aggression and hate speech against individual people or groups is seen as crime and heavy violation of social rules). After all, very popular, commercially successful and award winning modern Russian writer Viktor Pelevin (successful not only in Russia, his works are translated in many languages and Germany creating a film based on his works "Buddha's Little Finger") write in a genre that can be described as intellectual, sarcastic, philosophical fiction with hard social criticism, which is very funny but extremely political incorrect (not sure how such a political incorrectness is translated to German language and how film is created based on such a sharp sarcastic political incorrect material including black humor on drug abuse, alcoholism, mental instability, social exploitation, racial, national and gender stereotypes, violence, humor on nazi, communism, liberalism and capitalism ideologies alike, sarcasm on the actual political and cultural icons such a Barrack Obama, George W Bush, Vladimir Putin).

And for UWP platform, it is very promising in long perspective, but not generally polished, lacks vital functionality and certainly is not suited for now as a substitution for a classic desktop.NET/WPF applications. WCF support is limited, database support is even worse, essentially non existent.

@Opiumtm

This comment has been minimized.

Opiumtm commented Feb 26, 2016

@rowanmiller as for war, I am not against open source movement and accept Microsoft determination to make many of its technology accessible as open source. It's great, because to look into reference code of Microsoft tools and frameworks is a good school and helps understand technology which I as a developer use. Microsoft is going into a right direction with its open source initiative. But when we look from a commercial perspective (I am not a big fan of a capitalist system, because capitalism is well known for its ability to kill brilliant technology things in a commercial wars, bankruptcies, cost reduce process and so on), there are Apple, IBM and Google IT giants. They are heavily invest to drive customers and developers from a Microsoft platform and it is a dire reality of our days, in many aspects they are already succeeded. It's what I mean when I said "war". They openly abuse open source movement, bribe IT journalists and gadget dealers, heavily invest in a post-PC and non-Microsoft platforms and so on. Spirit of a collaboration and peace is a good thing, but war for a survival is already here and Google is a worst technology abusing domination hungry monster I ever seen in my professional life, no matter how Google is painting itself as an Empire of Good in a mass media (not unlike from a USA government strategy, USA government and Google corporation share too much similar traits, well it's just another political incorrect question). If Google will be a totally dominant winner in a platform wars we will see a much worse toxic IT world. So be brave to fight back.

@Opiumtm

This comment has been minimized.

Opiumtm commented Feb 26, 2016

@10Dev on your page

ESENT Managed Interop it is not clear if the .NET wrapper is UWP compatible

It is clear. ESENT Managed Interop is compatible with UWP. Not all nuget package are (for example, PersistentDictionary built on top of ESENT interop package is not compatible and it is certainly a problem with package manifest), but core Interop package on nuget is compatible with UWP, I tested it myself and it work.

@10Dev

This comment has been minimized.

10Dev commented Feb 26, 2016

Joseph Conrad was a native Polish speaker that somehow ended up writing some of the greatest works of English literature. Such things are rare and Optiumtm, your English is very eloquent and humbles me in that I couldn't write a paragraph of anything in another language.

Cultural differences can be quite large and form a communication gap perhaps larger than language. I would say this discussion has proceeded rather smoothly despite such gaps and is a testimony to the kindness and patience of all the participants.

I think the UWP platform is the next generation of both Silverlight and WPF and fixes many of the serious blocking issues in those previous platforms. It should be possible to harmonize the XAML and API a bit better between then and now that Microsoft has purchased Xamarin, there might be some motivation to create common platform that is more of a "superset" than the current "subset"

As it exists today UWP is mature, performant, reliable and very deployable in large applications within a few significant limitations. As we all know, the largest limitation is in the data area which is why we are here. In fact I finally got around to making a GitHub account for the sole purpose of determining the EF situation on UWP. And so I find out it is dismal. And that's frustrating and almost physically painful actually. For now it hurts Microsoft more than us since we can make work-arounds but all the enterprise project starts that switch to other technologies are lost forever. That's opportunity cost and the EF team has to juggle that as best they can for reasons we may never know. But for me, I choose not to let this apparent absurdity of a situation affect my view of the overal platform. Like the internet, we just need to route around the damaged part of the network!

@Opiumtm

This comment has been minimized.

Opiumtm commented Feb 26, 2016

@10Dev well, not all thing with UWP is such good as you described. For now UWP have one maybe corner case but very fundamental problem. In Silverlight and desktop .NET frameworks underlying operational system calls was carefully wrapped and internally optimized (just study how elaborated platform Interop in a classical .NET is). On the UWP platform things are much worse. Windows Runtime essentially is a COM with fancy winmd metadata on top. And here we got significant performance penalty when we deal with Windows Runtime calls. Even .NET Native is not good with Windows Runtime Interop from a performance point of view. And on UWP such Interop is a fundamental nature of things because all Windows Runtime API surface is exposed via COM ABI. COM/.NET Interop was never good, but in a traditional .NET vast majority of operational system calls was done via plain win32 platform invoke, not COM. And for now we have something similar to Frankenstein monster full of Interop between .NET/COM boundary suffering from a terrible performance penalty. It's not matter if you can avoid too many direct Windows Runtime calls, but File I/O and manual XAML UI manipulation in code (not rely on data templating engine and bindings, go more low level - which is unavoidable if you write very non trivial UI, custom markup rendering for a social app for example) can be painfully slow because it involve massive Interop between .NET managed code and native COM Windows Runtime API.

So, .NET Native toolchain must be improved to eliminate (or at least significantly reduce) .NET/COM Interop performance penalty. After all, Windows Runtime provide most basic and vital services, and in non trivial scenarios massive system calls are unavoidable. For now, .NET Native do not help much to reduce COM calls marshalling costs, on the other hand plain C++/CX code does not suffer from such an issue. Even on MSDN it is clearly suggested to avoid excessive marshalled calls to a Windows Runtime API. And C++/CX code is fundamentally COM based and its calls are marshalled only in a rare cross Thread Apartment scenario. So, now it is reasonable to write performance critical relatively low level code in C++ and split application into a C++ and C# parts. C# is not suited well for all scenarios and so we have no solid developer platform experience.

@Opiumtm

This comment has been minimized.

Opiumtm commented Feb 26, 2016

@10Dev to make things more clear, on a x86 desktop .NET/COM calls marshalling cost is tolerable. But on a ARM based smartphones it become a complete disaster. ARM processors are very resource constrained if compared with powerful x86 bulldozers. On ARM .NET/COM Interop penalty become a major performance issue. Such a subtle and not self obvious issue (one must inderstand internal Windows Runtime architecture and be familiar with not well known to a typical .NET developer details) must be known to all UWP library authors.

@jnm2

This comment has been minimized.

jnm2 commented Jul 24, 2016

Wow at this conversation! My respect for the EF team just went up tenfold.

To get all the way back on topic, I would love to see ESENT as a provider but I'm not sure what the difficulty level would be to implement it. Is anyone currently planning on doing this or should I start poking around when my schedule clears a bit?

@Opiumtm

This comment has been minimized.

Opiumtm commented Jul 24, 2016

@jnm2 I think, SQL providers are more difficult to implement than NoSQL providers. With ESENT challenge would be to integrate all ESENT benefits and optimizations within Entity Framework. For example, ESENT support some specific concepts like "pre-joined tables" and "tuple indexes". Also, ESENT model is very different from SQL and document oriented NoSQL. Additional challenge is to develop ISAM compatible layer. ESENT is a very advanced transactional relational storage engine, which is decoupled from SQL query language (it's not a database product, it's a foundation for database engines and is used as it by RavenDB, for example). So, ESENT is a very special case - it gives Entity Framework unique opportunity to build object-oriented persistence directly over an advanced data storage layer without SQL language overhead and its shortcomings. ESENT is ideal foundation for first-class object-oriented data storage transactional engine without any intermediate abstractions.

@ErikEJ

This comment has been minimized.

Contributor

ErikEJ commented Jul 25, 2016

@jnm2 : I am thinking about doing something, but as stated earlier, I think awaiting a NoSQL provider implemented by the EF Core team itself would in my opinion be a very good idea, to save a lot of effort.

@Opiumtm :

I think, SQL providers are more difficult to implement than NoSQL providers

I disagree, with all the work done in the core relational ef component, I believe it is actually quite simple to implement a ANSI SQL based provider - and there is currently no sample NoSQL provider implemented based on the current codebase. - And you appear to be repeating yourself for no reason...

@SleepyDeb

This comment has been minimized.

SleepyDeb commented Dec 6, 2016

SQL Query Languanges sucks, binary is the way

@Opiumtm

This comment has been minimized.

Opiumtm commented Dec 8, 2016

@ErikEJ after all, Entity Framework Core design notes state that NoSQL providers would be supported as first-class citizens. But in reality, EF Core provide very limited NoSQL support and as I can assume, EF Core architecture and codebase was made with SQL in mind and lack proper infrastructure to support NoSQL.

@rowanmiller

This comment has been minimized.

Member

rowanmiller commented Dec 8, 2016

@Opiumtm we prototyped providers for Azure Table Storage and Redis as we were building out EF Core, to make sure that main components weren't relational specific. Everything relational is split out into a separate package that each relational provider builds on. We also have the InMemory provider, which is somewhat simplistic, but not a relational provider.

That said, we fully expect that as providers for non-relational stores are built they will drive some changes in the central EF Core components to address things we didn't think of.

@Opiumtm

This comment has been minimized.

Opiumtm commented Apr 21, 2017

I have executed some benchmarks of embedded DBs for UWP platform.

On Intel Core i7 Windows 10 Desktop PC, ESENT shows much better performance than SQLite.
These results are very important too for resource constrained handheld devices or IoT.

Bulk DB inserts inside transaction:
SQLite: ~60 microseconds/insert
ESENT: ~15 microseconds/insert

Bulk random record searches by primary key:
SQLite: ~30 microseconds/search
ESENT: ~3 microseconds/search

These benchmark results are self-explanatory.

And these outstanding performance results for ESENT are accompanied by ESENT's snapshot transaction isolation, high fault-tolerance, practically unlimited scaleability, rich feature set (SQLite's feature set is virtually non-existent if compared with ESENT feature set).

@ErikEJ

This comment has been minimized.

Contributor

ErikEJ commented Apr 21, 2017

@Opiumtm Looking forward to your 3rd party provider!

@Opiumtm

This comment has been minimized.

Opiumtm commented Apr 21, 2017

@ErikEJ I have tested DBs performance using low-level APIs for each DB provider. So, no ORM was used. Just plain SQL commands for SQLite and raw ManagedEsent invokes for ESENT.

As far as I know, there is no EF provider exists for ESENT. Personally I have reasonable experience with raw ESENT APIs (using ManagedEsent lib) and have used ESENT this way in some of my projects, but have minimum knowledge about EF internals and how to develop DB providers for EF.

@sandorfr

This comment has been minimized.

sandorfr commented Feb 6, 2018

Back in the days I wrote some quick and dirty prototype for e LINQ QueryProvider and GenericRepository for esent. It's far from being production ready and making a good quality EF provider will be a much bigger job anyway. If anyone wants to tackle this I can share this code (I personnally would be more inclined to work on LiteDB)

@ajcvickers

This comment has been minimized.

Member

ajcvickers commented Feb 26, 2018

Closing this as we do not plan to implement and maintain a ESENT provider for EF Core. However, we would be supportive of a community effort to create such a provider.

@ajcvickers ajcvickers closed this Feb 26, 2018

@ajcvickers ajcvickers removed this from the Backlog milestone Feb 26, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment