New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why is DataTable/View/Set absent? #1039

Closed
FransBouma opened this Issue Feb 27, 2015 · 154 comments

Comments

@FransBouma
Collaborator

FransBouma commented Feb 27, 2015

I looked here: https://github.com/dotnet/corefx-progress and saw DataTable/View/Set isn't ported to .NET Core. IMHO this is a mistake, as DataTable and friends are a convenient way to handle untyped data at runtime, and is e.g. often used when fetching data through stored procedures as it's easy to e.g. fetch a dataset with the result of a proc which returns 3 cursors on e.g. Oracle.

I fail to see why this cornerstone of many people's data-access code is absent in a framework that targets servers.

@ellismg

This comment has been minimized.

Show comment
Hide comment
@ellismg

ellismg Feb 27, 2015

Contributor

@terrajobst When you have some free time perhaps you could give an overview about how we came up with what we think the surface area for .NET Core v1.0 is?

Contributor

ellismg commented Feb 27, 2015

@terrajobst When you have some free time perhaps you could give an overview about how we came up with what we think the surface area for .NET Core v1.0 is?

@FransBouma

This comment has been minimized.

Show comment
Hide comment
@FransBouma

FransBouma Feb 27, 2015

Collaborator

Whatever may be the reason, having no real story for how to fetch tabular data using stored procs, which is today mostly done using DbDataAdapters storing data in DataSet/Tables, is likely going to stop a lot of people migrating their code to .NET Core for the simple reason that a lot of people have to work with stored procs and the tabular data they return every day. While it might be that a datareader opened on a proc call's resultset and materializing the rows into objects is 'easy', fact is that on databases which return cursors, this isn't that easy, while DbDataAdapters solve that problem today with a few lines of code.

I do understand DataTable/View/Set and DbDataAdapter come with a fair set of interfaces you likely don't want to port now (not sure if they are, haven't checked), e.g. IBindingList, IListSource and the like, but IMHO it's a cornerstone for Data access on .NET, despite what some hip people think.

Collaborator

FransBouma commented Feb 27, 2015

Whatever may be the reason, having no real story for how to fetch tabular data using stored procs, which is today mostly done using DbDataAdapters storing data in DataSet/Tables, is likely going to stop a lot of people migrating their code to .NET Core for the simple reason that a lot of people have to work with stored procs and the tabular data they return every day. While it might be that a datareader opened on a proc call's resultset and materializing the rows into objects is 'easy', fact is that on databases which return cursors, this isn't that easy, while DbDataAdapters solve that problem today with a few lines of code.

I do understand DataTable/View/Set and DbDataAdapter come with a fair set of interfaces you likely don't want to port now (not sure if they are, haven't checked), e.g. IBindingList, IListSource and the like, but IMHO it's a cornerstone for Data access on .NET, despite what some hip people think.

@tthiery

This comment has been minimized.

Show comment
Hide comment
@tthiery

tthiery Mar 2, 2015

I want to extend the question: I understand that .Net Core is not the full .Net Framework and some features are cut out. Nevertheless, many of us rely on some features and want to migrate to .Net Core. But there are two groups: The impossible ones (e.g. WebForms / WPF) and the possible and even reasonable packages like e.g. DataTable/View, XmlSchema, SerialPort, System.Activities, etc. Some may have technical constraints, not be hip, or legally restrained but to know the status and strategy of them would be a great help for understanding this new .Net.

For these cut-out features will there be

  • separate NuGet packages outside (or in a later version) of the .Net Core?
  • and if yes, will the be created/maintained by Microsoft or should we expect that we need to go to the reference source and do it on our own? (not that this is wrong ... it is just something many employers will not pay for).
  • is there an overview of these gaps? I was surprised by XmlSchema (imagine the thousands of existing services), SerialPort (imagine all the inexpensive USB chips which create a virtual COM port) and now by DataTable. The analyzer only helps me with existing code but does not help me in first place when I have to decide on paper which framework to use.

I have not heard of a general strategy for these gaps, maybe I lost it somewhere.

tthiery commented Mar 2, 2015

I want to extend the question: I understand that .Net Core is not the full .Net Framework and some features are cut out. Nevertheless, many of us rely on some features and want to migrate to .Net Core. But there are two groups: The impossible ones (e.g. WebForms / WPF) and the possible and even reasonable packages like e.g. DataTable/View, XmlSchema, SerialPort, System.Activities, etc. Some may have technical constraints, not be hip, or legally restrained but to know the status and strategy of them would be a great help for understanding this new .Net.

For these cut-out features will there be

  • separate NuGet packages outside (or in a later version) of the .Net Core?
  • and if yes, will the be created/maintained by Microsoft or should we expect that we need to go to the reference source and do it on our own? (not that this is wrong ... it is just something many employers will not pay for).
  • is there an overview of these gaps? I was surprised by XmlSchema (imagine the thousands of existing services), SerialPort (imagine all the inexpensive USB chips which create a virtual COM port) and now by DataTable. The analyzer only helps me with existing code but does not help me in first place when I have to decide on paper which framework to use.

I have not heard of a general strategy for these gaps, maybe I lost it somewhere.

@RickStrahl

This comment has been minimized.

Show comment
Hide comment
@RickStrahl

RickStrahl Mar 3, 2015

I ran into a use case today where I needed DataTable in order to return dynamic data to a service client. JSON.NET can easily serialize data tables for example. There are no easy solutions for materializing DYNAMIC data in .NET and DataTables make it easy when you don't know what the heck you're going to get back and you need to iterate over it. It's very useful for a number of things...

I can see why DataSet/DataTable is falling out of favor. Personally I rarely use them, but for dynamic data scenarios they are often the best choice available in .NET. So I hope we get this at some point or else an alternative that makes it easier to get dynamic data that can actually be serialized.

RickStrahl commented Mar 3, 2015

I ran into a use case today where I needed DataTable in order to return dynamic data to a service client. JSON.NET can easily serialize data tables for example. There are no easy solutions for materializing DYNAMIC data in .NET and DataTables make it easy when you don't know what the heck you're going to get back and you need to iterate over it. It's very useful for a number of things...

I can see why DataSet/DataTable is falling out of favor. Personally I rarely use them, but for dynamic data scenarios they are often the best choice available in .NET. So I hope we get this at some point or else an alternative that makes it easier to get dynamic data that can actually be serialized.

@FransBouma

This comment has been minimized.

Show comment
Hide comment
@FransBouma

FransBouma Mar 5, 2015

Collaborator

There's another problem: DbConnection.GetSchema(). It's a way to obtain meta-data from the connected database and all ADO.NET providers have support for it, often with detailed schemas about metadata which can be used by data-access code to e.g. generate code on the fly for types or parameters. A lot of work has been put into this method by ADO.NET providers to make this method work on the database the provider is for: often they use detailed metadata queries so the user of the method doesn't have to.

GetSchema() returns a DataTable, as it's unknown what the types are of the rows returned as that depends on the schema requested and the ado.net provider used.

This method can't be used in .NET core if DataTable isn't there.

Collaborator

FransBouma commented Mar 5, 2015

There's another problem: DbConnection.GetSchema(). It's a way to obtain meta-data from the connected database and all ADO.NET providers have support for it, often with detailed schemas about metadata which can be used by data-access code to e.g. generate code on the fly for types or parameters. A lot of work has been put into this method by ADO.NET providers to make this method work on the database the provider is for: often they use detailed metadata queries so the user of the method doesn't have to.

GetSchema() returns a DataTable, as it's unknown what the types are of the rows returned as that depends on the schema requested and the ado.net provider used.

This method can't be used in .NET core if DataTable isn't there.

@terrajobst

This comment has been minimized.

Show comment
Hide comment
@terrajobst

terrajobst Mar 5, 2015

Member

Why is DataTable/View/Set absent?

We generally consider DataSet and friends (DataTable, DataView, TableAdapter) legacy. However, that doesn't necessarily mean that those APIs will never be available on .NET Core as we generally do support legacy APIs if they important for porting scenarios (for example, non-generic collections are also available for .NET Core).

So why are these types missing then? Simply because we're focusing on non-legacy components first and so far DataTable hasn't popped up yet.

Member

terrajobst commented Mar 5, 2015

Why is DataTable/View/Set absent?

We generally consider DataSet and friends (DataTable, DataView, TableAdapter) legacy. However, that doesn't necessarily mean that those APIs will never be available on .NET Core as we generally do support legacy APIs if they important for porting scenarios (for example, non-generic collections are also available for .NET Core).

So why are these types missing then? Simply because we're focusing on non-legacy components first and so far DataTable hasn't popped up yet.

@tthiery

This comment has been minimized.

Show comment
Hide comment
@tthiery

tthiery Mar 6, 2015

@terrajobst Answered also my question. A list of "legacy" .Net Framework technologies would be awesome (maybe including a state like "never", "maybe", "later"). I need to understand, in which technologies I can trust.

tthiery commented Mar 6, 2015

@terrajobst Answered also my question. A list of "legacy" .Net Framework technologies would be awesome (maybe including a state like "never", "maybe", "later"). I need to understand, in which technologies I can trust.

@FransBouma

This comment has been minimized.

Show comment
Hide comment
@FransBouma

FransBouma Mar 6, 2015

Collaborator

@terrajobst and what about the scenarios I brought up? You can state something is 'legacy' in your eyes but that doesn't necessarily mean use cases a lot of devs are faced with every day are simply 'gone' or have valuable alternatives (as there are no alternatives at this point).

Sure, someone can port the code from referencesource and that might look OK, but it's not, as those types aren't the same types exposed and used in the ADO.NET providers.

I don't have the feeling Microsoft really understands the seriousness of this problem.

Collaborator

FransBouma commented Mar 6, 2015

@terrajobst and what about the scenarios I brought up? You can state something is 'legacy' in your eyes but that doesn't necessarily mean use cases a lot of devs are faced with every day are simply 'gone' or have valuable alternatives (as there are no alternatives at this point).

Sure, someone can port the code from referencesource and that might look OK, but it's not, as those types aren't the same types exposed and used in the ADO.NET providers.

I don't have the feeling Microsoft really understands the seriousness of this problem.

@kazuk

This comment has been minimized.

Show comment
Hide comment
@kazuk

kazuk Mar 6, 2015

I wish an alternative to SqlDbReader GetScema before makes DataTable and DataSet etc to be legacy and discontinued

My point DataTable is needed for metadata access and some of development work in now
can not be obsolete and needs maintain works fine

kazuk commented Mar 6, 2015

I wish an alternative to SqlDbReader GetScema before makes DataTable and DataSet etc to be legacy and discontinued

My point DataTable is needed for metadata access and some of development work in now
can not be obsolete and needs maintain works fine

@robertmclaws

This comment has been minimized.

Show comment
Hide comment
@robertmclaws

robertmclaws Mar 6, 2015

@FransBouma It is a serious problem for you, because you maintain database products that depend on DataTable functionality. I bet the number of people like you vs the rest of .NET developers is like a fraction of 1%. And a small one, at that.

When rewriting a framework, some stuff gets left behind, especially anything in the edge cases. All of the cases you made are edge cases. You may spend most of your life in them, and many others might because they use your products. But they are not Microsoft's focus moving forward. DataTables are a 15 year old technology, and there has to be a better way to do some of the things you described.

There are reasons EF7 is only usable for simple next-gen ASP.NET apps, and some of the scenarios you mentioned are among those reasons.

For anyone building things that are beyond simple apps, ASP.NET v6 on .NET 4.6 is going to be the way to go. DotNet Core will not be usable for many people in v1, and that is OK. Not every single feature can come along for the ride, otherwise if everything is legacy than nothing is.

Frans, You can't tell me you couldn't come up with a better way to write DbConnection.GetSchema() that could work in your own apps for .NET Core if you really thought about it. Write it as an extension method on DbConnection and make it work with older versions of the framework by using your own objects. Can't be that hard.

robertmclaws commented Mar 6, 2015

@FransBouma It is a serious problem for you, because you maintain database products that depend on DataTable functionality. I bet the number of people like you vs the rest of .NET developers is like a fraction of 1%. And a small one, at that.

When rewriting a framework, some stuff gets left behind, especially anything in the edge cases. All of the cases you made are edge cases. You may spend most of your life in them, and many others might because they use your products. But they are not Microsoft's focus moving forward. DataTables are a 15 year old technology, and there has to be a better way to do some of the things you described.

There are reasons EF7 is only usable for simple next-gen ASP.NET apps, and some of the scenarios you mentioned are among those reasons.

For anyone building things that are beyond simple apps, ASP.NET v6 on .NET 4.6 is going to be the way to go. DotNet Core will not be usable for many people in v1, and that is OK. Not every single feature can come along for the ride, otherwise if everything is legacy than nothing is.

Frans, You can't tell me you couldn't come up with a better way to write DbConnection.GetSchema() that could work in your own apps for .NET Core if you really thought about it. Write it as an extension method on DbConnection and make it work with older versions of the framework by using your own objects. Can't be that hard.

@FransBouma

This comment has been minimized.

Show comment
Hide comment
@FransBouma

FransBouma Mar 7, 2015

Collaborator

@advancedrei

It is a serious problem for you, because you maintain database products that depend on DataTable functionality.

My tools don't depend on datatable in their core, they use datatables for meta-data retrieval in some situations (as metadata is only exposed through datatables in these situations)

I bet the number of people like you vs the rest of .NET developers is like a fraction of 1%. And a small one, at that.

Please don't make up numbers and pass them on as real. Almost every application calling stored procedures uses them, and of those applications there are many.

DataTables are a 15 year old technology, and there has to be a better way to do some of the things you described.

There's always a better way, sadly there's not one available at the moment.

There are reasons EF7 is only usable for simple next-gen ASP.NET apps, and some of the scenarios you mentioned are among those reasons.

I think you're mistaken. Plus it's besides the point.

You can't tell me you couldn't come up with a better way to write DbConnection.GetSchema() that could work in your own apps for .NET Core if you really thought about it. Write it as an extension method on DbConnection and make it work with older versions of the framework by using your own objects. Can't be that hard.

Sometimes GetSchema() is necessary, but it's just an example. You also have GetSchemaTable on a datareader, which is used in a lot more cases (as not everyone reads metadata from a schema), as it tells you what the layout of the resultset is. Essential information for creating datastructures to store resultsets of unknown layout. And yes, that's not in less than 1% of the applications out there. Try most enterprise applications which have to talk to databases with many stored procedures for example. Good luck with writing solid server side code without datatable.

Sure, like I said there's always a way, but the thing is: people will want to port code they already have to .NET core and e.g. run it on Linux. Now they can't do that.

Collaborator

FransBouma commented Mar 7, 2015

@advancedrei

It is a serious problem for you, because you maintain database products that depend on DataTable functionality.

My tools don't depend on datatable in their core, they use datatables for meta-data retrieval in some situations (as metadata is only exposed through datatables in these situations)

I bet the number of people like you vs the rest of .NET developers is like a fraction of 1%. And a small one, at that.

Please don't make up numbers and pass them on as real. Almost every application calling stored procedures uses them, and of those applications there are many.

DataTables are a 15 year old technology, and there has to be a better way to do some of the things you described.

There's always a better way, sadly there's not one available at the moment.

There are reasons EF7 is only usable for simple next-gen ASP.NET apps, and some of the scenarios you mentioned are among those reasons.

I think you're mistaken. Plus it's besides the point.

You can't tell me you couldn't come up with a better way to write DbConnection.GetSchema() that could work in your own apps for .NET Core if you really thought about it. Write it as an extension method on DbConnection and make it work with older versions of the framework by using your own objects. Can't be that hard.

Sometimes GetSchema() is necessary, but it's just an example. You also have GetSchemaTable on a datareader, which is used in a lot more cases (as not everyone reads metadata from a schema), as it tells you what the layout of the resultset is. Essential information for creating datastructures to store resultsets of unknown layout. And yes, that's not in less than 1% of the applications out there. Try most enterprise applications which have to talk to databases with many stored procedures for example. Good luck with writing solid server side code without datatable.

Sure, like I said there's always a way, but the thing is: people will want to port code they already have to .NET core and e.g. run it on Linux. Now they can't do that.

@nvivo

This comment has been minimized.

Show comment
Hide comment
@nvivo

nvivo Mar 7, 2015

@terrajobst,

Out of curiosity, is there any public information you are using to decide what is important to port or not? Mono used to use Moma to scan and publish a database of "what people really need" to help drive the implementation.

Is there any metric like that being used to help decisions, or is it based more on the direction the team want things to go?

nvivo commented Mar 7, 2015

@terrajobst,

Out of curiosity, is there any public information you are using to decide what is important to port or not? Mono used to use Moma to scan and publish a database of "what people really need" to help drive the implementation.

Is there any metric like that being used to help decisions, or is it based more on the direction the team want things to go?

@tthiery

This comment has been minimized.

Show comment
Hide comment
@tthiery

tthiery Mar 7, 2015

@FransBouma +1 for the answer. The 1% is totally unrealistic considering that EF showed up in 2011 (2/3 of .Net lifetime are before) and the consideration that DataTable was the dominant technology for data retrieval in all earlier examples. I would more go for 20% to 30% of all enterprise platforms will have dependencies to DataTable (numbers are random as well).
Nevertheless, @advancedrei is not completely wrong. He is very right that it is a wrong assumption that the .Net Core will be anytime close a reasonable migration or extension path for existing code. Primary focus are and will be for a while ASP.Net (where the EF is quite dominant in new designs) and .Net Native (which do not rely on local databases but data centers using services) as far as I understand.

@nvivo +1. I totally agree. It is a bit ... in-transparent currently (especially for a Open Source project which expects contributions). @terrajobst Like stated above ... having a forecast what might (DataTable, XmlSchema, etc) and what definitely will be never ever be in (remember the endless .. WPF/WinForms/WebForms discussions) would really be helpful.

tthiery commented Mar 7, 2015

@FransBouma +1 for the answer. The 1% is totally unrealistic considering that EF showed up in 2011 (2/3 of .Net lifetime are before) and the consideration that DataTable was the dominant technology for data retrieval in all earlier examples. I would more go for 20% to 30% of all enterprise platforms will have dependencies to DataTable (numbers are random as well).
Nevertheless, @advancedrei is not completely wrong. He is very right that it is a wrong assumption that the .Net Core will be anytime close a reasonable migration or extension path for existing code. Primary focus are and will be for a while ASP.Net (where the EF is quite dominant in new designs) and .Net Native (which do not rely on local databases but data centers using services) as far as I understand.

@nvivo +1. I totally agree. It is a bit ... in-transparent currently (especially for a Open Source project which expects contributions). @terrajobst Like stated above ... having a forecast what might (DataTable, XmlSchema, etc) and what definitely will be never ever be in (remember the endless .. WPF/WinForms/WebForms discussions) would really be helpful.

@robertmclaws

This comment has been minimized.

Show comment
Hide comment
@robertmclaws

robertmclaws Mar 7, 2015

Entity Framework came out of Object Spaces which puts it in the 2008-2009 timeframe. And the 1% I referred to was forward-looking development. How many of those 30% Enterprise projects aren't even on .NET 4.0 yet? I'd venture to say a lot.

NETCORE is not concerned with people who will never migrate anyways. It is concerned with building the next generation of apps. Not maintaining the last one. The Frameworks that have already shipped can handle that just fine.

robertmclaws commented Mar 7, 2015

Entity Framework came out of Object Spaces which puts it in the 2008-2009 timeframe. And the 1% I referred to was forward-looking development. How many of those 30% Enterprise projects aren't even on .NET 4.0 yet? I'd venture to say a lot.

NETCORE is not concerned with people who will never migrate anyways. It is concerned with building the next generation of apps. Not maintaining the last one. The Frameworks that have already shipped can handle that just fine.

@FransBouma

This comment has been minimized.

Show comment
Hide comment
@FransBouma

FransBouma Mar 7, 2015

Collaborator

@advancedrei .NET core on Linux could spark new interest in .NET from dev teams who now won't look at .NET as it's windows only but do have lots of databases with stored procs. There are a tremendous amount of database-driven applications out there which have to use stored procs (I use stored procs as an example, as it's one of the use cases for a datatable as the resultsets are often of unknown layout: as you can't determine the layout .... without a datatable ;)) and they can't pick .NET if there's not a solid scenario available for their situation.

There's little known about the ADO.NET story for .NET core, no code in that area is opened yet (to my knowledge) and how the provider model will look like is completely unknown. For server-side software that's an important point, and it now looks like v1.0 of .NET core is completely useless which IMHO wastes a tremendous opportunity to win people back to .NET.

But alas... I'm starting to repeat myself. :)

Collaborator

FransBouma commented Mar 7, 2015

@advancedrei .NET core on Linux could spark new interest in .NET from dev teams who now won't look at .NET as it's windows only but do have lots of databases with stored procs. There are a tremendous amount of database-driven applications out there which have to use stored procs (I use stored procs as an example, as it's one of the use cases for a datatable as the resultsets are often of unknown layout: as you can't determine the layout .... without a datatable ;)) and they can't pick .NET if there's not a solid scenario available for their situation.

There's little known about the ADO.NET story for .NET core, no code in that area is opened yet (to my knowledge) and how the provider model will look like is completely unknown. For server-side software that's an important point, and it now looks like v1.0 of .NET core is completely useless which IMHO wastes a tremendous opportunity to win people back to .NET.

But alas... I'm starting to repeat myself. :)

@nvivo

This comment has been minimized.

Show comment
Hide comment
@nvivo

nvivo Mar 7, 2015

@tthiery, @terrajobst

Is there any metric like that being used to help decisions?

I'm sorry. I know it sounded like I was questioning the decision, but I was not. =) I was genuinely curious how the process of selecting the API was/is done.

I have been using pure ADO.NET for years without EF, and I have not used DataTable for anything new in ages. I think this can be provided later by a package, but don't need to be in the core right now. We need to remember .NET 4.6 will still be released and all this stuff will be there.

nvivo commented Mar 7, 2015

@tthiery, @terrajobst

Is there any metric like that being used to help decisions?

I'm sorry. I know it sounded like I was questioning the decision, but I was not. =) I was genuinely curious how the process of selecting the API was/is done.

I have been using pure ADO.NET for years without EF, and I have not used DataTable for anything new in ages. I think this can be provided later by a package, but don't need to be in the core right now. We need to remember .NET 4.6 will still be released and all this stuff will be there.

@RickStrahl

This comment has been minimized.

Show comment
Hide comment
@RickStrahl

RickStrahl Mar 7, 2015

I'm with others here who point out that we should be looking forward not backward, but... I think we need to be careful here that we don't kill functionality that may be crucial to building applications.

The two things that have jumped out at me are lack of a good story for creating new providers so that third parties have an easy time to create new providers. This (or some other mechanism like standardized DI) is a MUST and currently not supported/documented. Right now you're stuck with SQL Server and third parties can build their own providers but have no effective way to swap providers at runtime (maybe there's something there that we can't see but this was what the provider model was for previously).

Personally I don't see DataTable/DataSet as needed tech, but the features it provides - an easy way to retrieve list based dynamic query DB data is a solid use case for building solutions on top of raw ADO.NET and for third parties that build on top of it. I think something is needed to provide this functionality out of box. It could be as simple as a dictionary, but it should be there natively IMHO because it is a common scenario if you use ADO.NET raw.

So to me the story isn't about backwards compatibility but making sure the use cases that are common are addressed. If that means new interfaces that's fine, as long as some fundamental scenarios that third party tools and frameworks can build on are addressed in some way.

Honestly I think this is all of utmost importance. After all most applications use data. And the story for data in vNext as presented by Microsoft so far looks like a bungling mess. I'm not sure if it actually is or not, but the message has been muddled and confused. From the outside it looks like EF with SQL Server is the only viable solution that's in box and that really sucks if that's how it ends up. Especially in light of .NET Core running on other platforms where SQL Server usage is not very likely. If this is just a messaging thing - then that needs to be corrected with some official commentary. If it's a feature thing then I think there should be discussion on what needs to be there.

RickStrahl commented Mar 7, 2015

I'm with others here who point out that we should be looking forward not backward, but... I think we need to be careful here that we don't kill functionality that may be crucial to building applications.

The two things that have jumped out at me are lack of a good story for creating new providers so that third parties have an easy time to create new providers. This (or some other mechanism like standardized DI) is a MUST and currently not supported/documented. Right now you're stuck with SQL Server and third parties can build their own providers but have no effective way to swap providers at runtime (maybe there's something there that we can't see but this was what the provider model was for previously).

Personally I don't see DataTable/DataSet as needed tech, but the features it provides - an easy way to retrieve list based dynamic query DB data is a solid use case for building solutions on top of raw ADO.NET and for third parties that build on top of it. I think something is needed to provide this functionality out of box. It could be as simple as a dictionary, but it should be there natively IMHO because it is a common scenario if you use ADO.NET raw.

So to me the story isn't about backwards compatibility but making sure the use cases that are common are addressed. If that means new interfaces that's fine, as long as some fundamental scenarios that third party tools and frameworks can build on are addressed in some way.

Honestly I think this is all of utmost importance. After all most applications use data. And the story for data in vNext as presented by Microsoft so far looks like a bungling mess. I'm not sure if it actually is or not, but the message has been muddled and confused. From the outside it looks like EF with SQL Server is the only viable solution that's in box and that really sucks if that's how it ends up. Especially in light of .NET Core running on other platforms where SQL Server usage is not very likely. If this is just a messaging thing - then that needs to be corrected with some official commentary. If it's a feature thing then I think there should be discussion on what needs to be there.

@nvivo

This comment has been minimized.

Show comment
Hide comment
@nvivo

nvivo Mar 7, 2015

I agree with @RickStrahl that EF + SqlServer is bad, and agree with @FransBouma that there is no way to do some things like just selecting untyped data without dealing with DataReader. But I don't see anything else filling this gap DataTables left.

What is really to be missed? All the select + order by is done much better by linq directly on objects. For other things, there are alternatives to EF. SimpleData is a famous one, Dapper from StackExchange is also very handy, and even I maintain a pretty useful one =).

I bet new alternatives will appear even more with .NET Core, and eventually a winner will come up. But we need to give it some time.

nvivo commented Mar 7, 2015

I agree with @RickStrahl that EF + SqlServer is bad, and agree with @FransBouma that there is no way to do some things like just selecting untyped data without dealing with DataReader. But I don't see anything else filling this gap DataTables left.

What is really to be missed? All the select + order by is done much better by linq directly on objects. For other things, there are alternatives to EF. SimpleData is a famous one, Dapper from StackExchange is also very handy, and even I maintain a pretty useful one =).

I bet new alternatives will appear even more with .NET Core, and eventually a winner will come up. But we need to give it some time.

@FransBouma

This comment has been minimized.

Show comment
Hide comment
@FransBouma

FransBouma Mar 7, 2015

Collaborator

I agree with @RickStrahl here. It's the concept / featureset provided by Datatable and friends that's essential, and if I may add, essential for v1.0 to be there. Datatable is an obvious choice to provide that featureset as a truckload of code is already written out there supporting it, but it's not essential to have it as the type to provide that featureset, however why write something else to replace something you already have?

Collaborator

FransBouma commented Mar 7, 2015

I agree with @RickStrahl here. It's the concept / featureset provided by Datatable and friends that's essential, and if I may add, essential for v1.0 to be there. Datatable is an obvious choice to provide that featureset as a truckload of code is already written out there supporting it, but it's not essential to have it as the type to provide that featureset, however why write something else to replace something you already have?

@nvivo

This comment has been minimized.

Show comment
Hide comment
@nvivo

nvivo Mar 7, 2015

@FransBouma,

Which part exactly do you miss from DataTable? It seems the only thing that you miss is a type that represents "an array of object arrays" for untyped data, but that is provided out of the box.

Is that it or am I missing something?

nvivo commented Mar 7, 2015

@FransBouma,

Which part exactly do you miss from DataTable? It seems the only thing that you miss is a type that represents "an array of object arrays" for untyped data, but that is provided out of the box.

Is that it or am I missing something?

@nvivo

This comment has been minimized.

Show comment
Hide comment
@nvivo

nvivo Mar 7, 2015

As a note, I saw the comments about the GetSchema. The point is that those things can be provided as separate modules, they are not exactly required by anything other than supporting the IDE around DataTable/Set/View stuff. And even this could be today exposed as real objects instead of datatables.

If all is needed is a way to retrieve untyped tabular data, we could strip away 90% of DataTable and have a single new type that have no change detection, no select capabilities, etc. It should only hold an array of "rows", so "an array of object arrays".

nvivo commented Mar 7, 2015

As a note, I saw the comments about the GetSchema. The point is that those things can be provided as separate modules, they are not exactly required by anything other than supporting the IDE around DataTable/Set/View stuff. And even this could be today exposed as real objects instead of datatables.

If all is needed is a way to retrieve untyped tabular data, we could strip away 90% of DataTable and have a single new type that have no change detection, no select capabilities, etc. It should only hold an array of "rows", so "an array of object arrays".

@robertmclaws

This comment has been minimized.

Show comment
Hide comment
@robertmclaws

robertmclaws Mar 7, 2015

I agree with @nvivo, a DataTable is just an array of object arrays. Not sure why the functionality can't be replicated with some custom code and without the "sky is falling" mentality.

robertmclaws commented Mar 7, 2015

I agree with @nvivo, a DataTable is just an array of object arrays. Not sure why the functionality can't be replicated with some custom code and without the "sky is falling" mentality.

@FransBouma

This comment has been minimized.

Show comment
Hide comment
@FransBouma

FransBouma Mar 8, 2015

Collaborator

@nvivo

Which part exactly do you miss from DataTable? It seems the only thing that you miss is a type that represents "an array of object arrays" for untyped data, but that is provided out of the box. Is that it or am I missing something?

DataTable isn't a construct with arrays of object arrays, it's a construct with typed columns (data isn't stored internally in object arrays, but vertically in typed columns), which has the ability to define custom views on top of it you can sort/filter without touching the original datatable, which can be filled with a DbDataAdapter which can create the typed columns for you during fetch based on metadata obtained from the resultset, which is also stored in a datatable.

Like I said, if something is replacing it with equal functionality, fine, but that's not in the cards. This means that ado.net providers can't provide the same functionality they do today. GetSchema is one thing, but as I said, DbDataReader.GetSchemaTable is another, and that one IS used a lot by user code (directly or indirectly through data-access code or dbdataadapters).

@advancedrei please stay on point with this discussion. I understand you don't give a shit about datatable and friends and that's fine, but just because it's irrelevant to you doesn't mean it's irrelevant to many thousands of developers out there.

Collaborator

FransBouma commented Mar 8, 2015

@nvivo

Which part exactly do you miss from DataTable? It seems the only thing that you miss is a type that represents "an array of object arrays" for untyped data, but that is provided out of the box. Is that it or am I missing something?

DataTable isn't a construct with arrays of object arrays, it's a construct with typed columns (data isn't stored internally in object arrays, but vertically in typed columns), which has the ability to define custom views on top of it you can sort/filter without touching the original datatable, which can be filled with a DbDataAdapter which can create the typed columns for you during fetch based on metadata obtained from the resultset, which is also stored in a datatable.

Like I said, if something is replacing it with equal functionality, fine, but that's not in the cards. This means that ado.net providers can't provide the same functionality they do today. GetSchema is one thing, but as I said, DbDataReader.GetSchemaTable is another, and that one IS used a lot by user code (directly or indirectly through data-access code or dbdataadapters).

@advancedrei please stay on point with this discussion. I understand you don't give a shit about datatable and friends and that's fine, but just because it's irrelevant to you doesn't mean it's irrelevant to many thousands of developers out there.

@nvivo

This comment has been minimized.

Show comment
Hide comment
@nvivo

nvivo Mar 8, 2015

@FransBouma,

DataTable isn't a construct with arrays of object arrays

I'm obviously not saying how DataTable is implemented, I'm saying this is the only part that is missing.

if something is replacing it with equal functionality, fine, but that's not in the cards

Don't get me wrong here. I have a lot of code that depends on GetSchemaTable and DataTables in general out there. I have data migration tools that automatically compare and move data between different providers using this functionality.

But that doesn't matter because we are not discussing removing this functionality from .NET. We are discussing if this functionality should be added to a new framework that has a different target.

You say that you want "equal" functionality, but most of what DataTable/View/Set provides can be achieved with POCO and LINQ in a much cleaner way. You can create views, group by, count, select different rows and columns, etc. And in most cases, that will be achieved with LINQ using less memory.

I believe the only way DataTable should be added to CoreFx would be with a simpler design that integrates better with LINQ and removes things like .Select, OrderBy, Views, etc. It should be just a container for tabular data, nothing more.

As someone that works a lot with DbProviderFactory, I liked how the design went. Providers are clean and do only what they need to allow database access using SQL. Everything else can be build on top of it later, and that's how it should be.

Version 1.0 is not even out yet, and @advancedrei said already, this version won't have a lot of stuff for a lot of people. But it's better to start clean and add pieces as it goes than start bloated.

nvivo commented Mar 8, 2015

@FransBouma,

DataTable isn't a construct with arrays of object arrays

I'm obviously not saying how DataTable is implemented, I'm saying this is the only part that is missing.

if something is replacing it with equal functionality, fine, but that's not in the cards

Don't get me wrong here. I have a lot of code that depends on GetSchemaTable and DataTables in general out there. I have data migration tools that automatically compare and move data between different providers using this functionality.

But that doesn't matter because we are not discussing removing this functionality from .NET. We are discussing if this functionality should be added to a new framework that has a different target.

You say that you want "equal" functionality, but most of what DataTable/View/Set provides can be achieved with POCO and LINQ in a much cleaner way. You can create views, group by, count, select different rows and columns, etc. And in most cases, that will be achieved with LINQ using less memory.

I believe the only way DataTable should be added to CoreFx would be with a simpler design that integrates better with LINQ and removes things like .Select, OrderBy, Views, etc. It should be just a container for tabular data, nothing more.

As someone that works a lot with DbProviderFactory, I liked how the design went. Providers are clean and do only what they need to allow database access using SQL. Everything else can be build on top of it later, and that's how it should be.

Version 1.0 is not even out yet, and @advancedrei said already, this version won't have a lot of stuff for a lot of people. But it's better to start clean and add pieces as it goes than start bloated.

@FransBouma

This comment has been minimized.

Show comment
Hide comment
@FransBouma

FransBouma Mar 8, 2015

Collaborator

@nvivo

You say that you want "equal" functionality, but most of what DataTable/View/Set provides can be achieved with POCO and LINQ in a much cleaner way. You can create views, group by, count, select different rows and columns, etc. And in most cases, that will be achieved with LINQ using less memory.

You can't create POCOs if you don't have meta-data of the fields of the resultset. You can't determine the schema of a resultset of a stored procedure unless GetSchemaTable() is implemented. Again, if that's implemented but it gives me a typed object, no problem. However if it returns an array of object arrays, what am I going to do with that? What element is what and of what type?

I'm not talking about the POCO/typed element route which is created from metadata obtained in some way, I'm talking about calling a stored procedure which returns a resultset and you now have to do something with that resultset.

Anyway I have given enough examples, @RickStrahl has given additional examples and background of why this feature is very important. I simply don't get why one would want to be limited by excluding functionality which is already implemented and has worked for over a decade.

Collaborator

FransBouma commented Mar 8, 2015

@nvivo

You say that you want "equal" functionality, but most of what DataTable/View/Set provides can be achieved with POCO and LINQ in a much cleaner way. You can create views, group by, count, select different rows and columns, etc. And in most cases, that will be achieved with LINQ using less memory.

You can't create POCOs if you don't have meta-data of the fields of the resultset. You can't determine the schema of a resultset of a stored procedure unless GetSchemaTable() is implemented. Again, if that's implemented but it gives me a typed object, no problem. However if it returns an array of object arrays, what am I going to do with that? What element is what and of what type?

I'm not talking about the POCO/typed element route which is created from metadata obtained in some way, I'm talking about calling a stored procedure which returns a resultset and you now have to do something with that resultset.

Anyway I have given enough examples, @RickStrahl has given additional examples and background of why this feature is very important. I simply don't get why one would want to be limited by excluding functionality which is already implemented and has worked for over a decade.

@nvivo

This comment has been minimized.

Show comment
Hide comment
@nvivo

nvivo Mar 8, 2015

@FransBouma,

Everything you said is perfectly achievable if you change your mind model.

You can't determine the schema of a resultset of a stored procedure unless GetSchemaTable() is implemented.

Why is that? Any database nowadays implements INFORMATION_SCHEMA to provide this kind of information in a standard way. If you need something very specific, you can just run any query against the database.

I'm not talking about the POCO/typed element route which is created from metadata obtained in some way

So, you are saying that if you ignore all the other ways to do it, there is no way to do it?

DataTable is a huge implementation that is intended to do a lot of things that are not that common anymore. I have been building commercial apps for ages with plain ADO.NET and didn't use DataTables for a long time, I see them as legacy as well. Most frameworks and controls today focus on objects with observables and other ways to track changes.

I do believe it has some features that are still valuable and have no replacement, but we need find what those features are and try to come up with an alternative for those only. Trying to bring this huge implementation full of duplicated stuff that is easily achievable with LINQ is not a good option.

nvivo commented Mar 8, 2015

@FransBouma,

Everything you said is perfectly achievable if you change your mind model.

You can't determine the schema of a resultset of a stored procedure unless GetSchemaTable() is implemented.

Why is that? Any database nowadays implements INFORMATION_SCHEMA to provide this kind of information in a standard way. If you need something very specific, you can just run any query against the database.

I'm not talking about the POCO/typed element route which is created from metadata obtained in some way

So, you are saying that if you ignore all the other ways to do it, there is no way to do it?

DataTable is a huge implementation that is intended to do a lot of things that are not that common anymore. I have been building commercial apps for ages with plain ADO.NET and didn't use DataTables for a long time, I see them as legacy as well. Most frameworks and controls today focus on objects with observables and other ways to track changes.

I do believe it has some features that are still valuable and have no replacement, but we need find what those features are and try to come up with an alternative for those only. Trying to bring this huge implementation full of duplicated stuff that is easily achievable with LINQ is not a good option.

@FransBouma

This comment has been minimized.

Show comment
Hide comment
@FransBouma

FransBouma Mar 8, 2015

Collaborator

Everything you said is perfectly achievable if you change your mind model.

It's not for me, it's for all those devs out there who can't proceed to .NET core because someone thought 'Datatable' is legacy while not considering the consequences of that decision.

You can't determine the schema of a resultset of a stored procedure unless GetSchemaTable() is implemented.

Why is that? Any database nowadays implements INFORMATION_SCHEMA to provide this kind of information in a standard way. If you need something very specific, you can just run any query against the database.

  1. Not every database implements all schemas of information_schema
  2. Not every database is 100% bugfree in those schemas (e.g. sql server)
  3. For a resultset from a stored procedure, e.g. the direct resultset like with sql server or the resultset of a cursor like with Oracle or PostgreSql, the schema isn't in one of those schemas, but is determined on the fly, as the resultset is dynamic. If you don't know what you're talking about, please don't derail this discussion with nonsense as it's important for people who have to work with DataTable and friends.
  4. I've written meta-data retrieval drivers for a lot of databases and for a lot of different meta-data elements. Trust me on this that INFORMATION_SCHEMA is not that helpful, if helpful at all.

The DbConnection.GetSchema() call is an example, not the use case. There are many more examples to give where DataTables are used today and are very useful. I have given a couple, others have given other examples.

DataTable is a huge implementation that is intended to do a lot of things that are not that common anymore. I have been building commercial apps for ages with plain ADO.NET and didn't use DataTables for a long time, I see them as legacy as well. Most frameworks and controls today focus on objects with observables and other ways to track changes.

Common in your world != common in everyone's world. I write ORMs for a living, I know DataTables are not the only way to do data-access. I also know that everything that even looks like a stored procedure is automatically useless if datatables are not in the framework anymore.

I do believe it has some features that are still valuable and have no replacement, but we need find what those features are and try to come up with an alternative for those only.

Why come up with alternatives for code that has already been proven to work for over a decade and which is used by a lot of people, which is supported today in ADO.NET providers (so when these are ported to .NET core, this code will work automatically).

Trying to bring this huge implementation full of duplicated stuff that is easily achievable with LINQ is not a good option.

You're not focusing on the real problem. I'm not talking about creating a projection of a resultset to a typed object, I'm talking about being able to store untyped resultsets in a datastructure which makes it possible to use that untyped resultset without knowing what the layout is (as e.g. that might differ based in input, some stored procedures are written that way).

Anyway, I've done my best. If MS wants to shoot themselves in the foot by limiting the Data-access story of their .NET core framework to 'whatever EF7 can do', so be it.

Collaborator

FransBouma commented Mar 8, 2015

Everything you said is perfectly achievable if you change your mind model.

It's not for me, it's for all those devs out there who can't proceed to .NET core because someone thought 'Datatable' is legacy while not considering the consequences of that decision.

You can't determine the schema of a resultset of a stored procedure unless GetSchemaTable() is implemented.

Why is that? Any database nowadays implements INFORMATION_SCHEMA to provide this kind of information in a standard way. If you need something very specific, you can just run any query against the database.

  1. Not every database implements all schemas of information_schema
  2. Not every database is 100% bugfree in those schemas (e.g. sql server)
  3. For a resultset from a stored procedure, e.g. the direct resultset like with sql server or the resultset of a cursor like with Oracle or PostgreSql, the schema isn't in one of those schemas, but is determined on the fly, as the resultset is dynamic. If you don't know what you're talking about, please don't derail this discussion with nonsense as it's important for people who have to work with DataTable and friends.
  4. I've written meta-data retrieval drivers for a lot of databases and for a lot of different meta-data elements. Trust me on this that INFORMATION_SCHEMA is not that helpful, if helpful at all.

The DbConnection.GetSchema() call is an example, not the use case. There are many more examples to give where DataTables are used today and are very useful. I have given a couple, others have given other examples.

DataTable is a huge implementation that is intended to do a lot of things that are not that common anymore. I have been building commercial apps for ages with plain ADO.NET and didn't use DataTables for a long time, I see them as legacy as well. Most frameworks and controls today focus on objects with observables and other ways to track changes.

Common in your world != common in everyone's world. I write ORMs for a living, I know DataTables are not the only way to do data-access. I also know that everything that even looks like a stored procedure is automatically useless if datatables are not in the framework anymore.

I do believe it has some features that are still valuable and have no replacement, but we need find what those features are and try to come up with an alternative for those only.

Why come up with alternatives for code that has already been proven to work for over a decade and which is used by a lot of people, which is supported today in ADO.NET providers (so when these are ported to .NET core, this code will work automatically).

Trying to bring this huge implementation full of duplicated stuff that is easily achievable with LINQ is not a good option.

You're not focusing on the real problem. I'm not talking about creating a projection of a resultset to a typed object, I'm talking about being able to store untyped resultsets in a datastructure which makes it possible to use that untyped resultset without knowing what the layout is (as e.g. that might differ based in input, some stored procedures are written that way).

Anyway, I've done my best. If MS wants to shoot themselves in the foot by limiting the Data-access story of their .NET core framework to 'whatever EF7 can do', so be it.

@HaloFour

This comment has been minimized.

Show comment
Hide comment
@HaloFour

HaloFour Mar 8, 2015

After reading through this thread I tried to find out what of ADO.NET is being brought over to CoreFX (or at least would be supported on top of it) and I couldn't really find anything. Are connections, commands and readers also getting the axe (or at least deferred until after the first release)? Doesn't EF7 depend on these things to function?

A server core framework for web applications that doesn't support DB access doesn't sound very useful to me.

I can't honestly say that I've used DataTables and their kin recently but I do use data readers frequently when I need both firehose performance and/or need to avoid the inherent limitations of ORMs. I also have a metric ton of extension methods around DbConnection, DbCommand and DbDataReader to vastly simplify their use which I'd love to propose for potential inclusion/discussion.

Update: My apologies, apparently I can't read. I do see the progress page which includes the types in System.Data.Common.

HaloFour commented Mar 8, 2015

After reading through this thread I tried to find out what of ADO.NET is being brought over to CoreFX (or at least would be supported on top of it) and I couldn't really find anything. Are connections, commands and readers also getting the axe (or at least deferred until after the first release)? Doesn't EF7 depend on these things to function?

A server core framework for web applications that doesn't support DB access doesn't sound very useful to me.

I can't honestly say that I've used DataTables and their kin recently but I do use data readers frequently when I need both firehose performance and/or need to avoid the inherent limitations of ORMs. I also have a metric ton of extension methods around DbConnection, DbCommand and DbDataReader to vastly simplify their use which I'd love to propose for potential inclusion/discussion.

Update: My apologies, apparently I can't read. I do see the progress page which includes the types in System.Data.Common.

@nvivo

This comment has been minimized.

Show comment
Hide comment
@nvivo

nvivo Mar 8, 2015

@HaloFour, check for System.Data.Common under https://github.com/dotnet/corefx-progress/tree/master/src-diff/README.md.

Specifically, check the definition of DbProviderFactory that defines what a Data Provider is.

Basically it was reduced to connection, command, transaction and datareader.

It seems all that is needed to me.

nvivo commented Mar 8, 2015

@HaloFour, check for System.Data.Common under https://github.com/dotnet/corefx-progress/tree/master/src-diff/README.md.

Specifically, check the definition of DbProviderFactory that defines what a Data Provider is.

Basically it was reduced to connection, command, transaction and datareader.

It seems all that is needed to me.

@nvivo

This comment has been minimized.

Show comment
Hide comment
@nvivo

nvivo Mar 9, 2015

@FransBouma,

It's not for me, it's for all those devs out there who can't proceed to .NET core

Who are you? Superman? Let's be honest here, any issue we open is because some personal need. But please bring some of those oppressed developers to the discussion, so they can participate, maybe having more points of view on this would help.

Not every database implements all schemas of information_schema

Most schemas provided by .NET are just queries to these tables. I have also worked a lot with those, and there are huge differences between different providers to the point that unless you just want a list of tables, you have to write different code to get what you want anyways. Why should .NET add another level of indirection? If the database can't provide this information in a sane way, what guarantees you have that the driver will?

You may argue this is useful in some cases, but is not required to have a data provider, and it's not required to use procedures in any way, even though you insist it's impossible to execute a store procedure without it, which I honestly don't get it as I have used them without that since forever.

You're not focusing on the real problem. I'm not talking about creating a projection of a resultset to a typed object, I'm talking about being able to store untyped resultsets in a datastructure which makes it possible to use that untyped resultset without knowing what the layout

Back to my point, we need to understand exactly which part of the DataTable stuff is missing and address this specifically. The things you are talking about don't need to include change tracking, select, orderby, table relationships, views, etc. This should be addressable with a simple container for tabular data, and it shouldn't be that hard to build that and even include in .NET later.

nvivo commented Mar 9, 2015

@FransBouma,

It's not for me, it's for all those devs out there who can't proceed to .NET core

Who are you? Superman? Let's be honest here, any issue we open is because some personal need. But please bring some of those oppressed developers to the discussion, so they can participate, maybe having more points of view on this would help.

Not every database implements all schemas of information_schema

Most schemas provided by .NET are just queries to these tables. I have also worked a lot with those, and there are huge differences between different providers to the point that unless you just want a list of tables, you have to write different code to get what you want anyways. Why should .NET add another level of indirection? If the database can't provide this information in a sane way, what guarantees you have that the driver will?

You may argue this is useful in some cases, but is not required to have a data provider, and it's not required to use procedures in any way, even though you insist it's impossible to execute a store procedure without it, which I honestly don't get it as I have used them without that since forever.

You're not focusing on the real problem. I'm not talking about creating a projection of a resultset to a typed object, I'm talking about being able to store untyped resultsets in a datastructure which makes it possible to use that untyped resultset without knowing what the layout

Back to my point, we need to understand exactly which part of the DataTable stuff is missing and address this specifically. The things you are talking about don't need to include change tracking, select, orderby, table relationships, views, etc. This should be addressable with a simple container for tabular data, and it shouldn't be that hard to build that and even include in .NET later.

@FransBouma

This comment has been minimized.

Show comment
Hide comment
@FransBouma

FransBouma Mar 9, 2015

Collaborator

@nvivo

It's not for me, it's for all those devs out there who can't proceed to .NET core

Who are you? Superman? Let's be honest here, any issue we open is because some personal need. But please bring some of those oppressed developers to the discussion, so they can participate, maybe having more points of view on this would help.

What does this add to answering my question, which wasn't directed to you at all btw. Is it necessary here to explain if I know what I'm talking about? It seems so. As if I don't know the field I'm specialized in and spend full time in.

Not every database implements all schemas of information_schema

Most schemas provided by .NET are just queries to these tables. I have also worked a lot with those, and there are huge differences between different providers to the point that unless you just want a list of tables, you have to write different code to get what you want anyways. Why should .NET add another level of indirection? If the database can't provide this information in a sane way, what guarantees you have that the driver will?

connection.GetSchema() gives also schemas about the type mappings provided by the ado.net provider, and an easy to use interface to otherwise complicated queries. But it's an example of where datatable's are used. There are many more.

You may argue this is useful in some cases, but is not required to have a data provider, and it's not required to use procedures in any way, even though you insist it's impossible to execute a store procedure without it, which I honestly don't get it as I have used them without that since forever.

sigh. of course you can execute a stored procedure, consuming the resultset returned by it in any reasonable way isn't possible, as the only interface you'll get in .NET core is DbDataReader.GetValues(). Good luck with that. Before you say 'but I know the schema', you don't: you can't determine the resultset from the proc. You might be able to determine the field names but no types. If a field in the object array returned from GetValues() is DBNull.Value, you can't determine the type of it. No metadata schema in the DB will help you there.

But of course, you did that many times over without problems.

Back to my point, we need to understand exactly which part of the DataTable stuff is missing and address this specifically.

'we'? Who are 'we' here? I don't see how you should understand anything. I asked a question to Microsoft and I got the dreaded 'it's legacy' excuse I've heard too many times before already. Other people have also asked questions in this thread without any answer at all.

I've described which parts are missing and others have too. That you don't get that, that's fine but irrelevant: you weren't the person I addressed in my initial question.

The things you are talking about don't need to include change tracking, select, orderby, table relationships, views, etc. This should be addressable with a simple container for tabular data, and it shouldn't be that hard to build that and even include in .NET later.

Yes, let's rewrite something from scratch to avoid porting perfectly fine code over we already have laying around. That's time well spent! :/ Especially because your 'simple container for tabular data' will very likely look like a datatable, which is already highly optimized. But again, whatever is added is fine as long as it addresses what's missing, and as there are millions of lines of code out there already working with datatable which can't be ported to .NET core without it (but of course you won't see the need of that at all, as you have done things differently many times over) I don't see why on earth something else has to be written from scratch while there's already a class which does this.

If this is the future of Microsoft Open Source, where asking a simple important question results in a serious lack of answers from MS coupled to defending the question to people who come in and think they have to show their lack of knowledge, it'll be a fun place to be, for sure.

Collaborator

FransBouma commented Mar 9, 2015

@nvivo

It's not for me, it's for all those devs out there who can't proceed to .NET core

Who are you? Superman? Let's be honest here, any issue we open is because some personal need. But please bring some of those oppressed developers to the discussion, so they can participate, maybe having more points of view on this would help.

What does this add to answering my question, which wasn't directed to you at all btw. Is it necessary here to explain if I know what I'm talking about? It seems so. As if I don't know the field I'm specialized in and spend full time in.

Not every database implements all schemas of information_schema

Most schemas provided by .NET are just queries to these tables. I have also worked a lot with those, and there are huge differences between different providers to the point that unless you just want a list of tables, you have to write different code to get what you want anyways. Why should .NET add another level of indirection? If the database can't provide this information in a sane way, what guarantees you have that the driver will?

connection.GetSchema() gives also schemas about the type mappings provided by the ado.net provider, and an easy to use interface to otherwise complicated queries. But it's an example of where datatable's are used. There are many more.

You may argue this is useful in some cases, but is not required to have a data provider, and it's not required to use procedures in any way, even though you insist it's impossible to execute a store procedure without it, which I honestly don't get it as I have used them without that since forever.

sigh. of course you can execute a stored procedure, consuming the resultset returned by it in any reasonable way isn't possible, as the only interface you'll get in .NET core is DbDataReader.GetValues(). Good luck with that. Before you say 'but I know the schema', you don't: you can't determine the resultset from the proc. You might be able to determine the field names but no types. If a field in the object array returned from GetValues() is DBNull.Value, you can't determine the type of it. No metadata schema in the DB will help you there.

But of course, you did that many times over without problems.

Back to my point, we need to understand exactly which part of the DataTable stuff is missing and address this specifically.

'we'? Who are 'we' here? I don't see how you should understand anything. I asked a question to Microsoft and I got the dreaded 'it's legacy' excuse I've heard too many times before already. Other people have also asked questions in this thread without any answer at all.

I've described which parts are missing and others have too. That you don't get that, that's fine but irrelevant: you weren't the person I addressed in my initial question.

The things you are talking about don't need to include change tracking, select, orderby, table relationships, views, etc. This should be addressable with a simple container for tabular data, and it shouldn't be that hard to build that and even include in .NET later.

Yes, let's rewrite something from scratch to avoid porting perfectly fine code over we already have laying around. That's time well spent! :/ Especially because your 'simple container for tabular data' will very likely look like a datatable, which is already highly optimized. But again, whatever is added is fine as long as it addresses what's missing, and as there are millions of lines of code out there already working with datatable which can't be ported to .NET core without it (but of course you won't see the need of that at all, as you have done things differently many times over) I don't see why on earth something else has to be written from scratch while there's already a class which does this.

If this is the future of Microsoft Open Source, where asking a simple important question results in a serious lack of answers from MS coupled to defending the question to people who come in and think they have to show their lack of knowledge, it'll be a fun place to be, for sure.

@nvivo

This comment has been minimized.

Show comment
Hide comment
@nvivo

nvivo Mar 9, 2015

'we'? Who are 'we' here? I don't see how you should understand anything. I asked a question to Microsoft

This is an open source channel to discuss features of an open source project with interested people. If you wanna talk to Microsoft only, you should call their support.

nvivo commented Mar 9, 2015

'we'? Who are 'we' here? I don't see how you should understand anything. I asked a question to Microsoft

This is an open source channel to discuss features of an open source project with interested people. If you wanna talk to Microsoft only, you should call their support.

@FransBouma

This comment has been minimized.

Show comment
Hide comment
@FransBouma

FransBouma Mar 9, 2015

Collaborator

To add: it's already frustratingly difficult enough to convince MS to include something or to change something. It's even harder if people who have no stake in the matter come in and try to defend what MS has apparently decided. If this gets included, nothing will change for you. You won't lose any time, code, hair whatever. Nothing. However people who do rely on this feature are helped. Yes, I asked this question on their behalf because I spotted the absence of DataTable, and most people likely won't look at .NET core till it's v1.0 Adding it then is much harder than when v1.0 hasn't been released.

I must say this whole debate has been very frustrating for me. I asked a serious question and got 1) a standard excuse from MS without any reasoning behind it and 2) have to defend why I even dared to ask the question to people without any stake in the matter.

From the lack of response from the .NET team I conclude that unless IBM or other big corp calls MS and asks them wtf they were thinking by not including datatable, we won't see it in v1. This is a shame, but at the end of the day, it's MS' responsibility to provide value for everyone.

This is an open source channel to discuss features of an open source project with interested people. If you wanna talk to Microsoft only, you should call their support.

Yes, but you're defending their decision while you have no stake in the matter: if you are or aren't convinced datatable should be added is irrelevant. All your posts do is make this whole debate a matter of whether you are convinced or not, but it should be whether Microsoft is convinced or not. This derails the discussion as it centers around what you think, which is irrelevant: you don't decide what's included in .NET core, MS does.

So again, unless you work for MS and have a stake in the matter, please stay out of the debate; that you aren't convinced it's useful because you don't use it is not an argument. I don't write websites, it doesn't make ASPNET not needed.

Collaborator

FransBouma commented Mar 9, 2015

To add: it's already frustratingly difficult enough to convince MS to include something or to change something. It's even harder if people who have no stake in the matter come in and try to defend what MS has apparently decided. If this gets included, nothing will change for you. You won't lose any time, code, hair whatever. Nothing. However people who do rely on this feature are helped. Yes, I asked this question on their behalf because I spotted the absence of DataTable, and most people likely won't look at .NET core till it's v1.0 Adding it then is much harder than when v1.0 hasn't been released.

I must say this whole debate has been very frustrating for me. I asked a serious question and got 1) a standard excuse from MS without any reasoning behind it and 2) have to defend why I even dared to ask the question to people without any stake in the matter.

From the lack of response from the .NET team I conclude that unless IBM or other big corp calls MS and asks them wtf they were thinking by not including datatable, we won't see it in v1. This is a shame, but at the end of the day, it's MS' responsibility to provide value for everyone.

This is an open source channel to discuss features of an open source project with interested people. If you wanna talk to Microsoft only, you should call their support.

Yes, but you're defending their decision while you have no stake in the matter: if you are or aren't convinced datatable should be added is irrelevant. All your posts do is make this whole debate a matter of whether you are convinced or not, but it should be whether Microsoft is convinced or not. This derails the discussion as it centers around what you think, which is irrelevant: you don't decide what's included in .NET core, MS does.

So again, unless you work for MS and have a stake in the matter, please stay out of the debate; that you aren't convinced it's useful because you don't use it is not an argument. I don't write websites, it doesn't make ASPNET not needed.

@nvivo

This comment has been minimized.

Show comment
Hide comment
@nvivo

nvivo Mar 9, 2015

Oh, for crying out loud! Stop complaining already. You didn't provide a single point on why any of these things can't be done in any other way. All I see is that you are losing your favorite tool and you don't want to learn anything new.

Again, this is not a microsoft support channel. This is an open source project and anyone can discuss here anything related to the future of the platform. Get over it.

nvivo commented Mar 9, 2015

Oh, for crying out loud! Stop complaining already. You didn't provide a single point on why any of these things can't be done in any other way. All I see is that you are losing your favorite tool and you don't want to learn anything new.

Again, this is not a microsoft support channel. This is an open source project and anyone can discuss here anything related to the future of the platform. Get over it.

@FransBouma

This comment has been minimized.

Show comment
Hide comment
@FransBouma

FransBouma Mar 9, 2015

Collaborator

Oh, for crying out loud! Stop complaining already. You didn't provide a single point on why any of these things can't be done in any other way. All I see is that you are losing your favorite tool and you don't want to learn anything new.

Sorry? My framework doesn't even rely on datatable. (only in the pipeline which projects to datatable, which is an additional feature). It's an O/R mapper, remember?

And I don't want to learn anything new? Where is that the case? I'm a software dev for over 21 years now. If I don't want to learn anything new I'd still be writing C on a Sun workstation. But thanks again for posting an offtopic post.

Again, this is not a microsoft support channel. This is an open source project and anyone can discuss here anything related to the future of the platform. Get over it.

I don't think you grasp the concept of why I asked this question. Here's a hint: it's not for my own code, I don't write LoB apps, it's not for you, it's not for the other 'I don't use it so it's garbage' intellectuals here, it's for the people who don't read this thread and don't know DataTable isn't supported and will be faced with a problem when v1.0 is released. That is of course irrelevant to you as you couldn't care less about these people. Duly noted.

Collaborator

FransBouma commented Mar 9, 2015

Oh, for crying out loud! Stop complaining already. You didn't provide a single point on why any of these things can't be done in any other way. All I see is that you are losing your favorite tool and you don't want to learn anything new.

Sorry? My framework doesn't even rely on datatable. (only in the pipeline which projects to datatable, which is an additional feature). It's an O/R mapper, remember?

And I don't want to learn anything new? Where is that the case? I'm a software dev for over 21 years now. If I don't want to learn anything new I'd still be writing C on a Sun workstation. But thanks again for posting an offtopic post.

Again, this is not a microsoft support channel. This is an open source project and anyone can discuss here anything related to the future of the platform. Get over it.

I don't think you grasp the concept of why I asked this question. Here's a hint: it's not for my own code, I don't write LoB apps, it's not for you, it's not for the other 'I don't use it so it's garbage' intellectuals here, it's for the people who don't read this thread and don't know DataTable isn't supported and will be faced with a problem when v1.0 is released. That is of course irrelevant to you as you couldn't care less about these people. Duly noted.

@Tragetaschen

This comment has been minimized.

Show comment
Hide comment
@Tragetaschen

Tragetaschen Mar 9, 2015

I'm a bit confused: .NET Core is about factoring the .NET Framework into individual, small and focused Nuget packages. So if anything, there will be such a separate package for DataTable and friends. A v1 Release of .NET core means the underlying bits (corlib, JIT, etc.) are done and some Nuget packages are released as well. These packages are currently driven by the ASP.NET 5 efforts.

I don't get why it would be a problem when (as well as many other parts of the framework) DataTable wouldn't be available yet when v1 is pushed out. This is something that can be easily added on a completely independent time base.

Tragetaschen commented Mar 9, 2015

I'm a bit confused: .NET Core is about factoring the .NET Framework into individual, small and focused Nuget packages. So if anything, there will be such a separate package for DataTable and friends. A v1 Release of .NET core means the underlying bits (corlib, JIT, etc.) are done and some Nuget packages are released as well. These packages are currently driven by the ASP.NET 5 efforts.

I don't get why it would be a problem when (as well as many other parts of the framework) DataTable wouldn't be available yet when v1 is pushed out. This is something that can be easily added on a completely independent time base.

@nvivo

This comment has been minimized.

Show comment
Hide comment
@nvivo

nvivo Mar 9, 2015

@Tragetaschen,

The issue is that removing DataTable causes some other parts of the framework to be removed, like GetSchemaTable and a type that can hold untyped data offline with no need to deal with datareader.

Those things can be added later as you said, and may impact some people but there are alternatives today.

As @FransBouma repeatedly said, he doesn't use these features neither it affects his projects. But he is worried that some other people will have their lives devastated by not having this specific support in v 1.0 for some reason.

I think it would be much more useful to discuss things that really affect someone so we can discuss alternatives and even create new proposals to add those things back in a saner way, instead of trying to bring the entire DataTable bundle to corefx.

nvivo commented Mar 9, 2015

@Tragetaschen,

The issue is that removing DataTable causes some other parts of the framework to be removed, like GetSchemaTable and a type that can hold untyped data offline with no need to deal with datareader.

Those things can be added later as you said, and may impact some people but there are alternatives today.

As @FransBouma repeatedly said, he doesn't use these features neither it affects his projects. But he is worried that some other people will have their lives devastated by not having this specific support in v 1.0 for some reason.

I think it would be much more useful to discuss things that really affect someone so we can discuss alternatives and even create new proposals to add those things back in a saner way, instead of trying to bring the entire DataTable bundle to corefx.

@redowl3

This comment has been minimized.

Show comment
Hide comment
@redowl3

redowl3 Aug 29, 2016

We've tried adding net45 framework and adding System.Data to the supported framework assemblies but that caused an issue with other dependencies that don't support that framework?

redowl3 commented Aug 29, 2016

We've tried adding net45 framework and adding System.Data to the supported framework assemblies but that caused an issue with other dependencies that don't support that framework?

@benaadams

This comment has been minimized.

Show comment
Hide comment
@benaadams

benaadams Aug 29, 2016

Collaborator

Try net451 its the earliest framework aspnet core supports

Collaborator

benaadams commented Aug 29, 2016

Try net451 its the earliest framework aspnet core supports

@redowl3

This comment has been minimized.

Show comment
Hide comment
@redowl3

redowl3 commented Aug 29, 2016

Thanks

@MelbourneDeveloper

This comment has been minimized.

Show comment
Hide comment
@MelbourneDeveloper

MelbourneDeveloper Aug 30, 2016

We shouldn't have to do this, but instead of us all howling to get this done. Why doesn't someone who has the time, export the code from System.Data using ILSpy, and then bring the necessary code for SqlDataAdapter to compile? The code is all there. You can export the whole lot with ILSpy. It wouldn't take much to get it compiling in .NET Core. Just exclude all the unnecessary code for now.

Maybe someone wants to set up a repo on GitHub somewhere and others could contribute until it's done?

MelbourneDeveloper commented Aug 30, 2016

We shouldn't have to do this, but instead of us all howling to get this done. Why doesn't someone who has the time, export the code from System.Data using ILSpy, and then bring the necessary code for SqlDataAdapter to compile? The code is all there. You can export the whole lot with ILSpy. It wouldn't take much to get it compiling in .NET Core. Just exclude all the unnecessary code for now.

Maybe someone wants to set up a repo on GitHub somewhere and others could contribute until it's done?

@MelbourneDeveloper

This comment has been minimized.

Show comment
Hide comment
@MelbourneDeveloper

MelbourneDeveloper Aug 30, 2016

Well, actually, DataTable is even simpler. It would take only a couple of hours to pull in to Core and then you could write your own adapters...

MelbourneDeveloper commented Aug 30, 2016

Well, actually, DataTable is even simpler. It would take only a couple of hours to pull in to Core and then you could write your own adapters...

@benaadams

This comment has been minimized.

Show comment
Hide comment
@benaadams

benaadams Aug 30, 2016

Collaborator

Why doesn't someone who has the time, export the code from System.Data using ILSpy

Well you might want to use the MIT licenced reference source instead; just saying...

Also its already on github https://github.com/Microsoft/referencesource/blob/master/System.Data/System/Data/DataTable.cs

Collaborator

benaadams commented Aug 30, 2016

Why doesn't someone who has the time, export the code from System.Data using ILSpy

Well you might want to use the MIT licenced reference source instead; just saying...

Also its already on github https://github.com/Microsoft/referencesource/blob/master/System.Data/System/Data/DataTable.cs

@MelbourneDeveloper

This comment has been minimized.

Show comment
Hide comment
@MelbourneDeveloper

MelbourneDeveloper Aug 30, 2016

In that case, what are we even worried about?

MelbourneDeveloper commented Aug 30, 2016

In that case, what are we even worried about?

@FransBouma

This comment has been minimized.

Show comment
Hide comment
@FransBouma

FransBouma Aug 30, 2016

Collaborator

@MelbourneDeveloper It might be a good idea to read this whole thread to know why it's not just the datatable class itself, but more importantly the ADO.NET api which exposes DataTable. As these methods are absent on .NET Core, you can port the datatable class all you want, but it won't get you anywhere. :) (and porting it is more time consuming than you might think as it comes with dataview, which contains a lot of databinding code, all of which won't work on .net core)

Collaborator

FransBouma commented Aug 30, 2016

@MelbourneDeveloper It might be a good idea to read this whole thread to know why it's not just the datatable class itself, but more importantly the ADO.NET api which exposes DataTable. As these methods are absent on .NET Core, you can port the datatable class all you want, but it won't get you anywhere. :) (and porting it is more time consuming than you might think as it comes with dataview, which contains a lot of databinding code, all of which won't work on .net core)

@MelbourneDeveloper

This comment has been minimized.

Show comment
Hide comment
@MelbourneDeveloper

MelbourneDeveloper Aug 31, 2016

?

The code is all there. If it's not in the repo mentioned above, you can pull it out with ILSpy. That includes DataViews and database specific adapters like SqlDataAdapter.

It's not a solution in and of itself. Someone will need to go through the process of getting the code to compile for .NET Core, and then test it. But, it is possible.

There's no reason legal or otherwise, as far as I can tell why someone can't just start a repo and go for it. Of course, I'm not a lawyer and am not offering legal advice.

MelbourneDeveloper commented Aug 31, 2016

?

The code is all there. If it's not in the repo mentioned above, you can pull it out with ILSpy. That includes DataViews and database specific adapters like SqlDataAdapter.

It's not a solution in and of itself. Someone will need to go through the process of getting the code to compile for .NET Core, and then test it. But, it is possible.

There's no reason legal or otherwise, as far as I can tell why someone can't just start a repo and go for it. Of course, I'm not a lawyer and am not offering legal advice.

@chadbr

This comment has been minimized.

Show comment
Hide comment
@chadbr

chadbr Aug 31, 2016

Mono? Or does it have the binding bits also?

chadbr commented Aug 31, 2016

Mono? Or does it have the binding bits also?

@MelbourneDeveloper

This comment has been minimized.

Show comment
Hide comment
@MelbourneDeveloper

MelbourneDeveloper commented Aug 31, 2016

Mono? Binding Bits?

Look! Here is the code for DataView.
https://github.com/Microsoft/referencesource/blob/master/System.Data/System/Data/DataView.cs

It's all there.

@FransBouma

This comment has been minimized.

Show comment
Hide comment
@FransBouma

FransBouma Aug 31, 2016

Collaborator

The code is all there. If it's not in the repo mentioned above, you can pull it out with ILSpy. That
includes DataViews and database specific adapters like SqlDataAdapter.

Yeah, but that's against the EULA.

It's not a solution in and of itself. Someone will need to go through the process of getting the code
to compile for .NET Core, and then test it. But, it is possible.

The question isn't whether it's theoretically possible, but whether it's doable. 'Someone will need' yeah, that's always the case, isn't it? 

There's no reason legal or otherwise, as far as I can tell why someone can't just start a repo and go
for it. Of course, I'm not a lawyer and am not offering legal advice.

Heh, yeah that's clear ;)

Mono? Binding Bits?

Look! Here is the code for DataView.
https://github.com/Microsoft/referencesource/blob/master/System.Data/System/Data/DataView.cs

It's all there.

yeah, but it's not compilable on .net core because it depends on apis which aren't in .NET core at the moment. 

And even if you spent all that effort to port these too, you'll run into the issue where DbDataReader doesn't expose a datatable with GetSchemaTable(). 

If you want to port all that, fine by me, but it will take more than just a couple of days. And I doubt MS will merge the PR. 

    FB
Collaborator

FransBouma commented Aug 31, 2016

The code is all there. If it's not in the repo mentioned above, you can pull it out with ILSpy. That
includes DataViews and database specific adapters like SqlDataAdapter.

Yeah, but that's against the EULA.

It's not a solution in and of itself. Someone will need to go through the process of getting the code
to compile for .NET Core, and then test it. But, it is possible.

The question isn't whether it's theoretically possible, but whether it's doable. 'Someone will need' yeah, that's always the case, isn't it? 

There's no reason legal or otherwise, as far as I can tell why someone can't just start a repo and go
for it. Of course, I'm not a lawyer and am not offering legal advice.

Heh, yeah that's clear ;)

Mono? Binding Bits?

Look! Here is the code for DataView.
https://github.com/Microsoft/referencesource/blob/master/System.Data/System/Data/DataView.cs

It's all there.

yeah, but it's not compilable on .net core because it depends on apis which aren't in .NET core at the moment. 

And even if you spent all that effort to port these too, you'll run into the issue where DbDataReader doesn't expose a datatable with GetSchemaTable(). 

If you want to port all that, fine by me, but it will take more than just a couple of days. And I doubt MS will merge the PR. 

    FB
@redowl3

This comment has been minimized.

Show comment
Hide comment
@redowl3

redowl3 Aug 31, 2016

Which (I think?) brings us back to my question of how can MSFT only support Core RTM apps on Azure when not everything has been ported?

redowl3 commented Aug 31, 2016

Which (I think?) brings us back to my question of how can MSFT only support Core RTM apps on Azure when not everything has been ported?

@MelbourneDeveloper

This comment has been minimized.

Show comment
Hide comment
@MelbourneDeveloper

MelbourneDeveloper Sep 1, 2016

"The question isn't whether it's theoretically possible, but whether it's doable. 'Someone will need' yeah, that's always the case, isn't it?"

Of course it's doable. Most people here only need certain bits and pieces to get their code compiling. Why doesn't someone who's got some time get their bit compiling and then stick it up in a repo somewhere? Then, others can contribute as they get their bits compiling. This is better than sitting around an complaining.

yeah, but it's not compilable on .net core because it depends on apis which aren't in .NET core at the moment.

Some will compile, some won't. In the majority of cases it will be a matter of finding the equivalent API call in .NET Core.

And even if you spent all that effort to port these too, you'll run into the issue where DbDataReader doesn't expose a datatable with GetSchemaTable().

It's exposed. The method is there in the public repo:

virtual public DataTable GetSchemaTable()
{
throw new NotSupportedException();
}

That method needs to be overridden by the inheriting class.

MelbourneDeveloper commented Sep 1, 2016

"The question isn't whether it's theoretically possible, but whether it's doable. 'Someone will need' yeah, that's always the case, isn't it?"

Of course it's doable. Most people here only need certain bits and pieces to get their code compiling. Why doesn't someone who's got some time get their bit compiling and then stick it up in a repo somewhere? Then, others can contribute as they get their bits compiling. This is better than sitting around an complaining.

yeah, but it's not compilable on .net core because it depends on apis which aren't in .NET core at the moment.

Some will compile, some won't. In the majority of cases it will be a matter of finding the equivalent API call in .NET Core.

And even if you spent all that effort to port these too, you'll run into the issue where DbDataReader doesn't expose a datatable with GetSchemaTable().

It's exposed. The method is there in the public repo:

virtual public DataTable GetSchemaTable()
{
throw new NotSupportedException();
}

That method needs to be overridden by the inheriting class.

@FransBouma

This comment has been minimized.

Show comment
Hide comment
@FransBouma

FransBouma Sep 1, 2016

Collaborator

@MelbourneDeveloper I'd suggest you read this whole thread, you're trying to rehash things which have already been discussed at length here and in other threads. The TL;DR: it's not as simple as you seem to think, not by far. Don't you think we would otherwise already have implemented a workaround?

Collaborator

FransBouma commented Sep 1, 2016

@MelbourneDeveloper I'd suggest you read this whole thread, you're trying to rehash things which have already been discussed at length here and in other threads. The TL;DR: it's not as simple as you seem to think, not by far. Don't you think we would otherwise already have implemented a workaround?

@VitaliyMF

This comment has been minimized.

Show comment
Hide comment
@VitaliyMF

VitaliyMF Sep 1, 2016

@MelbourneDeveloper , @redowl3 DataSet/DataTable/DataRow implementations are over-complicated and it is good that they are not ported to .NET Core "as is". The structure is too heavy; it implements too many functions and in overall this "monolith" component is not the best choice in every concrete case.

Instead of just asking for old DataTable in .NET Core, could anyone list the functionality/usage scenarios that still needed and not covered by existing .NET Core-compatible libraries?

VitaliyMF commented Sep 1, 2016

@MelbourneDeveloper , @redowl3 DataSet/DataTable/DataRow implementations are over-complicated and it is good that they are not ported to .NET Core "as is". The structure is too heavy; it implements too many functions and in overall this "monolith" component is not the best choice in every concrete case.

Instead of just asking for old DataTable in .NET Core, could anyone list the functionality/usage scenarios that still needed and not covered by existing .NET Core-compatible libraries?

@MihaMarkic

This comment has been minimized.

Show comment
Hide comment
@MihaMarkic

MihaMarkic Sep 1, 2016

Instead of just asking for old DataTable in .NET Core, could anyone list the functionality/usage scenarios that still needed and not covered by existing .NET Core-compatible libraries?

Oh, no, please, not again.

MihaMarkic commented Sep 1, 2016

Instead of just asking for old DataTable in .NET Core, could anyone list the functionality/usage scenarios that still needed and not covered by existing .NET Core-compatible libraries?

Oh, no, please, not again.

@mahara

This comment has been minimized.

Show comment
Hide comment
@mahara

mahara Sep 1, 2016

Of course it's doable. Most people here only need certain bits and pieces to get their code compiling. Why doesn't someone who's got some time get their bit compiling and then stick it up in a repo somewhere? Then, others can contribute as they get their bits compiling. This is better than sitting around an complaining.

@MelbourneDeveloper
You talk like it is something easy to be done. Why don't you start doing what you said, and let others benefit from the work you'd done? Piece of cake, isn't it?

mahara commented Sep 1, 2016

Of course it's doable. Most people here only need certain bits and pieces to get their code compiling. Why doesn't someone who's got some time get their bit compiling and then stick it up in a repo somewhere? Then, others can contribute as they get their bits compiling. This is better than sitting around an complaining.

@MelbourneDeveloper
You talk like it is something easy to be done. Why don't you start doing what you said, and let others benefit from the work you'd done? Piece of cake, isn't it?

@roji

This comment has been minimized.

Show comment
Hide comment
@roji

roji Sep 1, 2016

Contributor

@VitaliyMF please read this entire thread (as painful as that might be).

Note that since this was debated quite a bit of time has gone by, so here's a quick summary of the current state of affairs as I see it. .NET Core RTM has been released and it does include an alternate resultset metadata API which does not rely on DataTable/DataSet (#5915), addressing one of the main concerns originally raised. A database/table metadata API (as opposed to a resultset metadata API) is still missing (#5024), but it could be argued that users that really need this can query schema information manually (e.g. INFORMATION_SCHEMA or some database-specific structures), even if that's not database-independent and generally sucks.

IMHO this means that at this point there doesn't seem to be any missing functionality depending on DataTable/DataSet - you can do (almost?) anything without DataTable/DataSet (e.g. access dynamic results via DbDataReader APIs). So the remaining reason for DataTable/DataSet is for porting across code that already uses them, or a need for an untyped data access API that also includes the DataTable/DataSet extras (e.g. optimistic concurrency). These are of course valid requests/requirements.

Note that Microsoft seems to have also shifted their general strategy, and intend to make .NET Core more backwards-compatible with .NET Framework, to help porting (see this article). This means that DataTable/DataSet may come back, or they may not. Either way, IMHO it's great to have them as a totally separate package without any core aspects of ADO.NET depending on them (i.e. metadata).

Contributor

roji commented Sep 1, 2016

@VitaliyMF please read this entire thread (as painful as that might be).

Note that since this was debated quite a bit of time has gone by, so here's a quick summary of the current state of affairs as I see it. .NET Core RTM has been released and it does include an alternate resultset metadata API which does not rely on DataTable/DataSet (#5915), addressing one of the main concerns originally raised. A database/table metadata API (as opposed to a resultset metadata API) is still missing (#5024), but it could be argued that users that really need this can query schema information manually (e.g. INFORMATION_SCHEMA or some database-specific structures), even if that's not database-independent and generally sucks.

IMHO this means that at this point there doesn't seem to be any missing functionality depending on DataTable/DataSet - you can do (almost?) anything without DataTable/DataSet (e.g. access dynamic results via DbDataReader APIs). So the remaining reason for DataTable/DataSet is for porting across code that already uses them, or a need for an untyped data access API that also includes the DataTable/DataSet extras (e.g. optimistic concurrency). These are of course valid requests/requirements.

Note that Microsoft seems to have also shifted their general strategy, and intend to make .NET Core more backwards-compatible with .NET Framework, to help porting (see this article). This means that DataTable/DataSet may come back, or they may not. Either way, IMHO it's great to have them as a totally separate package without any core aspects of ADO.NET depending on them (i.e. metadata).

@VitaliyMF

This comment has been minimized.

Show comment
Hide comment
@VitaliyMF

VitaliyMF Sep 1, 2016

@roji Thank you for the detailed explanation. As I know that IDbColumnSchemaGenerator was specially introduced to cover schema aspect (not very helpful in practice - for example, Microsoft.Data.Sqlite.SqliteDataReader doesn't implement this interface).

Am I understand correctly, that only reason to have DataTable in .NET Core is backward compatibility?.. Well, not everything can be easily ported to .NET Core 1.0 and a lot of other incompatibilities will prevent old .NET projects from migration anyway (you don't say "System.Web" and WebForms should come back, right?). But new .NET Core projects will not use/depend on old DataTable!

Regarding untyped data access API: when needed, it may be provided by 3rd party libraries (I already mentioned my NReco.Data lib that implements command builder/data adapter and RecordSet with API similar to DataTable/DataRow). For me, it is still unclear, why old DataTable should come back into NETStandard ADO.NET API.

VitaliyMF commented Sep 1, 2016

@roji Thank you for the detailed explanation. As I know that IDbColumnSchemaGenerator was specially introduced to cover schema aspect (not very helpful in practice - for example, Microsoft.Data.Sqlite.SqliteDataReader doesn't implement this interface).

Am I understand correctly, that only reason to have DataTable in .NET Core is backward compatibility?.. Well, not everything can be easily ported to .NET Core 1.0 and a lot of other incompatibilities will prevent old .NET projects from migration anyway (you don't say "System.Web" and WebForms should come back, right?). But new .NET Core projects will not use/depend on old DataTable!

Regarding untyped data access API: when needed, it may be provided by 3rd party libraries (I already mentioned my NReco.Data lib that implements command builder/data adapter and RecordSet with API similar to DataTable/DataRow). For me, it is still unclear, why old DataTable should come back into NETStandard ADO.NET API.

@chadbr

This comment has been minimized.

Show comment
Hide comment
@chadbr

chadbr Sep 1, 2016

I thought this thread was closed :)

Could someone from Microsoft simply clarify if it's on the roadmap or not?

chadbr commented Sep 1, 2016

I thought this thread was closed :)

Could someone from Microsoft simply clarify if it's on the roadmap or not?

@roji

This comment has been minimized.

Show comment
Hide comment
@roji

roji Sep 1, 2016

Contributor

As I know that IDbColumnSchemaGenerator was specially introduced to cover schema aspect (not very helpful in practice - for example, Microsoft.Data.Sqlite.SqliteDataReader doesn't implement this interface).

Are you sure about that? It's documented on SqliteDataReader. The Npgsql provider also implements it.

Am I understand correctly, that only reason to have DataTable in .NET Core is backward compatibility?.. Well, not everything can be easily ported to .NET Core 1.0 and a lot of other incompatibilities will prevent old .NET projects from migration anyway (you don't say "System.Web" and WebForms should come back, right?). But new .NET Core projects will not use/depend on old DataTable!

Well, it's true that originally Microsoft seemed to want to remove legacy/unwanted APIs in .NET Core. But at some point (again, see this blog) the decision seems to have made to really prioritize making .NET Core as compatible as possible with .NET Framework, to help people port. Whatever we think of DataTable/DataSet, there are definitely tons of programs out there relying on it.

Especially with .NET Core being broken down into nugets, there doesn't seem to be any harm in making DataTable/DataSet available in .NET Core, as a separate package (as long as it isn't needed for any core operation, e.g. metadata). I'm guessing it might be somewhat low on the internal priority list though, with everything else going on.

Contributor

roji commented Sep 1, 2016

As I know that IDbColumnSchemaGenerator was specially introduced to cover schema aspect (not very helpful in practice - for example, Microsoft.Data.Sqlite.SqliteDataReader doesn't implement this interface).

Are you sure about that? It's documented on SqliteDataReader. The Npgsql provider also implements it.

Am I understand correctly, that only reason to have DataTable in .NET Core is backward compatibility?.. Well, not everything can be easily ported to .NET Core 1.0 and a lot of other incompatibilities will prevent old .NET projects from migration anyway (you don't say "System.Web" and WebForms should come back, right?). But new .NET Core projects will not use/depend on old DataTable!

Well, it's true that originally Microsoft seemed to want to remove legacy/unwanted APIs in .NET Core. But at some point (again, see this blog) the decision seems to have made to really prioritize making .NET Core as compatible as possible with .NET Framework, to help people port. Whatever we think of DataTable/DataSet, there are definitely tons of programs out there relying on it.

Especially with .NET Core being broken down into nugets, there doesn't seem to be any harm in making DataTable/DataSet available in .NET Core, as a separate package (as long as it isn't needed for any core operation, e.g. metadata). I'm guessing it might be somewhat low on the internal priority list though, with everything else going on.

@MelbourneDeveloper

This comment has been minimized.

Show comment
Hide comment
@MelbourneDeveloper

MelbourneDeveloper Sep 2, 2016

@MelbourneDeveloper
You talk like it is something easy to be done. Why don't you start doing what you said, and let others benefit from the work you'd done? Piece of cake, isn't it?

I already did some work around this in Silverlight by simply looking at the metadata of the System.Data namespace in Visual Studio (ILDasm like functionality) - not with any use of ILSpy, or with any code from the reference source. It's not as hard as people make out.

Here's an example of production code Fill method for a DataAdapter:

            for (int i = 0; i < reader.FieldCount; i++)
            {
                var columnName = reader.GetName(i);
                if (!dataTable.Columns.Contains(columnName))
                {
                    dataTable.Columns.Add(new DataColumn(columnName));
                }
            }

            while (reader.Read())
            {
                var dataRow = new DataRow(dataTable);

                for (int i = 0; i < reader.FieldCount; i++)
                {
                    var columnName = reader.GetName(i);
                    dataRow[columnName] = reader.GetValue(i);
                }

                dataTable.Rows.Add(dataRow);
            }

Originally, @benaadams mentioned that this code was using the MIT license

Well you might want to use the MIT licenced reference source instead; just saying...
Also its already on github https://github.com/Microsoft/referencesource/blob/master/System.Data/System/Data/DataTable.cs

But, after looking at the license agreement, it only allows us to use the code as a "reference":
http://referencesource.microsoft.com/license.html
http://referencesource.microsoft.com

The source is actually released under the MS-RSL license. Which according to Wikipedia "is the most restrictive of the Microsoft Shared Source licenses" (https://en.wikipedia.org/wiki/Shared_source#Microsoft_Reference_Source_License_.28Ms-RSL.29).

Are we able to get clarification on this? Are we allowed to take the System.Data code, slightly modify it for the purpose of compilation and use on .NET Core, and then stick that code up in a GitHub repo? The code may, or may not be used by Microsoft at a later point as a basis for building more functionality in to the System.Data namespace in .NET Core.

Just to clarify, I have no intention of breaking the license agreements put forward by Microsoft, or breaking the spirit of the agreements. I am merely raising this as a point that if Microsoft were to allow us to do it, we could recreate this library ourselves. If 10 programmers spent a small amount of time working on just the area of the functionality they need, and those areas were different, we'd be able to knock this over in a very short time span. But, if Microsoft doesn't want us to do this, then I withdraw any comments about attempting this, and will be happy to wait for the team to go ahead with this.

MelbourneDeveloper commented Sep 2, 2016

@MelbourneDeveloper
You talk like it is something easy to be done. Why don't you start doing what you said, and let others benefit from the work you'd done? Piece of cake, isn't it?

I already did some work around this in Silverlight by simply looking at the metadata of the System.Data namespace in Visual Studio (ILDasm like functionality) - not with any use of ILSpy, or with any code from the reference source. It's not as hard as people make out.

Here's an example of production code Fill method for a DataAdapter:

            for (int i = 0; i < reader.FieldCount; i++)
            {
                var columnName = reader.GetName(i);
                if (!dataTable.Columns.Contains(columnName))
                {
                    dataTable.Columns.Add(new DataColumn(columnName));
                }
            }

            while (reader.Read())
            {
                var dataRow = new DataRow(dataTable);

                for (int i = 0; i < reader.FieldCount; i++)
                {
                    var columnName = reader.GetName(i);
                    dataRow[columnName] = reader.GetValue(i);
                }

                dataTable.Rows.Add(dataRow);
            }

Originally, @benaadams mentioned that this code was using the MIT license

Well you might want to use the MIT licenced reference source instead; just saying...
Also its already on github https://github.com/Microsoft/referencesource/blob/master/System.Data/System/Data/DataTable.cs

But, after looking at the license agreement, it only allows us to use the code as a "reference":
http://referencesource.microsoft.com/license.html
http://referencesource.microsoft.com

The source is actually released under the MS-RSL license. Which according to Wikipedia "is the most restrictive of the Microsoft Shared Source licenses" (https://en.wikipedia.org/wiki/Shared_source#Microsoft_Reference_Source_License_.28Ms-RSL.29).

Are we able to get clarification on this? Are we allowed to take the System.Data code, slightly modify it for the purpose of compilation and use on .NET Core, and then stick that code up in a GitHub repo? The code may, or may not be used by Microsoft at a later point as a basis for building more functionality in to the System.Data namespace in .NET Core.

Just to clarify, I have no intention of breaking the license agreements put forward by Microsoft, or breaking the spirit of the agreements. I am merely raising this as a point that if Microsoft were to allow us to do it, we could recreate this library ourselves. If 10 programmers spent a small amount of time working on just the area of the functionality they need, and those areas were different, we'd be able to knock this over in a very short time span. But, if Microsoft doesn't want us to do this, then I withdraw any comments about attempting this, and will be happy to wait for the team to go ahead with this.

@MelbourneDeveloper

This comment has been minimized.

Show comment
Hide comment
@MelbourneDeveloper

MelbourneDeveloper Sep 2, 2016

it could be argued that users that really need this can query schema information manually (e.g. INFORMATION_SCHEMA or some database-specific structures)

This is what I have done in the past, but it's not ideal. You will need to write a layer over the top if you are implementing for multiple database platforms.

with .NET Core being broken down into nugets, there doesn't seem to be any harm in making DataTable/DataSet available in .NET Core, as a separate package (as long as it isn't needed for any core operation, e.g. metadata).

Exactly. And, why not let the community build it?

MelbourneDeveloper commented Sep 2, 2016

it could be argued that users that really need this can query schema information manually (e.g. INFORMATION_SCHEMA or some database-specific structures)

This is what I have done in the past, but it's not ideal. You will need to write a layer over the top if you are implementing for multiple database platforms.

with .NET Core being broken down into nugets, there doesn't seem to be any harm in making DataTable/DataSet available in .NET Core, as a separate package (as long as it isn't needed for any core operation, e.g. metadata).

Exactly. And, why not let the community build it?

@MelbourneDeveloper

This comment has been minimized.

Show comment
Hide comment
@MelbourneDeveloper

MelbourneDeveloper Sep 2, 2016

@VitaliyMF

@MelbourneDeveloper , @redowl3 DataSet/DataTable/DataRow implementations are over-complicated and it is good that they are not ported to .NET Core "as is". The structure is too heavy; it implements too many functions and in overall this "monolith" component is not the best choice in every concrete case.

Instead of just asking for old DataTable in .NET Core, could anyone list the functionality/usage scenarios that still needed and not covered by existing .NET Core-compatible libraries?

On a basic level you are right. These classes are bloated and unnecessary. People should rewrite their code so as to avoid them.

That doesn't change the fact that legacy code does exist, and in order to embrace the new .NET Core technology, legacy code must be made to compile and work with the new technology. With the flick of a switch, large swathes of legacy code could be made to compile, and since this library does not need to be deployed with every instance of the .NET Core architecture, it's a mute point to say that it's too heavy. Any library could be too heavy. But, just taking some existing code and compiling it for .NET Core won't slow .NET Core down. Every developer still needs to make conscious decisions about performance no matter what.

MelbourneDeveloper commented Sep 2, 2016

@VitaliyMF

@MelbourneDeveloper , @redowl3 DataSet/DataTable/DataRow implementations are over-complicated and it is good that they are not ported to .NET Core "as is". The structure is too heavy; it implements too many functions and in overall this "monolith" component is not the best choice in every concrete case.

Instead of just asking for old DataTable in .NET Core, could anyone list the functionality/usage scenarios that still needed and not covered by existing .NET Core-compatible libraries?

On a basic level you are right. These classes are bloated and unnecessary. People should rewrite their code so as to avoid them.

That doesn't change the fact that legacy code does exist, and in order to embrace the new .NET Core technology, legacy code must be made to compile and work with the new technology. With the flick of a switch, large swathes of legacy code could be made to compile, and since this library does not need to be deployed with every instance of the .NET Core architecture, it's a mute point to say that it's too heavy. Any library could be too heavy. But, just taking some existing code and compiling it for .NET Core won't slow .NET Core down. Every developer still needs to make conscious decisions about performance no matter what.

@svick

This comment has been minimized.

Show comment
Hide comment
@svick
Contributor

svick commented Sep 2, 2016

@MelbourneDeveloper

This comment has been minimized.

Show comment
Hide comment
@MelbourneDeveloper

MelbourneDeveloper Sep 3, 2016

Boom!

Off we go!

MelbourneDeveloper commented Sep 3, 2016

Boom!

Off we go!

@bharatmangal04

This comment has been minimized.

Show comment
Hide comment
@bharatmangal04

bharatmangal04 Nov 14, 2016

Can anyone suggest what would be the recommended approach to pass data ( User Defined Table type or paramenters) to stored procedure in ASp.NET Core if using ADO.NET

bharatmangal04 commented Nov 14, 2016

Can anyone suggest what would be the recommended approach to pass data ( User Defined Table type or paramenters) to stored procedure in ASp.NET Core if using ADO.NET

@saurabh500

This comment has been minimized.

Show comment
Hide comment
@saurabh500

saurabh500 Nov 14, 2016

Member

@bharatmangal04 Are you referring to SQL CLR UDTs on Sql Server?
Can you open a new issue for this question?

Member

saurabh500 commented Nov 14, 2016

@bharatmangal04 Are you referring to SQL CLR UDTs on Sql Server?
Can you open a new issue for this question?

@sureba73

This comment has been minimized.

Show comment
Hide comment
@sureba73

sureba73 Nov 23, 2016

Tracking.... need DataTable/DataSet badly :(

sureba73 commented Nov 23, 2016

Tracking.... need DataTable/DataSet badly :(

@chadbr

This comment has been minimized.

Show comment
Hide comment
@chadbr

chadbr commented Nov 23, 2016

@sureba73 it's in... #8622

@roozbehid

This comment has been minimized.

Show comment
Hide comment
@roozbehid

roozbehid Nov 30, 2016

So it is in daily build. How can I use it?
I am so confused on how these work
I've installed the newset sdk and so in my global.json I have

"sdk": { "version": "1.0.0-preview4-004130" }

So does it mean I can use DataTable/DataSet?
on my csproj I have

`

  <Version>1.0.0-alpha-20161104-2</Version>
  <PrivateAssets>All</PrivateAssets>
</PackageReference>
<PackageReference Include="NETStandard.Library">
  <Version>1.6.1-preview1-24530-04</Version>
</PackageReference>

`

<TargetFramework>netcoreapp1.0</TargetFramework>

So should I be able to just use them?!

roozbehid commented Nov 30, 2016

So it is in daily build. How can I use it?
I am so confused on how these work
I've installed the newset sdk and so in my global.json I have

"sdk": { "version": "1.0.0-preview4-004130" }

So does it mean I can use DataTable/DataSet?
on my csproj I have

`

  <Version>1.0.0-alpha-20161104-2</Version>
  <PrivateAssets>All</PrivateAssets>
</PackageReference>
<PackageReference Include="NETStandard.Library">
  <Version>1.6.1-preview1-24530-04</Version>
</PackageReference>

`

<TargetFramework>netcoreapp1.0</TargetFramework>

So should I be able to just use them?!

@karelz karelz modified the milestones: Future, 1.0.0-rtm Dec 3, 2016

@StingyJack

This comment has been minimized.

Show comment
Hide comment
@StingyJack

StingyJack Mar 8, 2017

Lack of DataTable blocks pretty much anyone using EnterpriseLibrary.Data UpdateDataSet() for higher performance batch insertions, unless the requirement is to rewrite entire DAL to build the SQL statements manually. We dont want to be putting DataReader dependencies in services, either.

This also prevents using TableValuedParameters,

EF has never been performant enough, and is too easy to misuse.

StingyJack commented Mar 8, 2017

Lack of DataTable blocks pretty much anyone using EnterpriseLibrary.Data UpdateDataSet() for higher performance batch insertions, unless the requirement is to rewrite entire DAL to build the SQL statements manually. We dont want to be putting DataReader dependencies in services, either.

This also prevents using TableValuedParameters,

EF has never been performant enough, and is too easy to misuse.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment