Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement MySqlBulkLoader #15

Closed
bgrainger opened this issue Jul 1, 2016 · 6 comments · Fixed by #180
Closed

Implement MySqlBulkLoader #15

bgrainger opened this issue Jul 1, 2016 · 6 comments · Fixed by #180

Comments

@bgrainger
Copy link
Member

@bgrainger bgrainger commented Jul 1, 2016

Clone the MySqlBulkLoader API (specified here: https://dev.mysql.com/doc/connector-net/en/connector-net-programming-bulk-loader.html).

As an extension, support a Stream property instead of a FileName. Behind the scenes, create a GUID and pass that to MySQL as the LOAD DATA LOCAL INFILE file name (https://dev.mysql.com/doc/refman/5.7/en/load-data.html).

When the LOCAL_INFILE_Request is received with that GUID as the name, stream the in-memory data to the server.

@gitsno
Copy link

@gitsno gitsno commented Oct 3, 2016

I'm working on this - I'm also intending to create a MySqlBulkCopy class that works like the SqlBulkCopy class does with MS SQL Server - so it will create an INSERT ... VALUES statement with multiple rows for a given enumerable.

@VitaliyMF
Copy link

@VitaliyMF VitaliyMF commented Oct 3, 2016

@gitsno just a notice, original MySqlBulkCopy uses LOAD DATA INFILE syntax and I think it offers much better performance for large datasets;
I assume that efficiency of INSERT .. VALUES (row1), (row2) ... (rowN) syntax is very close to several insert statements executed with one MySqlCommand. This approach works for any database (not only MySql/MsSql), and this use-case is already covered by many data access libs. What do you think?

@gitsno
Copy link

@gitsno gitsno commented Oct 3, 2016

Agreed - I'm not suggesting that it's preferred over LOAD DATA INFILE, but given that developers are used to working with SqlBulkCopy and it does work well, I figure it would be nice to have it as an alternative for the MySql client. If you already have the records in a .NET DataTable or IEnumerable you have to write extra code to get them into the file format that LOAD DATA INFILE expects, whereas if the SqlBulkCopy is implemented you can insert them easily.

@bgrainger
Copy link
Member Author

@bgrainger bgrainger commented Oct 3, 2016

No objection to adding a MySqlBulkCopy class that exposes the same/similar API to SqlBulkCopy.

For this issue specifically, a MySqlBulkLoader class should implement the existing MySQL API and use LOAD DATA INFILE.

@gitsno
Copy link

@gitsno gitsno commented Dec 10, 2016

Just added, see PR #148

gitsno pushed a commit to gitsno/MySqlConnector that referenced this issue Jan 24, 2017
bgrainger pushed a commit that referenced this issue Feb 17, 2017
bgrainger added a commit that referenced this issue Feb 20, 2017
@bgrainger
Copy link
Member Author

@bgrainger bgrainger commented Feb 20, 2017

Implemented in 0.13.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging a pull request may close this issue.

3 participants