Enhancement: DataTable or IDataReader support for Bulk Insert

[Grauenwolf]

It would be really nice if we could just pass in a DataTable or IDataReader like we do with SQL Server and let the driver handle the CSV conversion and steam management.

This way we don't have people guessing about things like how to encode strings, dates, and nulls.

[Grauenwolf]

To do it 'right' I'm thinking a subclass of IO.Stream is needed so that it can consume the IDataReader at the same time MySqlBulkLoader consumes the stream. But I don't know the implementation details well enough to say that with certainty.

In my own code I just dump everything into a massive MemoryStream, which isn't great for performance.

[bgrainger]

Are you talking about the SqlBulkCopy.WriteToServer method?

It seems like a helper class could be written that sets up a MySqlBulkLoader and sets the SourceStream property to a custom Stream implementation that reads from a DataTable (or IDataReader, but there are better options for inserting directly from one table to another if that IDataReader is from the same database).

[Grauenwolf]

Essentially yes.

As for IDataReader, it makes for a useful abstraction. For SQL Server I'll wrap a collection of objects, a enumeration of objects, or a file parser in an IDataReader and stream it in instead of building a fat DataTable.

In fact, DataTable exposes an IDataReader so we'd only need one implementation.

[Grauenwolf]

Whether it is part of MySqlBulkLoader or a separate extension method doesn't matter to me. Though it should be easily discoverable.

[bgrainger]

I've developed a prototype version of this at e0a2de5. Current API (subject to change):

using var connection = new MySqlConnection("...;AllowLoadLocalInfile=true");
var bulkLoader = new MySqlBulkLoader(connection, dataTableOrDataReader)
{
    TableName = "destination_table",
};
bulkLoader.Load();

[Grauenwolf]

That's not at all like I expected it to look. It's certainly a lot cleaner than what I could have done from outside the library.

[bgrainger]

Updated API proposal: 98db02d#diff-e5e5c76f457fba4a79beedb1995a9f5f

using (var connection = new MySqlConnection("...;AllowLoadLocalInfile=True"))
{
	await connection.OpenAsync();
	var bulkCopy = new MySqlBulkCopy(connection);
	bulkCopy.DestinationTableName = "some_table_name";
	await bulkCopy.WriteToServerAsync(dataTable);
}

@Grauenwolf how important/useful to you would it be to have the capability to load from IEnumerable<object[]> (or IAsyncEnumerable<object[]>)?

[Grauenwolf]

Do you mean just passing in stuff like new object[] ("Tom", "Jones", 15, true)?

If so, I don't need it personally, but I think it would be useful to others.

[bgrainger]

Yes, I was just referring to the latter (which would be a simple extension to the current code); I didn't have any plans to add reflection, matching property names with field names, etc.

[dennis-gr]

I've been testing this and it's working nicely except for text columns that contain newlines:

Testcase:

CREATE TABLE `test` (
  `ID` INT NOT NULL,
  `Content` TEXT NOT NULL,
  `AnotherID` INT NOT NULL,
  PRIMARY KEY (`ID`));

  private const string ContentWithNewLine = @"foo
                                              bar";
  [Fact]
  public void MySqlBulkCopy_With_NewLines()
  {
      var table = new DataTable();
      table.Columns.AddRange(new[] { new DataColumn("ID", typeof(int)), new DataColumn("Content", typeof(string)), new DataColumn("AnotherID", typeof(int)) });
      
      table.Rows.Add(1, "text", 2);
      table.Rows.Add(2, ContentWithNewLine, 3);
      table.Rows.Add(3, "just some text", 4);

      using (var connection = new MySqlConnection())
      {
          connection.Open();

          var bulkCopy = new MySqlBulkCopy(connection) { DestinationTableName = "test" };
          bulkCopy.WriteToServer(table);
      }
  }   

Produces this:

mysql> select * from test;
+----+----------------+-----------+
| ID | Content        | AnotherID |
+----+----------------+-----------+
|  0 | 3              |         0 |
|  1 | text           |         2 |
           |         0 |
|  3 | just some text |         4 |
+----+----------------+-----------+
4 rows in set (0.00 sec)

[bgrainger]

it's working nicely except for text columns that contain newlines

The string-escaping routine is missing a check for \n:

MySqlConnector/src/MySqlConnector/MySql.Data.MySqlClient/MySqlBulkCopy.cs

Line 359 in 8762229

if (value[index] == '\t' || value[index] == '\\')

Thanks for pointing this out; a fix will be in the next release.

[bgrainger]

@dennis-gr 0.62.0-beta3 is available and should fix this problem; thanks for reporting it!

[Grauenwolf]

Thank you both for this!