Repository navigation
MySqlBulkCopy: MySqlConnector.MySqlException: 'Error during LOAD DATA LOCAL INFILE' #974
Description
Hi,
Thank you for this great library! ;)
I periodically get exceptions from the MySqlBulkCopy class.
This happens when the utf8 encoder tries to encode a unicode character into a buffer with a length of 1.
Exception:
The output byte buffer is too small to contain the encoded data, encoding 'Unicode (UTF-8)' fallback 'System.Text.EncoderReplacementFallback'. (Parameter 'bytes')
Example code to reproduce:
static async Task Main(string[] args)
{
/*
CREATE TABLE `testtbl` (
`a` MEDIUMTEXT NOT NULL,
`b` VARCHAR(255) NOT NULL
)
*/
var a = new StringBuilder();
for (int i = 0; i < 1048573; i++)
{
a.Append("a");
}
var b = "Å string that crashes application";
var dataTable = new DataTable();
dataTable.Columns.Add("a");
dataTable.Columns.Add("b");
dataTable.Rows.Add(new object[] { a, b });
using var connection = new MySqlConnection("...;AllowLoadLocalInfile=true");
await connection.OpenAsync();
var bulkCopy = new MySqlBulkCopy(connection);
bulkCopy.DestinationTableName = "testtbl";
await bulkCopy.WriteToServerAsync(dataTable);
}
Activity
bgrainger commentedon Apr 23, 2021
Thanks for the great repro!
bgrainger commentedon Apr 23, 2021
I naively assumed that
Convertwould output the leading bytes and maintain internal state to output the rest of the UTF-8 bytes in the next block. (And apparently I wasn't the only one: npgsql/npgsql#2849.)The fix will be to call
GetByteCountto ensure that at least onecharcan be converted.bgrainger commentedon Apr 23, 2021
Even worse, I already made this mistake but didn't learn from it ☹ #520.
Ensure that at least one character can be converted. Fixes #974
RobertGlynWilliams commentedon Apr 23, 2021
Thank you for the bugfix 💯 .
bgrainger commentedon Apr 23, 2021
Fixed in 1.3.5.
chrisdcmoore commentedon May 14, 2021
Hey @bgrainger - I noticed this issue referenced an issue I opened and fixed in Npgsql a while back, and I just wanted to give you a heads up that I've discovered the fix I wrote doesn't appear to be entirely correct - I've opened another issue against Npgsql describing what I think is going on over there, so you may wish to check whether or not it also affects your fix to this issue here!
npgsql/npgsql#3733
Cheers
bgrainger commentedon May 14, 2021
Thanks @chrisdcmoore; it looks like my fix is subject to this same problem. I appreciate your bringing it to my attention.
Ensure at least one complete code point is converted. Fixes #974
bgrainger commentedon Jun 16, 2021
Improved fix in 1.3.9.