Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LOAD DATA LOCAL INFILE implemetation #158

Merged
merged 7 commits into from Sep 10, 2012
Merged

Conversation

anton-kotenko
Copy link
Contributor

  1. Make queries like "load data local infile" to work in their normal way (data is taken from file on client's machine and sent to mysql server)
  2. Make possible to use instance of nodejs's Buffer class instead of file, as source of rows.
    (feature similar to postgresql's COPY FROM STDIN). Sometimes it's more convenient that using temporary file or pipe.

This feature requires update in query and querySync interface (without breaking compatibility)
Examples:

var buf = new Buffer('1,2,3\n4,5,6');
conn.querySync('LOAD DATA LOCAL INFILE \'some_meaningless_name\' INTO TABLE test_table FIELDS TERMINATED BY \',\'  LINES TERMINATED BY \'\n''', buf);

@Sannis
Copy link
Owner

Sannis commented Sep 6, 2012

Thanks! I will merge this after fix failed tests in master.

And I prefer to cherry-pick commits manually to avoid duplications from your fork. You can help me with this if you have some time.

void MysqlConnection::RestoreLocalInfileHandlers(local_infile_data * infile_data,
MYSQL * conn) {
if (infile_data) {
mysql_set_local_infile_default(conn);
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Мы не должны вызвать mysql_thread_init(); перед этой строкой?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Хм, всё равно это восстановление и если дальше будет использоваться стандартный обработчик - в его инициализации функция всё равно вызовется. Но всё равно странный порядок вызовов:

mysql_set_local_infile_default(conn);
...
mysql_thread_init();
mysql_set_local_infile_default(conn);

Так что пусть будет.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

В случае стандартного load infile (без буфера), последовательность становиться

mysql_thread_init();
mysql_set_local_infile_default(conn);
mysql_thread_end(); //чтобы не было memory leak  если в libuv в пуле потоков  какой то закроется

в случае наличия буфера последовательность

mysql_set_local_infile_handler(...) 
// не зависит от mysql_thread_init
mysql_set_local_infile_default(conn); 
// в принципе не очень хорошо, так как стандартная реализация зависит от
// mysql_thread_init(), но не очень и плохо, так как в следующем запросе, 
// в случае надобности, будет вызван mysql_thread_init

@ghost ghost assigned Sannis Sep 6, 2012
@anton-kotenko
Copy link
Contributor Author

As I understand, you are busy with some bugs with Travis, but if there will be any problems with this pull request, please, write. I would like to help, as this feature is critical for me (it makes massive inserts into database 3-5 times faster that simple inserts)

@Sannis
Copy link
Owner

Sannis commented Sep 9, 2012

As I understand, you are busy with some bugs with Travis

Done!

but if there will be any problems with this pull request, please, write.

Can you rebase your branch onto upstream master?
I'm not commit-counter hunter, but it will be great to keep history more linear for such simple project.
I hope to have some time for this next evening.

add optional argument to  query/querySync method, with buffer,
that is used instead of file in load data infile local operation
TODO:
  test syncronous version of all this
  test if normal load data infile local works
  remove set option MYSQL_OPT_LOCAL_INFILE from InitLocalInfileCallbacks
@@ -1081,8 +1079,15 @@ void MysqlConnection::EIO_Query(uv_work_t *req) {
query_req->connection_closed = false;

MYSQLCONN_DISABLE_MQ;

pthread_mutex_lock(&conn->query_lock);
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now this is second call in this function that causes dead lock.

@Sannis
Copy link
Owner

Sannis commented Sep 10, 2012

Good work, Anton!

@anton-kotenko
Copy link
Contributor Author

Thanks, don't forget to update code, because I've just fixed bug (my, incorrect rebase) with locks (double lock in EIO_Query);

@Sannis
Copy link
Owner

Sannis commented Sep 10, 2012

Good to merge — The Travis build passed

Now passes :)

Sannis added a commit that referenced this pull request Sep 10, 2012
LOAD DATA LOCAL INFILE implemetation
@Sannis Sannis merged commit e6e0762 into Sannis:master Sep 10, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants