-
Notifications
You must be signed in to change notification settings - Fork 6.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] Add support for data write to MySQL storage table #2294
Conversation
Almost done, could someone help review this? |
@sundy-li as a potential user it would be nice to have a config option to allow ON DUPLICATE KEY |
I'm also interested in this task. Maybe we can work together to make it better?
Look forward to your reply @sundy-li : ) |
@zhang2014 Great to hear that. Let's work it together!!! |
dbms/src/Storages/StorageMySQL.cpp
Outdated
query.execute(); | ||
} | ||
|
||
Blocks splitBlocks(const Block & block, const size_t & max_rows) const |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Excessive copy when block is small enough.
dbms/src/Storages/StorageMySQL.cpp
Outdated
WriteBufferFromOwnString sqlbuf; | ||
sqlbuf << "INSERT INTO " << remote_database_name << "." << remote_table_name << " ( " << block.dumpNames() << " ) VALUES "; | ||
|
||
auto writer = std::make_shared<BlockOutputStreamFromRowOutputStream>(std::make_shared<ValuesRowOutputStream>(sqlbuf), sample_block); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably better way is to use FormatFactory (for looser coupling).
dbms/src/Storages/StorageMySQL.cpp
Outdated
void writeBlockData(const Block & block) | ||
{ | ||
WriteBufferFromOwnString sqlbuf; | ||
sqlbuf << "INSERT INTO " << remote_database_name << "." << remote_table_name << " ( " << block.dumpNames() << " ) VALUES "; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
backQuote names
As I remember, the most efficient way to insert data is |
I see no reason why it shouldn't work automatically.
And REPLACE. It can be postponed for the next commit. |
dbms/src/Storages/StorageMySQL.cpp
Outdated
void write(const Block & block) override | ||
{ | ||
auto blocks = splitBlocks(block, max_batch_rows); | ||
mysqlxx::Transaction trans(entry); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe inserting in a single transaction should be optional.
Sometimes user is going to export huge amount of data and would like to watch while it loads.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But user did't know how much data it inserts, it is hard to control. So I make it in a single transaction.
Because Table functions can only be specified in the FROM clause for select usage.
The Such as |
I created a issues[#2346] to track subsequent changes. |
We have |
Could you please also add something into |
@alexey-milovidov thanks for your detail comment, I will do those improvement. |
I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en