Skip to content
This repository has been archived by the owner on Aug 14, 2019. It is now read-only.

Load table data without decompressing first #64

Open
hectoras opened this issue May 13, 2013 · 1 comment
Open

Load table data without decompressing first #64

hectoras opened this issue May 13, 2013 · 1 comment

Comments

@hectoras
Copy link
Contributor

Currently we download, decompress and use LOAD DATA INFILE on https://github.com/elliotchance/mbzdb/blob/master/backend/mysql.pl, but I think decompressing to disk isn't really needed: we could decompress to a pipe and LOAD DATA from it, avoiding slow seek times and extra disk space needed for decompression :

mkfifo /tmp/fifo
tar -xjf mbdump.tgz artist -O > /tmp/fifo &
LOAD DATA LOCAL INFILE '/tmp/fifo' INTO TABLE ....

BTW, this should be configurable by user (fifo path, etc).

@hectoras
Copy link
Contributor Author

Probably for this to work we need to increase timeouts first:
set global net_write_timeout = 28800;
set global net_read_timeout = 28800;

http://blog.rasilon.net/articles/mysqldump_error_2013_lost_connection_to_mysql_server/index.html

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant