Skip to content

Latest commit

 

History

History
43 lines (38 loc) · 1.91 KB

README.md

File metadata and controls

43 lines (38 loc) · 1.91 KB
Raw

selective-importer-4-ghtorrent

a fast and easy way to import a MySQL dump of GHTorrent, selecting only the tables and indexes you need

#HowTo

  • set the parameters to connect to your database in database_manager.py
  • set the importation parameters in settings.py (name of the dump file, tables to filter/select, etc.)
  • optionally you can modify the files create_triggers.py and create_indexes.py to 1) filter data while populating the database 2) define indexes on the columns you want
  • execute selective_ghtorrent_importer.py

#Tuning the target database In order to speed-up the importation process, you should modify the MySQL config.ini with the following settings:

# SETTINGS (tested)
# innodb_buffer_pool_size = 4G (70-80% of system memory)
# innodb_log_buffer_size = 256M (if the innodb_log_waits status variable is not 0, increase the buffer size)
# innodb_log_file_size = 1G (25%-50% of innodb_buffer_pool_size)
# innodb_flush_log_at_commit = 0
# innodb_flush_log_at_trx_commit = 0
# innodb_doublewrite = 0
# innodb_checksums = 0
# innodb_support_xa = 0
# init_connect = 'SET autocommit=0'
# key_buffer_size=64M
#
# OTHER SETTINGS (not tested)
# innodb_flush_method = O_DIRECT (for Linux)
# innodb_thread_concurrency = 17 or 2 x (number of cores + number of disks)
# innodb_read_io_threads = 16 (4 x number of cores)
# innodb_write_io_threads = 16 (4 x number of cores)
# max_allowed_packet = 32M (increase it if you have blob and text to insert)

#Evaluation Tests executed on a Intel Core i7-3720 QM CPU @ 2.60GHz 2.60GHz with just 4GB of RAM.

Unzipped Dump Size Triggers No Triggers Tables
19.2 GB 7h30m 1h51m all
42.2 GB skipped 12h10m all
63.7 GB skipped 19h all
96.3 GB skipped 28h all