New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AB-354: Include a sleep schedule of 5 seconds after every batch import in musicbrainz importer #288

Merged
merged 5 commits into from Jul 24, 2018
View
@@ -66,3 +66,8 @@ FEATURE_EVAL_LOCATION = False
# Maximum number of recordings to fetch at a time for importing MusicBrainz metadata.
RECORDINGS_FETCHED_PER_BATCH = 10000
# Sleep duration for musicbrainz importer to wait after a complete import and
# between every 2 batches
SLEEP_DURATION = 30 # number of seconds to wait between runs
BATCH_SLEEP_DURATION = 5 # number of seconds to wait between batches
View
@@ -5,8 +5,10 @@
import logging
from flask import current_app
BATCH_SLEEP_DURATION = 5 # number of seconds to wait between batches
logging.basicConfig(format='%(asctime)s - %(levelname)s - %(message)s', level=logging.INFO)
def load_artist_credit(connection, MB_release_data, MB_release_group_data, MB_track_data, MB_artist_credit_name_data, artist_credit_from_recording):
"""Fetch artist_credit table data from MusicBrainz database for the
recording MBIDs in AcousticBrainz database.
@@ -2121,6 +2123,9 @@ def start_import():
if gids_in_AB:
fetch_and_insert_musicbrainz_data(gids_in_AB)
batch_sleep = current_app.config['BATCH_SLEEP_DURATION']
logging.info("Sleeping %s seconds before starting next batch's import." % batch_sleep)
time.sleep(batch_sleep)
else:
break
logging.info('Done!')
@@ -2,8 +2,10 @@
import time
import db.data
import db.import_mb_data
from flask import current_app
SLEEP_DURATION = 30 # number of seconds to wait between runs
BATCH_SLEEP_DURATION = 5 # number of seconds to wait between batches
def main():
@@ -14,6 +16,10 @@ def main():
logging.info("Importing MusicBrainz data...")
logging.info('Inserting data for %d recordings...' % (len(gids_in_AB)))
db.import_mb_data.fetch_and_insert_musicbrainz_data(gids_in_AB)
batch_sleep = current_app.config['BATCH_SLEEP_DURATION']
logging.info("Sleeping %s seconds before starting next batch's import." % batch_sleep)
time.sleep(batch_sleep)
else:
logging.info("No new recording found. Sleeping %s seconds." % SLEEP_DURATION)
time.sleep(SLEEP_DURATION)
sleep = current_app.config['SLEEP_DURATION']
logging.info("No new recording found. Sleeping %s seconds." % sleep)
time.sleep(sleep)
ProTip! Use n and p to navigate between commits in a pull request.