Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

discogs: Rate limiting #3081

Closed
arawde opened this Issue Nov 14, 2018 · 6 comments

Comments

Projects
None yet
3 participants
@arawde
Copy link

arawde commented Nov 14, 2018

Problem

I'm trying to run beets to import my music library, using the discogs plugin.

I run

$ beet import /mnt/music

Which yields:

Traceback (most recent call last):
  File "/usr/share/beets/beetsplug/discogs.py", line 226, in get_master_year
    year = result.fetch('year')
  File "/home/avr/.local/lib/python3.6/site-packages/discogs_client/models.py", line 245, in fetch
    self.refresh()
  File "/home/avr/.local/lib/python3.6/site-packages/discogs_client/models.py", line 211, in refresh
    data = self.client._get(self.data['resource_url'])
  File "/home/avr/.local/lib/python3.6/site-packages/discogs_client/client.py", line 123, in _get
    return self._request('GET', url)
  File "/home/avr/.local/lib/python3.6/site-packages/discogs_client/client.py", line 120, in _request
    raise HTTPError(body['message'], status_code)
discogs_client.exceptions.HTTPError: 429: You are making requests too quickly.

Some googling reveals that Discogs rate limits requests to about 3 per minute.

Running this command in verbose (-vv) mode:

$ beet -vv import /mnt/music

Causes the application to hang:

Candidate: Arctic Monkeys - Whatever People Say I Am, That’s What I’m Not (aa94ffcc-1c9b-39d6-953f-7aa803d8146c)
Computing track assignment...
...done.
Success. Distance: 0.01
discogs: Searching for master release 76279
discogs: Searching for master release 76279
discogs: Searching for master release 76279

Setup

  • OS: Ubuntu 18.10
  • Python version: 3.6.7
  • beets version: 1.4.7
  • Turning off plugins made problem go away (yes/no): Problem is with plugin

My configuration (output of beet config) is:

directory: /mnt/music
library: /mnt/music/beets.db
ignore: [0bleep, beats]
max_filename_length: 255
original_date: yes
per_disc_numbering: yes

ui:
    color: yes

import:
    copy: no
    move: yes
    timid: yes

paths:
    default: $albumartist/$album ($year) %aunique{}/$discn$track $title

plugins: inline edit discogs embedart fetchart replaygain scrub web
item_fields:
    discn: str(disc).lstrip('0') + '.' if disctotal > 1 else ''
discogs:
    source_weight: 0.0
    user_token: REDACTED
    apikey: REDACTED
    apisecret: REDACTED
    tokenfile: discogs_token.json
replaygain:
    backend: gstreamer
    overwrite: no
    auto: yes
    targetlevel: 89
    r128: [Opus]
web:
    host: 172.16.0.4
    port: 8000
    cors: '*'
    cors_supports_credentials: no
    reverse_proxy: no
    include_paths: no
edit:
    albumfields: album albumartist
    itemfields: track title artist album
    ignore_fields: id path
pathfields: {}
album_fields: {}
embedart:
    maxwidth: 0
    auto: yes
    compare_threshold: 0
    ifempty: no
    remove_art_file: no
scrub:
    auto: yes
fetchart:
    auto: yes
    minwidth: 0
    maxwidth: 0
    enforce_ratio: no
    cautious: no
    cover_names:
    - cover
    - front
    - art
    - album
    google_key: REDACTED
    google_engine: 001442825323518660753:hrh5ch1gjzm
    fanarttv_key: REDACTED
    store_source: no

@sampsyo sampsyo added the feature label Nov 14, 2018

@sampsyo sampsyo changed the title discogs plugin makes requests too quickly discogs: Rate limiting Nov 14, 2018

@sampsyo

This comment has been minimized.

Copy link
Member

sampsyo commented Nov 14, 2018

Thanks for pointing this out! Looks like we need to add rate limiting to the Discogs plugin (like we already have for MusicBrainz and a few other APIs).

Causes the application to hang:

I think the importer prompt is just being buried by the verbose output. (Try hitting enter.)

@arawde

This comment has been minimized.

Copy link
Author

arawde commented Nov 14, 2018

That was definitely it! I'm going to add the full traceback in case you need it:

Sending event: database_change
Reimported item added 1542237203.1288793 from item 216 for /mnt/music/Aphex Twin/Drukqs (2001)/2.15 Nanou 2.mp3
Reimported item flexible attributes dict_keys(['data_source', 'track_alt']) from item 216 for /mnt/music/Aphex Twin/Drukqs (2001)/2.15 Nanou 2.mp3
Sending event: database_change
Traceback (most recent call last):
  File "/usr/share/beets/beetsplug/discogs.py", line 226, in get_master_year
    year = result.fetch('year')
  File "/home/avr/.local/lib/python3.6/site-packages/discogs_client/models.py", line 245, in fetch
    self.refresh()
  File "/home/avr/.local/lib/python3.6/site-packages/discogs_client/models.py", line 211, in refresh
    data = self.client._get(self.data['resource_url'])
  File "/home/avr/.local/lib/python3.6/site-packages/discogs_client/client.py", line 123, in _get
    return self._request('GET', url)
  File "/home/avr/.local/lib/python3.6/site-packages/discogs_client/client.py", line 120, in _request
    raise HTTPError(body['message'], status_code)
discogs_client.exceptions.HTTPError: 429: You are making requests too quickly.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/beet", line 11, in <module>
    load_entry_point('beets==1.4.7', 'console_scripts', 'beet')()
  File "/usr/share/beets/beets/ui/__init__.py", line 1256, in main
    _raw_main(args)
  File "/usr/share/beets/beets/ui/__init__.py", line 1243, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/usr/share/beets/beets/ui/commands.py", line 943, in import_func
    import_files(lib, paths, query)
  File "/usr/share/beets/beets/ui/commands.py", line 913, in import_files
    session.run()
  File "/usr/share/beets/beets/importer.py", line 329, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/usr/share/beets/beets/util/pipeline.py", line 445, in run_parallel
    six.reraise(exc_info[0], exc_info[1], exc_info[2])
  File "/home/avr/.local/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/share/beets/beets/util/pipeline.py", line 312, in run
    out = self.coro.send(msg)
  File "/usr/share/beets/beets/util/pipeline.py", line 194, in coro
    func(*(args + (task,)))
  File "/usr/share/beets/beets/importer.py", line 1351, in lookup_candidates
    task.lookup_candidates()
  File "/usr/share/beets/beets/importer.py", line 641, in lookup_candidates
    autotag.tag_album(self.items, search_ids=self.search_ids)
  File "/usr/share/beets/beets/autotag/match.py", line 460, in tag_album
    va_likely):
  File "/usr/share/beets/beets/plugins.py", line 561, in decorated
    for v in generator(*args, **kwargs):
  File "/usr/share/beets/beets/autotag/hooks.py", line 616, in album_candidates
    for candidate in plugins.candidates(items, artist, album, va_likely):
  File "/usr/share/beets/beets/plugins.py", line 371, in candidates
    for candidate in plugin.candidates(items, artist, album, va_likely):
  File "/usr/share/beets/beetsplug/discogs.py", line 151, in candidates
    return self.get_albums(query)
  File "/usr/share/beets/beetsplug/discogs.py", line 216, in get_albums
    return [album for album in map(self.get_album_info, releases[:5])
  File "/usr/share/beets/beetsplug/discogs.py", line 216, in <listcomp>
    return [album for album in map(self.get_album_info, releases[:5])
  File "/usr/share/beets/beetsplug/discogs.py", line 303, in get_album_info
    original_year = self.get_master_year(master_id) if master_id else year
  File "/usr/share/beets/beetsplug/discogs.py", line 230, in get_master_year
    self._log.debug(u'API Error: {0} (query: {1})', e, result._uri)
AttributeError: 'Master' object has no attribute '_uri'
@arawde

This comment has been minimized.

Copy link
Author

arawde commented Nov 15, 2018

It looks like the Discogs client exposes a rate limit in the header of the response

Perhaps there is a way to check this before performing the next query

@sampsyo

This comment has been minimized.

Copy link
Member

sampsyo commented Nov 15, 2018

Neat! That should make it pretty straightforward.

@jattenberg

This comment has been minimized.

Copy link

jattenberg commented Jan 8, 2019

experiencing the same issue.

@sampsyo

This comment has been minimized.

Copy link
Member

sampsyo commented Feb 9, 2019

Fixed in #3138.

@sampsyo sampsyo closed this Feb 9, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.