New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Beets uses all available memory during import and then crashes #3082

Open
hjmuller opened this Issue Nov 23, 2018 · 12 comments

Comments

Projects
None yet
4 participants
@hjmuller
Copy link

hjmuller commented Nov 23, 2018

Problem

Beets program crashes after all memory appears to be consumed. I noticed that as an import continues, my memory usage increases. Once all system memory is consumed, beets crashes (see end of verbose output below). If I allocate more memory, then it only prolongs the crash. Doesn't seem to be related to specific files. Disabling all plugins seems to prevent the memory consumption and crash from happening.

Running this command in verbose (-vv) mode:

$ beet -vv import -qp /mnt/music/

Led to this problem:

scrub: auto-scrubbing /mnt/downloads/new_music_collection/Avantasia/The Mystery of Time_ A Rock Epic [2013]/10 - The Great Mystery.mp3
scrub: writing new tags after scrub
Sending event: write
Sending event: after_write
scrub: restoring art
scrub: auto-scrubbing /mnt/downloads/new_music_collection/Avantasia/The Mystery of Time_ A Rock Epic [2013]/07 - Invoke the Machine.mp3
chroma: chroma: fingerprinted '/mnt/music/Avantasia/The Scarecrow [2008]/04 - Carry Me Over.mp3'
chroma: no match found
scrub: writing new tags after scrub
Sending event: write
Sending event: after_write
scrub: restoring art
Sending event: album_imported
chroma: chroma: fingerprinted '/mnt/music/Avantasia/The Scarecrow [2008]/05 - What Kind of Love.mp3'
chroma: no match found
chroma: chroma: fingerprinted '/mnt/music/Avantasia/The Scarecrow [2008]/06 - Another Angel Down.mp3'
chroma: no match found
chroma: chroma: fingerprinted '/mnt/music/Avantasia/The Scarecrow [2008]/07 - The Toy Master.mp3'
chroma: no match found
chroma: chroma: fingerprinted '/mnt/music/Avantasia/The Scarecrow [2008]/08 - Devil in the Belfry.mp3'
chroma: no match found
chroma: chroma: fingerprinted '/mnt/music/Avantasia/The Scarecrow [2008]/09 - Cry Just a Little.mp3'
chroma: no match found
chroma: chroma: fingerprinted "/mnt/music/Avantasia/The Scarecrow [2008]/10 - I Don't Believe in Your Love.mp3"
chroma: no match found
chroma: chroma: fingerprinted '/mnt/music/Avantasia/The Scarecrow [2008]/11 - Lost in Space.mp3'
chroma: no match found
Looking up: /mnt/music/Avantasia/The Scarecrow [2008]
Tagging Avantasia - The Scarecrow
Searching for discovered album ID: f72ea7dc-5316-36ac-9833-2476d88b62e6
Sending event: albuminfo_received
Candidate: Avantasia - The Scarecrow
Success. Distance: 0.00
Album ID match recommendation is Recommendation.strong
ID match.
Sending event: import_task_start

/mnt/music/Avantasia/The Scarecrow [2008] (11 items)
Tagging:
    Avantasia - The Scarecrow
URL:
    http://musicbrainz.org/release/f72ea7dc-5316-36ac-9833-2476d88b62e6
(Similarity: 100.0%) (CD, 2008, BR, Nuclear Blast)
Sending event: import_task_choice
Sending event: import_task_apply
Killed

Setup

  • OS: Debian GNU/Linux 9.6 (stretch)
  • Python version: Python 2.7.13
  • beets version: 1.3.18
  • Turning off plugins made problem go away (yes/no): yes

My configuration (output of beet config) is:

## beets config file
## See https://beets.readthedocs.io for more info

directory: /mnt/downloads/new_music_collection
library: ~/.config/beets/library.db
threaded: yes
ui:
    color: yes
    colors:
        text_success: green
        text_warning: yellow
        text_error: red
        text_highlight: red
        text_highlight_minor: lightgray
        action_default: turquoise
        action: blue
import:
    write: yes ## Either yes or no, controlling whether metadata (e.g., ID3) tags are written to files when using beet import. Defaults to yes.
#    copy: no ## Either yes or no, indicating whether to copy files into the library directory when using beet import. Defaults to yes.
#    move: yes ## Either yes or no, indicating whether to move files into the library directory when using beet import. Defaults to no.
    resume: ask ## Controls whether interrupted imports should be resumed.
    log: ~/.config/beets/log.txt
    quiet_fallback: skip
    duplicate_action: skip

match:
    strong_rec_thresh: 0.02

paths:
    default: $albumartist/$album [$year]/$track - $title
    singleton: Non-Album/$artist/$title
    comp: Compilations/$album [$year]/$track - $title
    albumtype:soundtrack: Soundtracks/$album [$year]/$track - $title

plugins:
    chroma
    fromfilename
    scrub
    info
    missing
    duplicates

@sampsyo sampsyo added the needinfo label Nov 23, 2018

@sampsyo

This comment has been minimized.

Copy link
Member

sampsyo commented Nov 23, 2018

Hi! Is it true, as your issue form implied, that this occurs the same way even if you disable all the plugins?

@hjmuller

This comment has been minimized.

Copy link

hjmuller commented Nov 24, 2018

Hi! Is it true, as your issue form implied, that this occurs the same way even if you disable all the plugins?

I thought I had disabled them and still had the issue, but I tried again with disabled plugins and the crash/memory consumption doesn't occur.

I'm not sure which plugin is the culprit.

@sampsyo

This comment has been minimized.

Copy link
Member

sampsyo commented Nov 24, 2018

OK! Please post again if you can narrow it down.

@hjmuller

This comment has been minimized.

Copy link

hjmuller commented Nov 24, 2018

Looks like the chroma plugin is the problem. I re-enabled only that plugin, ran the import again, and the issue occurred. All memory was consumed and then beets crashed.

@sampsyo

This comment has been minimized.

Copy link
Member

sampsyo commented Nov 24, 2018

Got it; thanks! Any chance you've got GStreamer (and its Python bindings) installed? It's probably the same memory leak reported here: beetbox/audioread#62

@hjmuller

This comment has been minimized.

Copy link

hjmuller commented Nov 24, 2018

Got it; thanks! Any chance you've got GStreamer (and its Python bindings) installed? It's probably the same memory leak reported here: beetbox/audioread#62

I have a few gstreamer packages installed. Is there a recommended workaround?

$ dpkg-query -l | grep gstreamer
ii  gir1.2-gstreamer-1.0                 1.10.4-1                       amd64        GObject introspection data for the GStreamer library
ii  gstreamer1.0-plugins-base:amd64      1.10.4-1                       amd64        GStreamer plugins from the "base" set
ii  libgstreamer-plugins-base1.0-0:amd64 1.10.4-1                       amd64        GStreamer libraries from the "base" set
ii  libgstreamer1.0-0:amd64              1.10.4-1                       amd64        Core GStreamer libraries and elements
@sampsyo

This comment has been minimized.

Copy link
Member

sampsyo commented Nov 24, 2018

Well, if it is in fact that very memory leak, one thing you could do is uninstall the Python bindings for GStreamer (more specifically, the pygobject library). Or, if you're feeling like it, maybe you can help us track down what's causing the leak, as the aforelinked thread discusses!

@motorific

This comment has been minimized.

Copy link

motorific commented Dec 14, 2018

I also have this issue, running in a docker container. I'm happy to do testing that could assist with my limited knowledge of programming.

I have to abort the process when the memory gets high and then restart. It's corrupted the database a few times and i've had to start from scratch.

I am using the linuxserver version of the container.

@sampsyo

This comment has been minimized.

Copy link
Member

sampsyo commented Dec 14, 2018

@motorific, a great place to start would be to make sure this can be blamed on the chroma plugin. If so, the next place to look would be to investigate which audio-reading backend is being used—as mentioned above, to try uninstalling GStreamer, for example, to see if it's possible to blame that.

@motorific

This comment has been minimized.

Copy link

motorific commented Dec 14, 2018

@sampsyo It's for sure chroma.

I added this to my config.yaml:


chroma:
    auto: no

and there are no longer memory leak issues. Please let me know if i can provide other information that would be useful.

@sampsyo

This comment has been minimized.

Copy link
Member

sampsyo commented Dec 14, 2018

Yo! Can you try uninstalling GStreamer and seeing what happens?

@ssssam

This comment has been minimized.

Copy link

ssssam commented Jan 12, 2019

I suspect this is the issue: beetbox/audioread#84

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment