-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Open
Description
Problem
Using even a simple album_field
definition such as:
album_fields:
album_bitrate: |
total = 0
for item in items:
total += item.bitrate
significantly degrades performance.
Without:
root@6ea1b5cf64a4:/# time beet move -p 'title:Von dutch'
Moving 3 items.
[...]
real 0m2.702s
user 0m4.211s
sys 0m0.399s
With:
Moving 3 items.
[...]
real 0m15.705s
user 0m5.028s
sys 0m1.892s
With more items, it was significantly worse:
w/o:
root@6ea1b5cf64a4:/# time beet move -p 'artist:Charli XCX'
Moving 175 items.
[...]
real 0m5.264s
user 0m6.991s
sys 0m0.467s
with: (I actually gave up letting it run)
Moving 175 items.
[...]
^CTraceback (most recent call last):
File "/lsiopy/lib/python3.12/site-packages/beets/ui/__init__.py", line 1870, in main
_raw_main(args)
File "/lsiopy/lib/python3.12/site-packages/beets/ui/__init__.py", line 1849, in _raw_main
subcommand.func(lib, suboptions, subargs)
File "/lsiopy/lib/python3.12/site-packages/beets/ui/commands.py", line 2216, in move_func
move_items(
File "/lsiopy/lib/python3.12/site-packages/beets/ui/commands.py", line 2146, in move_items
objs = [o for o in objs if (isalbummoved if album else isitemmoved)(o)]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/ui/commands.py", line 2141, in isitemmoved
return item.path != item.destination(basedir=dest)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/library.py", line 1115, in destination
subpath = self.evaluate_template(subpath_tmpl, True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/dbcore/db.py", line 701, in evaluate_template
return t.substitute(
^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/util/functemplate.py", line 559, in substitute
res = self.interpret(values, functions)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/util/functemplate.py", line 552, in interpret
return self.expr.evaluate(Environment(values, functions))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/util/functemplate.py", line 240, in evaluate
out.append(part.evaluate(env))
^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/util/functemplate.py", line 177, in evaluate
arg_vals = [expr.evaluate(env) for expr in self.args]
^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/util/functemplate.py", line 240, in evaluate
out.append(part.evaluate(env))
^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/util/functemplate.py", line 145, in evaluate
if self.ident in env.values:
^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen _collections_abc>", line 813, in __contains__
File "/lsiopy/lib/python3.12/site-packages/beets/library.py", line 489, in __getitem__
value = self._get(key)
^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/library.py", line 475, in _get
return self._get_formatted(self.album, key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/dbcore/db.py", line 124, in _get_formatted
value = model._type(key).format(model.get(key))
^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/dbcore/db.py", line 449, in _get
return getters[key](self)
^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beetsplug/inline.py", line 120, in _func_func
func.__globals__.update(_dict_for(obj))
^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beetsplug/inline.py", line 102, in _dict_for
out["items"] = list(obj.items())
^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/library.py", line 1295, in items
return self._db.items(dbcore.MatchQuery("album_id", self.id))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/library.py", line 1697, in items
return self._fetch(Item, query, sort or self.get_default_item_sort())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/library.py", line 1675, in _fetch
return super()._fetch(model_cls, query, sort)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/dbcore/db.py", line 1257, in _fetch
rows = tx.query(sql, subvals)
^^^^^^^^^^^^^^^^^^^^^^
File "/lsiopy/lib/python3.12/site-packages/beets/dbcore/db.py", line 956, in query
cursor = self.db._connection().execute(statement, subvals)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
real 5m15.560s
user 0m19.222s
sys 0m36.162s
Setup
- OS: Unraid/Docker (linuxserver.io image)
- Python version: Python 3.12.10
- beets version: beets version 2.3.0
- Turning off plugins made problem go away (yes/no): Yes
My configuration (output of beet config
) is:
directory: /music/music
# --------------- Main ---------------
library: /config/library.db
pluginpath: /config/plugins
# --------------- Plugins ---------------
plugins:
- embedart
- fetchart
- info
- albumtypes
- lastgenre
- yearfixer
- fromfilename
- mbsync
- mbsubmit
- unimported
- zero
- missing
- inline
- badfiles
- substitute
- duplicates
- importreplace
- web
- originquery
# --------------- Performance ---------------
threaded: yes
# --------------- UI ---------------
verbose: no
# --------------- Tagging ---------------
per_disc_numbering: yes
import:
log: /config/beet.log
write: yes
copy: no
move: yes
resume: yes
incremental: yes
incremental_skip_later: yes
quiet: no
quiet_fallback: skip
from_scratch: no
default_action: apply
none_rec_action: ask
duplicate_action: ask
autotag: yes
art_filename: folder
mbcollection:
auto: yes
collection: XXXX
remove: yes
musicbrainz:
user: XXXX
pass: XXXX
extra_tags:
- year
- catalognum
- country
- media
- label
zero:
fields: comments
auto: yes
keep_fields: []
update_database: no
# --------------- Import ---------------
clutter:
- Thumbs.DB
- Thumbs.db
- .DS_Store
- '**.zip'
- '**.torrent'
- '**.ini'
- '**.txt'
- '**.nfo'
- '**.m3u'
missing:
format: $albumartist - $album - $title
count: no
total: no
album: no
match:
strong_rec_thresh: 0.08
max_rec:
missing_tracks: medium
unmatched_tracks: medium
preferred:
countries:
- US
- XE
- JP
- GB|UK
media: [Digital Media|File, CD, SACD]
original_year: no
item_fields:
is_mp3_aac: 1 if format == "MP3" or format == "AAC" else 0
is_flac: 1 if format == "FLAC" else 0
disc0: disc
disctotal0: disctotal
album_fields:
title0: title
albumtypes:
types:
- ep: 'EP - '
- single: 'Single - '
- live: 'Live - '
- remix: 'Remix - '
bracket: ''
ignore_va: [compilation]
paths:
default: '%substitute{$albumartist}/($original_year) $albumartist - %ifdef{altalbum,$altalbum,$album}%ifdef{transtitle, [$transtitle]} [%ifdef{disambig,$disambig - }%if{$atypes,$atypes}${format}%if{$is_mp3_aac, - $album_bitrate}%if{$is_flac, - $max_bitdepth - $max_samplerate}]/%if{$multidisc,Disc $disc0/}${track} - %ifdef{alttitle,$alttitle,$title}'
collection::^.+: _Other/$collection/($original_year) $albumartist - %ifdef{altalbum,$altalbum,$album}%ifdef{transtitle, [$transtitle]} [%ifdef{disambig,$disambig - }${format}%if{$is_mp3_aac, - $album_bitrate}%if{$is_flac, - $max_bitdepth - $max_samplerate}]/%if{$multidisc,Disc $disc0/}${track} - %ifdef{alttitle,$alttitle,$title}
substitute:
bladee &.*: Bladee
blu &.*: Blu
bob dylan &.*: Bob Dylan
broadcast and.*: Broadcast
casiopea.*: Casiopea
czarface.*: Czarface
"death\u2019s dynamic shroud and.*": "death\u2019s dynamic shroud"
dizzy gillespie.*: Dizzy Gillespie
drake &.*: Drake
"el\u2010p feat.*": "El\u2010P"
evaboy.*: miya lowe
fred again...*: Fred again..
.*gene clark with.*: Gene Clark
gorillaz feat.*|spacemonkeyz &.*: Gorillaz
"Hiromi\u2019s Sonicbloom|\u4E0A\u539F\u3072\u308D\u307F \u30B6\u30FB\u30C8\u30EA\u30AA\u30FB\u30D7\u30ED\u30B8\u30A7\u30AF\u30C8": Hiromi
iglooghost.*: Iglooghost
"jay\u2010z &.*": "JAY\u2010Z"
john williams.*: John Williams
jpegmafia.*: JPEGMAFIA
"kanye west.*|kids see ghosts|\\\xA5\\$": Kanye West
laufey with.*: Laufey
macroblank &.*: Macroblank
mf doom &.*|jj doom|dangerdoom|viktor vaughn|king geedorah|nehruviandoom|westsidedoom: MF DOOM
neil young &.*: Neil Young
quasimoto: Madlib
rita lee &.*: Rita Lee
round table.*: Round Table
ryusenkei.*: Ryusenkei
saint pepsi.*: SAINT PEPSI
"Stan Getz & Jo\xE3o Gilberto.*": "Stan Getz & Jo\xE3o Gilberto"
Sufjan Stevens &.*: Sufjan Stevens
thaiboy digital &.*: Thaiboy Digital
the moody blues.*: The Moody Blues
the velvet underground.*: The Velvet Underground
.*tommy heavenly.*: "Tommy february\u2076"
trent reznor & atticus ross.*: Trent Reznor & Atticus Ross
yung lean &.*: Yung Lean
^(.*?)( & Madlib| & The Alchemist).*: \1
importreplace:
replacements: [{item_fields: artist artist_credit artists artists_credit composer, album_fields: artist artist_credit artists artists_credit composer, replace: {'Ye(?!\w)': Kanye West, 'Charli xcx(?!\w)': Charli XCX, 'RYUSENKEI(?!\w)': Ryusenkei}}, {item_fields: artist_sort artists_sort composer_sort, album_fields: artist_sort artists_sort composer_sort, replace: {'Ye(?!\w)': 'West, Kanye', 'Charli xcx(?!\w)': Charli XCX, 'RYUSENKEI(?!\w)': Ryusenkei}}]
fetchart:
auto: no
cover_names: folder cover front
minwidth: 500
maxwidth: 3000
high_resolution: yes
enforce_ratio: 0.5%
sources:
- filesystem
- itunes
- amazon
- albumart
- coverart: releasegroup
deinterlace: yes
cover_format: JPEG
quality: 98
max_filesize: 0
cautious: no
store_source: no
google_key: REDACTED
google_engine: REDACTED
fanarttv_key: REDACTED
lastfm_key: REDACTED
embedart:
auto: no
maxwidth: 720
quality: 95
compare_threshold: 0
ifempty: no
remove_art_file: no
unimported:
ignore_extensions: jpg png zip jepg log cue yaml db
ignore_subdirectories: []
lastgenre:
whitelist: /config/whitelist.txt
auto: no
canonical: yes
count: 5
fallback: ''
force: yes
min_weight: 10
prefer_specific: no
source: album
title_case: yes
separator: '; '
keep_existing: no
extended_debug: no
originquery:
origin_file: origin.yaml
tag_patterns:
media: $.Media
year: $."Edition year"
label: $."Record label"
catalognum: $."Catalog number"
albumdisambig: $.Edition
web:
host: 0.0.0.0
port: 8337
cors: ''
cors_supports_credentials: no
reverse_proxy: no
include_paths: no
readonly: yes
duplicates:
album: no
checksum: ''
copy: ''
count: no
delete: no
format: ''
full: no
keys: []
merge: no
move: ''
path: no
tiebreak: {}
strict: no
tag: ''
pathfields: {}
yearfixer:
auto: no
force: no
mbsubmit:
format: $track. $title - $artist ($length)
threshold: medium
picard_path: picard
RollingStar
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
Milestone
Relationships
Development
Select code repository
Activity
RollingStar commentedon May 10, 2025
I had very similar code and I don't remember 5+ minutes to move 180 files. But it's been a while. Could also be a regression in the 7ish years it's been since I wrote my inline config.
https://gist.github.com/RollingStar/86e041338df295afbbf77a9027903068#file-beets_config-yaml-L281
BGarber42 commentedon May 11, 2025
If I have time I'll see if I can bisect a version to narrow it down.
BGarber42 commentedon May 11, 2025
Well I may have misattributed this to Inline since that just happened to make the issue less severe, but it seems more related to the path templating than the actual inline value generation.
Using a simple path template of:
It runs in about 3 seconds:
Running something slightly more advanced:
It's about the same:
But as soon as I put in one of the inline fields:
That's when it gets slow:
However if I make it an itemfield, it's fast again:
RollingStar commentedon May 12, 2025
I know nothing about how the inline code works. But, could it be multiplying the overhead by the number of tracks in the album? Say f(item) = 1 second, f(album) = 10 * f(item) if it has 10 tracks.
BGarber42 commentedon May 12, 2025
My test case was using a title query that I knew only returned a few tracks, in this case 3. However, it was split across (iirc) two albums, so if I need to repro again, I'd be sure to do it on a single album to see what the call counts end up as.
snejus commentedon May 13, 2025
Attempting to get to the root cause of this
Added a logging call to investigate interaction with the database.
Using my largest album that has 136 tracks:
Checking three sets of configurations
item_fields
item_fields
andalbum_fields
See the results
Case 1: No plugins enabled: 1.9s
Case 2: inline with
item_fields
: 2.2sCase 3: inline with
item_fields
andalbum_fields
: 11.2sWill suggest some improvements in the next comment.
snejus commentedon May 14, 2025
See #5784 for a potential fix:
Case 1: No plugins enabled: 1.7s
Case 2: inline with
item_fields
: 1.9sCase 3: inline with
item_fields
andalbum_fields
: 2.2sNote this introduces issues (see failing tests) as the cache needs resetting, but I've coded this in just to start the work towards an eventual fix.
BGarber42 commentedon Jun 8, 2025
As an aside, #5809 does a lot to speed this up also without having to rework a bunch of DB caching and having to worry about cache invalidation.
This is with multiple albumfields.
semohr commentedon Jun 11, 2025
Additionally to caching, I think this could also be an problem that could benefit from creating indexes for the database joins. See #5809 for reference.
If you still have the same setup, I would love to see the results for the same benchmarks while having the following database indexes:
Be sure to delete all indexes between benchmarks tho ;)
BGarber42 commentedon Jun 26, 2025
I've unfortunately moved data and tweaked a bunch of system things to get a reliable benchmark to compare with before.