Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lazily fetch album in FormattedItemMapping, it's not needed in most cases #3260

Merged
merged 7 commits into from May 12, 2019

Conversation

Projects
None yet
2 participants
@SimonPersson
Copy link
Contributor

commented May 9, 2019

Broken out from #3258, as it seems orthogonal.

The item formatter is fetching the related album for each item, even when it is not needed. This PR lazily fetches the album if it is really needed, and does not hit the DB in other cases.

@sampsyo

This comment has been minimized.

Copy link
Member

commented May 9, 2019

So far so good! Maybe we could make the code a little more legible by introducing a simple @lazy_property decorator; something like:

def lazy_property(func):
    field_name = '_' + func.__name__

    @property
    @functools.wraps(func)
    def wrapper(self):
        if hasattr(self, field_name):
            return getattr(self, field_name)

        value = func(self)
        setattr(self, field_name, value)
        return value

    return wrapper

This would work for all three of the properties that are currently manually made lazy, if I'm not mistaken.

@SimonPersson

This comment has been minimized.

Copy link
Contributor Author

commented May 9, 2019

Neat! I pushed a new commit, introducing lazy_property.

@SimonPersson

This comment has been minimized.

Copy link
Contributor Author

commented May 9, 2019

I looked into adding some tests, but it seems like test_library.py covers this functionality already.

@sampsyo
Copy link
Member

left a comment

Nice!! This looks great. And I agree that the test coverage is probably already good here. Would you mind doing just a few menial tasks?

  • Let’s move the new decorator to util. It could be useful elsewhere.
  • A short docstring for the decorator would be nice too.
  • Maybe a quick changelog entry would be in order.

Then we can measure the performance impact of this in isolation (both with and without album-level fields) and merge it!

@SimonPersson

This comment has been minimized.

Copy link
Contributor Author

commented May 10, 2019

Sure, I put lazy_property in util, added the docstring, and added a changelog entry. Here are some quick numbers from my machine:

(virtualenv) ~/beets $ git checkout lazy-album-fetch-in-formatter
M       tox.ini
Switched to branch 'lazy-album-fetch-in-formatter'
Your branch is up to date with 'fork/lazy-album-fetch-in-formatter'.
(virtualenv) ~/beets $ time beet -l /mnt/storage/beets/beets2019-05-05.blb ls | wc -l
14190

real    0m6.156s
user    0m5.929s
sys     0m0.348s
(virtualenv) ~/beets $ time beet -l /mnt/storage/beets/beets2019-05-05.blb ls -f '$artpath' | wc -l
14190

real    0m9.502s
user    0m8.605s
sys     0m1.264s
(virtualenv) ~/beets $ git checkout master
M       tox.ini
Switched to branch 'master'
Your branch is up to date with 'origin/master'.
(virtualenv) ~/beets $ time beet -l /mnt/storage/beets/beets2019-05-05.blb ls | wc -l
14190

real    0m10.440s
user    0m9.498s
sys     0m1.463s
(virtualenv) ~/beets $ time beet -l /mnt/storage/beets/beets2019-05-05.blb ls -f '$artpath' | wc -l
14190

real    0m9.447s
user    0m8.511s
sys     0m1.352s
(virtualenv) ~/beets $

Not a very thorough benchmark, but it looks pretty significant. Queries that don't use album level fields run about 1.5x slower on master compared to on lazy-album-fetch-in-formatter. Queries that do use the album level fields run about the same speed.

@sampsyo

This comment has been minimized.

Copy link
Member

commented May 12, 2019

Nice! I ran my own little performance test here, with three runs for each of the 4 treatments:

command master lazy-album-fetch-in-formatter
beet ls 7.21s, 7.10s, 7.08s 4.96s, 3.95s, 3.79s
beet ls -f '$artpath' 6.74s, 6.44s, 6.57s 7.03s, 6.45s, 6.46s

Long story short, that's a 1.7x speedup for the default format and no significant change for the format that includes an album-level field. Wahoo!!!

@sampsyo sampsyo merged commit 89aa5c7 into beetbox:master May 12, 2019

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@sampsyo

This comment has been minimized.

Copy link
Member

commented May 12, 2019

Merged! Just for fun, I did a quick comparison between the new master and the previous release, before all this focus on performance. v1.4.7 listed my database in an average of 13.0 seconds, and master got an average time of 3.9 seconds (across 3 runs each). That comes to a shockingly good 3.4x speedup! 🎉 🐎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.