Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

beetsplug web: memory leak #2600

Closed
robot3498712 opened this issue Jun 17, 2017 · 8 comments
Closed

beetsplug web: memory leak #2600

robot3498712 opened this issue Jun 17, 2017 · 8 comments
Labels
needinfo We need more details or follow-up from the filer before this can be tagged "bug" or "feature." stale

Comments

@robot3498712
Copy link

robot3498712 commented Jun 17, 2017 •
edited

Problem

Memory leak.

Edit - minimal test case:

run 20 times in browser
http://127.0.0.1:8337/album/query/noresults234
{"results":[]}

beet web
127.0.0.1 - - [17/Jun/2017 18:17:46] "GET /album/query/disco HTTP/1.1" 200 -

Repeat the same request and monitor memory usage (python.exe).
JSON result for test case: 365
Memory usage (starting at about 15MB) about 50MB after ~20 requests; memory is not freed.
Real usage eventually results in consuming several hundred megabytes.

Item queries (several thousand results) seem to be fine or at least leaking a lot less.

Troubleshooting attempts:
gc.garbage showing empty list.
pympler :: tracker.print_diff() attached below.

Setup

  • OS:
  • Python version: Python 2.7.11 (v2.7.11:6d1b6a68f775, Dec 5 2015, 20:32:19) [MSC v.1500 32 bit (Intel)] on win32
  • beets version: beets version 1.4.4
  • Turning off plugins made problem go away (yes/no): problem with web plugin

beets version 1.4.4
Python version 2.7.11
plugins: web
Flask 0.12.2

Same behaviour for Python 3.6.1

My configuration (output of beet config) is:

import:
    copy: no
    write: no

plugins: web
web:
    cors: '*'
    include_paths: yes
    host: 127.0.0.1
    port: 8337
    reverse_proxy: no
@app.route('/album/query/<query:queries>')
@resource_query('albums')
def album_query(queries):
    tracker.print_diff()
    return g.lib.albums(queries)

                types |   # objects |   total size
===================== | =========== | ============
                  str |        6314 |    326.16 KB
                 list |        5055 |    258.44 KB
                 dict |         331 |    154.19 KB
                 code |         490 |     34.45 KB
                 type |          42 |     18.54 KB
                  set |          51 |     12.28 KB
              unicode |         184 |     10.72 KB
                tuple |         159 |      5.95 KB
                  int |         482 |      5.65 KB
     _sre.SRE_Pattern |          18 |      4.43 KB
             instance |          86 |      3.02 KB
              weakref |          62 |      2.66 KB
  function (__init__) |          29 |      1.70 KB
             classobj |          27 |      1.37 KB
                 cell |          50 |      1.37 KB
                     types |   # objects |   total size
========================== | =========== | ============
                      list |         392 |     21.37 KB
                       str |         383 |     16.78 KB
                   weakref |         168 |      7.22 KB
                      dict |           2 |    664     B
                       int |           5 |     60     B
  <class 'codecs.CodecInfo |           1 |     48     B
                   unicode |          -2 |    -56     B
                     tuple |          -7 |   -420     B
    types |   # objects |   total size
========= | =========== | ============
  weakref |         168 |      7.22 KB
     list |           3 |    816     B
     dict |           1 |    140     B
      int |           1 |     12     B
    types |   # objects |   total size
========= | =========== | ============
  weakref |         168 |      7.22 KB
     list |           3 |    816     B
     dict |           1 |    140     B
      int |           1 |     12     B
    types |   # objects |   total size
========= | =========== | ============
  weakref |         168 |      7.22 KB
     list |           3 |    816     B
     dict |           1 |    140     B
      int |           1 |     12     B
                    types |   # objects |   total size
========================= | =========== | ============
                  weakref |         168 |      7.22 KB
                     list |           3 |    816     B
  collections.defaultdict |           0 |    384     B
                     dict |           1 |    140     B
                      int |           1 |     12     B
    types |   # objects |   total size
========= | =========== | ============
  weakref |         168 |      7.22 KB
     list |           3 |    816     B
     dict |           1 |    140     B
      int |           2 |     24     B
    types |   # objects |   total size
========= | =========== | ============
  weakref |         168 |      7.22 KB
     list |           3 |    816     B
     dict |           1 |    140     B
      int |           1 |     12     B
    types |   # objects |   total size
========= | =========== | ============
  weakref |         168 |      7.22 KB
     list |           3 |    816     B
     dict |           1 |    140     B
      int |           1 |     12     B
    types |   # objects |   total size
========= | =========== | ============
  weakref |         168 |      7.22 KB
     list |           3 |    816     B
     dict |           1 |    140     B
      int |           1 |     12     B
      str |           0 |      1     B
    types |   # objects |   total size
========= | =========== | ============
  weakref |         168 |      7.22 KB
     list |           3 |    816     B
     dict |           1 |    140     B
      int |           1 |     12     B
    types |   # objects |   total size
========= | =========== | ============
  weakref |         168 |      7.22 KB
     list |           3 |    816     B
     dict |           1 |    140     B
      int |           1 |     12     B
    types |   # objects |   total size
========= | =========== | ============
  weakref |         168 |      7.22 KB
     list |           3 |    816     B
     dict |           1 |    140     B
      int |           1 |     12     B
    types |   # objects |   total size
========= | =========== | ============
  weakref |         168 |      7.22 KB
     list |           3 |    816     B
     dict |           1 |    140     B
      int |           1 |     12     B
    types |   # objects |   total size
========= | =========== | ============
  weakref |         168 |      7.22 KB
     list |           3 |    816     B
     dict |           1 |    140     B
      int |           1 |     12     B
    types |   # objects |   total size
========= | =========== | ============
  weakref |         168 |      7.22 KB
     list |           3 |    816     B
     dict |           1 |    140     B
      int |           1 |     12     B
    types |   # objects |   total size
========= | =========== | ============
  weakref |         168 |      7.22 KB
     list |           3 |    816     B
     dict |           1 |    140     B
      int |           1 |     12     B
    types |   # objects |   total size
========= | =========== | ============
  weakref |         168 |      7.22 KB
     list |           3 |    816     B
     dict |           1 |    140     B
      int |           1 |     12     B
    types |   # objects |   total size
========= | =========== | ============
  weakref |         168 |      7.22 KB
     list |           3 |    816     B
     dict |           1 |    140     B
      int |           1 |     12     B
    types |   # objects |   total size
========= | =========== | ============
  weakref |         168 |      7.22 KB
     list |           3 |    816     B
     dict |           1 |    140     B
      int |           1 |     12     B
    types |   # objects |   total size
========= | =========== | ============
  weakref |         168 |      7.22 KB
     list |           3 |    816     B
     dict |           1 |    140     B
      int |           1 |     12     B
                    types |   # objects |   total size
========================= | =========== | ============
                  weakref |         168 |      7.22 KB
  collections.defaultdict |           0 |      1.12 KB
                     list |           3 |    816     B
                     dict |           1 |    140     B
                      int |           1 |     12     B
    types |   # objects |   total size
========= | =========== | ============
  weakref |         168 |      7.22 KB
     list |           3 |    816     B
     dict |           1 |    140     B
      int |           1 |     12     B
    types |   # objects |   total size
========= | =========== | ============
  weakref |         168 |      7.22 KB
     list |           3 |    816     B
     dict |           2 |    280     B
      int |           1 |     12     B
                               types |   # objects |   total size
==================================== | =========== | ============
                                list |          21 |    283.62 KB
                                dict |          16 |      7.06 KB
                                 str |          46 |      2.00 KB
                               tuple |          16 |    616     B
          builtin_function_or_method |          10 |    360     B
                             unicode |           6 |    280     B
                                cell |           9 |    252     B
                      instancemethod |           5 |    200     B
                             weakref |           4 |    176     B
  <class 'flask.sessions.NullSession |           1 |    140     B
          <class 'socket._fileobject |           2 |    128     B
                                 int |           6 |     72     B
        <class 'threading._Condition |           2 |     64     B
          function (shutdown_server) |           1 |     60     B
                    function (write) |           1 |     60     B
                                      types |   # objects |   total size
=========================================== | =========== | ============
                                       list |        5136 |      1.16 MB
                                        str |        5155 |    222.64 KB
                                       dict |          20 |      6.48 KB
                                sqlite3.Row |         211 |      3.30 KB
                                        int |         278 |      3.26 KB
                                      tuple |          18 |    912     B
                                    unicode |          11 |    442     B
                 builtin_function_or_method |          11 |    396     B
                                    weakref |           6 |    264     B
                                       cell |           9 |    252     B
                             instancemethod |           6 |    240     B
         <class 'flask.sessions.NullSession |           1 |    140     B
                 <class 'socket._fileobject |           2 |    128     B
  <class 'beets.dbcore.query.SubstringQuery |           3 |     96     B
               <class 'threading._Condition |           2 |     64     B
                    types |   # objects |   total size
========================= | =========== | ============
                      str |       -1011 |     60.90 KB
                  weakref |         339 |     14.57 KB
                     code |         148 |     10.41 KB
         _sre.SRE_Pattern |          17 |      5.79 KB
                     type |           9 |      3.97 KB
                     dict |          -7 |      3.73 KB
      function (__init__) |           8 |    480     B
                 classobj |           9 |    468     B
                 property |          10 |    440     B
        getset_descriptor |          10 |    360     B
          function (save) |           3 |    180     B
      function (__repr__) |           3 |    180     B
  function (run_wsgi_app) |           2 |    120     B
       function (__str__) |           2 |    120     B
        function (setter) |           2 |    120     B
@sampsyo sampsyo added the needinfo We need more details or follow-up from the filer before this can be tagged "bug" or "feature." label Jun 17, 2017
@sampsyo
Copy link
Member

sampsyo commented Jun 17, 2017

Thanks for the report! If you have experience reading this output from this memory profiling tool, can you please help interpret its conclusions? I don't see any category consuming any suspiciously large amount of memory… the largest quantity is the total size of list objects, which is only 1.16 MB.

@robot3498712
Copy link
Author

Hi sampsyo,

this is me memory profiling for the first time - sorry, I can't assist much interpreting the results.
I was hoping it would help someone more familiar with beets to get a better idea where to look.
This isn't a supercritical issue - the server can be restarted periodically.

Thanks

@sampsyo
Copy link
Member

sampsyo commented Jun 17, 2017

Sure, that makes sense. But as it stands now, someone will more or less need to go digging from scratch. So please update this ticket if you come up with any more evidence about where to look for a problem.

@robot3498712
Copy link
Author

Thank you, I'll definitely try to dig myself ;)
Cheers

@robot3498712
Copy link
Author

Quick update - adding following to https://github.com/beetbox/beets/blob/master/beetsplug/web/__init__.py results in memory usage being stable:

@app.teardown_appcontext
def teardown_appcontext(error):
    g.lib._close()

(clearing connections in dbcore)
If this looks acceptable I'll submit a pull request.

@sampsyo
Copy link
Member

sampsyo commented Jun 19, 2017

Huh! I don't quite see why the teardown is necessary (vs. just letting the value get deleted), but this can't hurt. Yes, would you mind opening a PR?

robot3498712 pushed a commit to robot3498712/beets that referenced this issue Jun 19, 2017
fix: beetsplug web: memory leak beetbox#2600 - beetbox#2600
9d49e60
@robot3498712 robot3498712 mentioned this issue Jun 19, 2017
Closed
@4kind
Copy link

4kind commented Apr 30, 2020

Is this still an issue?

Setup

  • OS: Ubuntu 18.04.4 LTS (Linux 4.15.0-99-generic)
  • beets: 1.4.9
  • Python: 2.7.17 / 3.6.9
  • plugins: web
  • Flask 1.1.2

Tested with Python 2.7 and Python 3.6.
Called http://127.0.0.1:8337/album/query/noresults234 around 30 times.
No change in memory consumption.

                       types |   # objects |   total size
============================ | =========== | ============
                        list |        5933 |    603.41 KB
                         str |        5932 |    349.55 KB
                         int |         581 |     13.62 KB
                     weakref |          10 |    880     B
                        dict |           2 |    560     B
                       tuple |           2 |    184     B
                        code |           1 |    128     B
       function (store_info) |           1 |    120     B
                        cell |           2 |    112     B
                       float |           1 |     24     B
                 thread.lock |          -1 |    -32     B
  builtin_function_or_method |          -1 |    -72     B
              instancemethod |          -2 |   -160     B

@stale
Copy link

stale bot commented Jul 11, 2020

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Jul 11, 2020
@stale stale bot closed this as completed Jul 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
needinfo We need more details or follow-up from the filer before this can be tagged "bug" or "feature." stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@sampsyo @robot3498712 @4kind