New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

increasingly slow searches #49

Closed
daftscience opened this Issue Mar 16, 2015 · 5 comments

Comments

3 participants
@daftscience

daftscience commented Mar 16, 2015

I've run into an issue where searching gets slower over time. I know I may be pushing this TinyDB to it's performance limits. I have a database of around 10k dicts. Each Dict has five items.

So, Here is some psuedo code

class tiny_db():
   def __init__(self):
        self.db = TinyDB('db.json')
        self.table = self.db.table('_default', smart_cache=True)
   def file(self, item):
       self.table.insert(item)
   def locate(name)
       result = self.table.search(where('name') == name)
       return result

Every time I call tiny_db.locate(name) it takes 1 second longer than it did the last time I called. it.
I am probably doing something horribly wrong, Im not entirely sure how the caching middleware works, my grasp of python is pretty weak.

My duct tape fix has been to copy the db into a variable when my class initializes. I'm using that to search. But, I have to insert new items into both. I feel like there may be a better solution.

Sorry for the amateurish question, but thanks in advance.

@eugene-eeo

This comment has been minimized.

Show comment
Hide comment
@eugene-eeo

eugene-eeo Mar 16, 2015

Contributor

The smart cache only caches the same query- i.e. if the name argument passed to tiny_db.locate is always some constant. I don't see why you would hit any performance issues... can you inspect the live table's _cache property and see what you get? With that being said, I think you should try and disable the smart caching to see what happens.

Contributor

eugene-eeo commented Mar 16, 2015

The smart cache only caches the same query- i.e. if the name argument passed to tiny_db.locate is always some constant. I don't see why you would hit any performance issues... can you inspect the live table's _cache property and see what you get? With that being said, I think you should try and disable the smart caching to see what happens.

@msiemens

This comment has been minimized.

Show comment
Hide comment
@msiemens

msiemens Mar 16, 2015

Owner

@daftscience: You're definitely pushing limits here. With 10k elements you should at least consider moving to a more serious database that TinyDB.
That being said, performance across searches should be consistent. I also was able to reproduce this behaviour (10k elements, searching for an element not in the query cache takes 40s, 60s, 120s, 140s, ...). I'll definitely have a look at it!

Owner

msiemens commented Mar 16, 2015

@daftscience: You're definitely pushing limits here. With 10k elements you should at least consider moving to a more serious database that TinyDB.
That being said, performance across searches should be consistent. I also was able to reproduce this behaviour (10k elements, searching for an element not in the query cache takes 40s, 60s, 120s, 140s, ...). I'll definitely have a look at it!

@msiemens msiemens closed this in 1411daa Mar 16, 2015

@msiemens

This comment has been minimized.

Show comment
Hide comment
@msiemens

msiemens Mar 16, 2015

Owner

Found the offender. Basically, tinydb.utils.catch_warning (which was used in Query.__eq__ on Python 2.x only) failed to properly remove the handlers it added which resulted in O(n) time behaviour.

@daftscience Can you confirm this is fixed in the latest development version?

Owner

msiemens commented Mar 16, 2015

Found the offender. Basically, tinydb.utils.catch_warning (which was used in Query.__eq__ on Python 2.x only) failed to properly remove the handlers it added which resulted in O(n) time behaviour.

@daftscience Can you confirm this is fixed in the latest development version?

@daftscience

This comment has been minimized.

Show comment
Hide comment
@daftscience

daftscience Mar 17, 2015

Works like a charm. Thanks for the quick response.

daftscience commented Mar 17, 2015

Works like a charm. Thanks for the quick response.

@msiemens

This comment has been minimized.

Show comment
Hide comment
@msiemens

msiemens Apr 8, 2015

Owner

I've now finally released v2.3.0 which includes a fix for this :)

Owner

msiemens commented Apr 8, 2015

I've now finally released v2.3.0 which includes a fix for this :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment