MongoEngine is very slow when fetching LazyReferenceField

[alterEgo123]

I'm working on referencing objects from the same class using MongoEngine stackoverflow_issue. This is my code below

import random
import string

from mongoengine import *

import timeit


class Person(Document):
    name = StringField()
    best_friend = LazyReferenceField('Person')
    friends = ListField(LazyReferenceField('Person'))

    def fetch_(self, attribute):
        return attribute.fetch().to_dict() if attribute else None

    def to_dict(self):
        return {
            'name': self.name,
            'best_friend': self.fetch_(self.best_friend),
            'friends': [self.fetch_(friend) for friend in self.friends] if self.friends else []

        }


def timer(function):
    def func(*args):
        start_time = timeit.default_timer()
        function(*args)
        elapsed = timeit.default_timer() - start_time
        print('Function "{name}" took {time} seconds to complete.'.format(name=function.__name__, time=elapsed))

    return func


def get_random_string():
    letters = string.ascii_lowercase
    result_str = ''.join(random.choice(letters) for _ in range(8))
    return result_str


@timer
def create_people(number):
    Person.drop_collection()
    everyone = []
    for _ in range(number):
        a = Person(name=get_random_string(), friends=everyone).save()
        b = Person(name=get_random_string(), best_friend=a.id, friends=everyone).save()
        everyone += [a.id, b.id]


@timer
def get_people():
    return [p.to_dict() for p in Person.objects()]


@timer
def main():
    connect()
    create_people(8)
    get_people()


main()

This is the program's execution time:

Function "create_people" took 0.0675540619995445 seconds to complete.
Function "get_people" took 45.54736871900968 seconds to complete.
Function "main" took 45.6171299170237 seconds to complete.

How I reduce the fetching query and add a cache using MongoEngine or at MongoDB level? Isn't MongoEngine supposed to cache results for the same query, even for a short period of time?

[bagerard]

The results are indeed being cached, but on the queryset objects, so it applies in the following scenario:

persons = Person.obejcts()
assert isinstance(persons, QuerySet)

# first iteration over the queryset does:
# 1) fire the query to the database and get pymongo dict back
# 2) load the dict into Person instances
#3) add the Person instances to queryset._cache
for person in persons:    
    pass
    

# second iteration over the queryset does:
# return the Person instances from the queryset._cache
for person in persons:    
    pass

In your case, the additional queries are being fired outside of the queryset's context, they are being fired from Person.to_dict on your Person instances and this is done in a loop.

To reflect the amount of work going on under the hood in your example, we can use query_counter , it gives a rough idea of the number of queries being fired

from mongoengine.context_managers import query_counter

@timer
def main():
    connect()
    create_people(8)
    with query_counter() as q:
        get_people()
        print("n queries fired during get_people():", q)

main()

Function "create_people" took 0.04241644899957464 seconds to complete.
Function "get_people" took 44.49782170300023 seconds to complete.
n queries fired during get_people(): 1143
Function "main" took 44.56344539700149 seconds to complete.

So under the hood, 1143 queries to the Person collection were fired (and due to the lazy .fetch(): 1143 Person instances were created)

The situation improves if you use plain ReferenceFIeld instead of Lazy ones

Function "create_people" took 0.06802764600070077 seconds to complete.
Function "get_people" took 12.158331843998894 seconds to complete.
n queries fired during get_people(): 840
Function "main" took 12.248315486001957 seconds to complete.

this is due to the ListField(ReferenceField) that gets loaded in 1 shot through 1 query (per Person), as opposed to how it works in your .fetch method (1 query per item in the Person.friends)

But here the problem is clearly in the recursion, the difficulty is that MongoEngine can't figure out all the person's IDs upfront (because of the way your call to_dict() and the recursion occuring within [p.to_dict for p in person.friends], and each friend, as its own friends, etc).

I assume this is not your real use case but maybe there is a way to improve this by modelling differently the relations, or by making to_dict() not going out of the document scope. Generally speaking having a to_dict() that does some dereferencing will perform poorly when executed in a loop (famous N+1 problem of ORM).

I hope this helps...

[alterEgo123]

I've used LazyReferenceField because it's stated in the documentation that using ReferenceField can lead to poor performances. I'll experiment with it and see what works best.

By adding a simple dict that acts like a cache, even if using LazyReferenceField there wasn't any performances issues.

import random
import string

from mongoengine import *

import timeit

cache = {}


class Person(Document):
    name = StringField()
    best_friend = LazyReferenceField('Person')
    friends = ListField(LazyReferenceField('Person'))

    def cache_or_get_from_cache(self, attribute):
        if not attribute:
            return

        _id = str(attribute.id)
        if _id not in cache:
            cache[_id] = self.fetch_(attribute)
        return cache[_id]

    def fetch_(self, attribute):
        return attribute.fetch().to_dict() if attribute else None

    def to_dict(self):
        return {
            'name': self.name,
            'best_friend': self.cache_or_get_from_cache(self.best_friend),
            'friends': [self.cache_or_get_from_cache(friend) for friend in self.friends] if self.friends else []

        }


def timer(function):
    def func(*args):
        start_time = timeit.default_timer()
        function(*args)
        elapsed = timeit.default_timer() - start_time
        print('Function "{name}" took {time} seconds to complete.'.format(name=function.__name__, time=elapsed))

    return func


def get_random_string():
    letters = string.ascii_lowercase
    result_str = ''.join(random.choice(letters) for _ in range(8))
    return result_str


@timer
def create_people(number):
    Person.drop_collection()
    everyone = []
    for _ in range(number):
        a = Person(name=get_random_string(), friends=everyone).save()
        b = Person(name=get_random_string(), best_friend=a.id, friends=everyone).save()
        everyone += [a.id, b.id]


@timer
def get_people():
    return [p.to_dict() for p in Person.objects()]


@timer
def main():
    connect()
    create_people(100)
    get_people()


main()

The results are the following

Function "create_people" took 0.5443411420565099 seconds to complete.
Function "get_people" took 0.7840798320248723 seconds to complete.
Function "main" took 1.3315250900341198 seconds to complete.

[bagerard]

We should indeed elaborate on why ReferenceField can lead to poor performance to avoid that developers use LazyReferenceField blindly, thinking it will improve. In reality it's not and it depends on how you manipulate your objects.

Your solution does the trick, it has a few drawback to note:

• it may end up holding everything in memory
• the cached object will get out of sync

I wouldn't build on top of that trick for complex or large applications but for simple use case, it's probably fine