no_dereference not working with to_json()

Hi ,

why no_derefernce() not working with to_json() ?

how can i pass values for angular for example ?

[JohnAD]

In both cases, the reference to Post has not been pulled. You are strictly getting the ObjectID not the corresponding Post contents (a dictionary containing the title and comments.)

Is there a different result you are wanting?

If, perhaps, you are wanting the opposite. The equivalent to an "always_dereference", you will need to manually force the dereferences. Something akin to:

page_list = []
for page in Page.objects:
    for post in page.posts: # getting the post should pull it (dereference it)
        pass
    page_list.append(page.to_json())

[stratosgear]

Hello,

Originally, I was thinking that Page.objects.select_related() ought to dereference all the Post objects but it is not working as I was expecting to.

For example this does not have "full" Post objects:

return jsonify([p for p in Page.objects.select_related()])

(I'm running it in Flask, as a response to a Vue application. I would like to respond the full contents of each Post, so I will not have to make further requests later on to populate Post related fields (OK, i'm not using Page/Posts but other entities but the issue remains the same)

Unfortunately, your little for loop trick @JohnAD still does not work for me, ending up with an array of "$oid"

Is this a regression error (I'm running v0.15.0 with pymongo 3.5.1 on mongo 3.6.3) ?

Thanks..

[Edit]: I created a new full issue for this #1780

[JohnAD]

Ah, my bad. to_json() creates a string not a dictionary, so there is no way to carry the dereference. I should have used to_mongo(). A functioning script:

from mongoengine import *
connect("test_database")

class Post(Document):
    title = StringField()
    text = StringField()

class Page(Document):
    posts = ListField(ReferenceField(Post))
    
p = Post(title="title 1", text="blah").save()
q = Post(title="title 2", text="blah2").save()
pg = Page(posts = [p, q]).save()

page_list = []
for page in Page.objects:
    page_dict = page.to_mongo()
    for n, post in enumerate(page.posts):
        page_dict["posts"][n] = post.to_mongo()
    page_list.append(page_dict.to_dict())

print(page_list)

creates the output:

[{'posts': [{'title': 'title 1', 'text': 'blah', '_id': ObjectId('5ae37b8285f48231303ffa91')}, {'title': 'title 2', 'text': 'blah2', '_id': ObjectId('5ae37b8285f48231303ffa92')}], '_id': ObjectId('5ae37b8285f48231303ffa93')}]

Or, to make it a string:

>>> import bson
>>> print(bson.json_util.dumps(page_list, indent=4))
[
    {
        "posts": [
            {
                "title": "title 1",
                "text": "blah",
                "_id": {
                    "$oid": "5ae37b8285f48231303ffa91"
                }
            },
            {
                "title": "title 2",
                "text": "blah2",
                "_id": {
                    "$oid": "5ae37b8285f48231303ffa92"
                }
            }
        ],
        "_id": {
            "$oid": "5ae37b8285f48231303ffa93"
        }
    }
]

[JohnAD]

But, while correct, this is a non-obvious solution. Interested in adding to the library with a PR?

Perhaps a method for a Document that let's you:

answer = doc.export_dict(deref_depth =2)

Where the depth parameter represents how many levels for dereferencing you are willing to descend. (Otherwise, infinite loops are very possible.)

[stratosgear]

Hey, thanks. That's quite a chunk of code, allright.

I tried for that PR, but got stuck while iterating ListFields with an error of TypeError: 'ListField' object is not iterable that I couldn't figure out.

The amount of introspection and abstraction in that library is too much for me, but I think it should be a much easier task for someone more familiar. And this is a utility function that would be quite useful, too.

[stratosgear]

But then again, what is the purpose of select_related anyways? Isn't it supposed to "walk" through all the referenced fields to force load them? It also has a max_depth param just like your proposed export_dict.

What is the use case of select_related? Apparently in this case it does not help!

[JohnAD]

There is more that one layer of late binding purposefully done in MongoDB.

If, for example, you where to query:

my_list = Fruit.objects(color="yellow")

The the resulting list had 9 entries in it, MongoEngine will not pull in any of the Fruit documents. If you were to then examine the 4th document:

print(my_list[3].name)

It would only query exactly one document from MongoDB; not reading the other documents until they are examined in any way.

The purpose of select_related is to skip the late binding and pull the documents immediately. To be honest, I don't know what max_depth means in this context. I'm curious. Will look at code.

[JohnAD]

Looking at the code, it is also used for an individual Document. So that:

apple = Fruit.objects(name="apple").first()

Does not pull the document; just the reference. Not until it is accessed anyway.

But,

apple = Fruit.objects(name="apple").first().select_related()

does pull it. Not entirely sure that is a very useful method. Still digging...

[JohnAD]

This does remind me: this also pulls the query itself. Another layer of late binding:

my_list = Fruit.objects(name="apple")

does not make a query yet. Nor does

apple = my_list.first()

The query is not made until you check the length of the results or actually access an item. Apparently select_related also forces the query. Makes sense.

[stratosgear]

Yes, it seems that there is tremendous amount of effort, through multiple layers of late binding to be as "lazy" as possible, but it seems it is too aggressive.

It might make sense in special cases were you're not interested in pulling in everything, but I still think that I should have the ability to clearly and easily be able to dereference everything (up to a max_depth). Especially in cases where I, as a progammer knowing the underlying data structure, know that I will not be pulling in half the database.

Also I'm pretty sure this could happen in an intelligent way, minimizing the amount of mongo queries (eg, doing a single $in operation for all items in a ListField, knowing all ObjectIDs, etc)

Maybe that code exists in one of the existing methods and I'm not seeing it? I'm no Mongoengine expert, just started using it, but I assumed that something as basic (?) as that, should be easily done?

Thanks for the investigative work though, your replies are quite informative.

[JohnAD]

Still investigating select_related. In the mean time, I have learned a quirk of to_json. To wit:

>>> y = Page.objects(id=pg.id).first()
>>> y.to_json()
'{"_id": {"$oid": "5ae5f1b885f48218d38f257d"}, "posts": [{"$oid": "5ae5f1b885f48218d38f257b"}, {"$oid": "5ae5f1b885f48218d38f257c"}]}'
>>> y.posts[0].title  # this absolutely causes a dereference
'title 1'
>>> y.to_json()
'{"_id": {"$oid": "5ae5f1b885f48218d38f257d"}, "posts": [{"$oid": "5ae5f1b885f48218d38f257b"}, {"$oid": "5ae5f1b885f48218d38f257c"}]}'

That to_mongo does not parse past a reference field makes sense. The reference fields really are just OIDs from MongoDB's point of view.

But to_json also has the same restraint. Viewing the source, it calls as_pymongo which explicitly states that it does not parse past reference fields. So, this isn't an accident, but is not self-evident.

I'm a big fan of mongoengine, but I've come to realize just how incomplete it's documentation is. Some day, I'll start working on that with more PRs.