Querying GenericReferenceField is broken for pre 0.8.4 data

[cybertoast]

Steps:

• Create a GenericReferenceField definition with 0.8.4 and save the parent and save

class Child(Document):
    name = StringField()    

class Parent(Document):
    child = GenericReferenceField()

child = Child(name='test')
child.save()
parent = Parent(child=child)
parent.save()

• Switch to using 0.8.7 and attempt to retrieve the data using:

child = Child.objects.first()
child = Parent.objects(child=child)
if not child:
    raise Exception("This is the problem")

The root cause seems to be that in 0.8.4 the GenericReferenceField creates:

"child" : { 
    "_ref" : DBRef("child", ObjectId("4e9f46cb6e48caa47f00084d")), 
    "_cls" : "child" 
}

but 0.8.5 creates:

"child" : { 
    "_cls" : "child"],
    "_ref" : DBRef("child", ObjectId("4e9f46cb6e48caa47f00084d"))
}

What seems to be happening is that the order of the dictionary is causing the query to fail. I say this because if I change the definition in mongoengine/fields.py as follows my query returns the referenced docs in my database with data that was created with <=0.8.4:

diff --git a/mongoengine/fields.py b/mongoengine/fields.py
index 82642cd..b32a9c6 100644
--- a/mongoengine/fields.py
+++ b/mongoengine/fields.py
@@ -1024,8 +1024,8 @@ class GenericReferenceField(BaseField):
         collection = document._get_collection_name()
         ref = DBRef(collection, id_)
         return SON((
-            ('_cls', document._class_name),
-            ('_ref', ref)
+            ('_ref', ref),
+            ('_cls', document._class_name)
         ))

     def prepare_query_value(self, op, value):

I'm not yet sure what the fix is since <=0.8.4 will have the older order and >0.8.4 will have the newer order for the dictionary keys.

This is a pretty big deal since legacy databases can't upgrade to 0.8.4+ since these genericreferences will not return correctly, from what I can tell.

Is there a way to handle both orders of keys?

I'm assuming this is the problem, but maybe there's a different root cause?

[lucashowell]

We too just encountered this issue and spent quite a long time debugging it. We will have to manually code up a migration script to migrate our mongo data unless there is a fix for this.

[lucashowell]

Here's a couple example scripts to correct this issue in case any one else encounters it. Fairly easy to resolve once you realize it, but surprisingly hard to debug / figure out why filtering on GenericReferenceField's just stops working after you upgrade mongoengine!

// Script for fixing plain GenericReferenceField()
db.<collection_name>.find().forEach(function(gm) {
    var obj_id = gm._id;
    var m_cls = gm.member._cls;
    var m_ref = gm.member._ref;

    print('Updating Generic Keys for <collection_name> ObjectId: ' + obj_id);

    db.<collection_name>.updateOne(
        { "_id" : obj_id },
        {
            $set: { "member" : { "_cls" : m_cls, "_ref" : m_ref } }
        }
    );

});

// Script for fixing ListField(GenericReferenceField())
db.<collection_name>.find().forEach(function(t) {
    var obj_id = t._id;
    var members_array = [];

    print('Updating Generic Keys for <collection_name> ObjectId: ' + obj_id);
    t.members.forEach(function(m) {
        members_array.push({"_cls":m._cls, "_ref":m._ref});
    });

    db.<collection_name>.updateOne(
        { "_id" : obj_id },
        { $set: { "members" : members_array } }
    );

});

[bagerard]

@erdenezul I believe we can close this. It is from 2014 and seems to relate to a breaking change in 0.8.x . We are now 0.15.x :)