a really strange thing happening with get_or_create

[tborg]

Hi there. I love your engine. Usually it treats me rather well. A very funny thing seems to have happened by a freak coincidence. I am parsing some XML files. I have a Word class to record their occurrences. It looks like this:

class Word(Document):
    stem = StringField()
    count = IntField(default=1)
    forms = ListField(StringField(), default=list)
    occurs = ListField(EmbeddedDocumentField(Occurrence), default=list)

(My version of Python is 2.7.1, running on OSX Lion; this happens from an IDLE launched in a ZSH iTerm2 console, and as executed in a script launched from same.)

I ran into the issue while using a SAX parser; it kept choking on the word greenvill (which happened to be the very first word! I thought it wasn't working at all, at first). It hadn't done this to me ever before. So I pulled the class into an IDLE session, just to see if other stuff worked, or if it worked in different contexts or something, and discovered something bizarre. A quick sesh below:

>>> Word.objects.get_or_create(stem='g')
(<Word: Word object>, True)
>>> Word.objects.get_or_create(stem='gr')
(<Word: Word object>, True)
>>> Word.objects.get_or_create(stem='gre')
(<Word: Word object>, True)
>>> Word.objects.get_or_create(stem='gree')
(<Word: Word object>, True)
>>> Word.objects.get_or_create(stem='green')
(<Word: Word object>, False)
>>> Word.objects.get_or_create(stem='greenv')
(<Word: Word object>, True)
>>> Word.objects.get_or_create(stem='greenvi')
(<Word: Word object>, True)
>>> Word.objects.get_or_create(stem='greenvil')
(<Word: Word object>, True)
>>> Word.objects.get_or_create(stem='greenvill')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/mongoengine/queryset.py", line 799, in get_or_create
    doc = self.get(*q_objs, **query)
  File "/Library/Python/2.7/site-packages/mongoengine/queryset.py", line 761, in get
    result1 = self.next()
  File "/Library/Python/2.7/site-packages/mongoengine/queryset.py", line 916, in next
    return self._document._from_son(self._cursor.next())
  File "/Library/Python/2.7/site-packages/mongoengine/base.py", line 968, in _from_son
    else field.to_python(value))
  File "/Library/Python/2.7/site-packages/mongoengine/base.py", line 291, in to_python
    value_dict = dict([(key, self.field.to_python(item)) for key, item in value.items()])
  File "/Library/Python/2.7/site-packages/mongoengine/fields.py", line 414, in to_python
    return self.document_type._from_son(value)
  File "/Library/Python/2.7/site-packages/mongoengine/base.py", line 950, in _from_son
    class_name = son.get(u'_cls', cls._class_name)
AttributeError: 'unicode' object has no attribute 'get'
>>> Word.objects.get_or_create(stem='l')
Traceback (most recent call last):
# the Traceback is identical, and so omitted

So, as you can see, for whatever reason 'greenvill' and 'l' (I haven't discovered any others, or a pattern) cause the Attribute error. Why could this be? I am so confused. And even weirder: I tried it with a different class and label name, to much better effect:

>>> Leaf.objects.get_or_create(name='greenvill')
(<Leaf: Leaf object>, True)

so I added an artificial stem attribute to the leaf class, which by the way now looks like this:

class Leaf(Document):                                                                                                                                                             
    stem = StringField()                                                                                                                                                          
    name = StringField()                                                                                                                                                          
    kind = StringField()                                                                                                                                                          
    type = StringField()                                                                                                                                                          
    count = IntField(default=1)                                                                                                                                                   
    occurs = ListField(EmbeddedDocumentField(Occurrence)) 

and tried again:

>>> Leaf.objects.get_or_create(stem='greenvill')
(<Leaf: Leaf object>, True)

still good! What is it about the Word class? I changed the attribute name that I wanted, so that the Word class now looks like this:

class Word(Document):                                                                                                                                                             
    stems = StringField()                                                                                                                                                         
    count = IntField(default=1)                                                                                                                                                   
    forms = ListField(StringField(), default=list)                                                                                                                                
    occurs = ListField(EmbeddedDocumentField(Occurrence), default=list) 

and it works!

>>> Word.objects.get_or_create(stems='greenvill')
(<Word: Word object>, True)

but then when I switch back to stem instead of stems and try it again, still the same error with the same trace as above.

So, you know, this isn't really a big deal. But I am mystified, and I wonder if you might have some idea about what is causing this problem. In the meantime I'll just rename my attribute.

[rozza]

Hmm very odd - I'll look into this and get back to you :)

[rozza]

Ok I think you may have some bad data in your Word collection. The get_or_create is finding a match where stem = "greenvill" but the data itself is not being converted to the Word object properly.

This might be the result of changes in your Document definition. But running the same test but first doing a: Word.drop_collection() would work.