Add dynamic fields in documents #112

Closed
wants to merge 25 commits into
from

Conversation

Projects
None yet

Menda commented Dec 19, 2010

I forked joshourisman fork for having the possibility to add extra fields to documents and merged it to trunk. It passes all tests. I ommited an annoying error when calling "create_dynamic_field" when that field exists.

An example of use could be:

class Category(Document):
    language_code = StringField(required=True)

    def save(self, *args, **kwargs):
        # Add extra args if they were found
        for k, v in self.__dict__.iteritems():
            if not k.startswith("_"):
                self.create_dynamic_field(k, v)
        super(Category, self).save(*args, **kwargs)

Note that save should be overriden just for classes which could have extra dynamic fields.

And then for example:

c = Category(language_code="es_ES", comment="This is a dynamic field")
c.save()

o = Category.objects.get(language_code="es_ES")
print o.comment # prints 'This is a dynamic field'

I think you should merge this into mongoengine because one of the main advantages of document databases is to have extra fields whenever you want, and many people like me need this feature.

Thanks!

Josh Ourisman and others added some commits Mar 4, 2010

Josh Ourisman Completed basic implementation of dynamic documents. Uses create_dyna…
…mic_field() method to initially define the dynamic fields, afterwords operates transparents through normal Document API. Tests included.
d705fda
Josh Ourisman moved dynamic document tests into separate file and using two differe…
…nt test cases in order to test across database connections
fe7a031
Josh Ourisman Appear to finally be successfully reading dynamic fields back from da…
…tabase; all tests pass.
af7bbd3
Josh Ourisman Merge branch 'master' of git://github.com/hmarr/mongoengine 3ef1ac0
Josh Ourisman merged in 0.3 changes 7877c4b
Josh Ourisman Added SortedListField 366cda5
Josh Ourisman set up SortedListField to work with non-collection datatypes 7f24a05
Josh Ourisman modified SortedListField to make ordering argument optional 0ea3ea8
Josh Ourisman added tests for SortedListField 60196c7
@joshourisman joshourisman Merge branch 'master' of git://github.com/hmarr/mongoengine 39a6953
@joshourisman joshourisman Merge branch 'SortedListField' 3985748
@joshourisman joshourisman Merge branch 'master' of git://github.com/hmarr/mongoengine 911ae83
@joshourisman joshourisman Merge branch 'master' of git://github.com/hmarr/mongoengine 6558840
@joshourisman joshourisman set up connection.py to reset connection if connect is called a secon…
…d time with a different database name
c5968f7
@joshourisman joshourisman set up connection.py to reset connection if connect is called a secon…
…d time with a different database name
d5fe9c1
@joshourisman joshourisman merged changes from ChangeConnection branch bab8507
@alex alex Fix changing databases 3c1e7af
@joshourisman joshourisman Merge branch 'master' of git://github.com/hmarr/mongoengine 2abb933
@joshourisman joshourisman upped version to 0.3.1 bea22a5
@joshourisman joshourisman Merge branch 'master' of git://github.com/alex/mongoengine 6b9f3a8
@joshourisman joshourisman upped version to 0.3.2 ac31ee6
@joshourisman joshourisman this version of mongoengine doesn't work with pymongo 1.9 8e53d0d
@joshourisman joshourisman upped version to 0.3.3 d75a47b
@Menda Menda Cloned joshourisman fork and merged upstream 0.4 1960a7a
@Menda Menda Don't raise error if the extra field has already been added cc7f358
Owner

hmarr commented Jan 9, 2011

Sorry it's taken me so long to get to this - I'm really keen on getting support for this. I'll check it out properly this week and almost certainly merge it in. Thanks a lot for contributing, I definitely agree that this is a great feature!

whats the status of this? id really like to use this functionality. anything I can do to help move it along?

Menda commented Feb 9, 2011

@justquick I think the only thing you can do to use this feature is cloning my repo and use this functionality.

@rozza rozza added a commit that referenced this pull request May 19, 2011

@rozza rozza Updated test name and changed BaseField calls to use db_field
BaseField name kwarg is being depreciated, in favour of db_field.
Cleaned up the tests to make independent.

Refs #112
8a18d57

rozza commented May 19, 2011

I've created a feature/dynamic_fields branch which is against the current dev branch.

Once reviewed will merge into dev

frankdu commented Jun 5, 2011

It's not in dev yet, right? I look forward to the feature, because it's a must feature in mongodb :-)

rozza commented Jun 6, 2011

@frankdu still being reviewed

rozza commented Jun 6, 2011

I'm not 100% sold on the syntax on this feature:

class Person(Document):
    name = StringField()
    age = IntField()

test_person = Person(name='Test')
test_person.create_dynamic_field(key, value)
test_person.save()

It feels slightly clunky, ideally it could be made more transparent - maybe a meta option on the class eg:

class Person(Document):
    name = StringField()
    age = IntField()

   meta = {'dynamic': True}

value = 1

test_person = Person(name='Test')
test_person.key = value
test_person.save()

Feels nicer / I may have a play and see how easy that would be.

Also, might it be worth storing field information as well, so to allow conversions back eg:

class Person(Document):
    name = StringField()
    age = IntField()

   meta = {'dynamic': True}

value = ListField([1,2,3])

test_person = Person(name='Test')
test_person.key = value
test_person.save()

frankdu commented Jun 6, 2011

@rozza +1 vote for your syntax comment. AS3 supports dynamic class. For example:

dynamic class Person {
var name:String;
}

The difference is that you can manipulate with properties not defined beforehand. Like below:
Person p= new Person();
p.name=”Joe”;
p.age=25;
p.printMe = function () {
trace (p.name, p.age);
}
p.printMe(); // Joe 25

However, such feature is supported by python inherently. I think it is better to have a DynamicDocument for this, or have a @dynamic decorator. Or, just let Document supports the feature. :-)

It can be as simple as:

class Tags(DynamicDocument):
    name = StringField()

t = Tags(name="test")
t.created_by = "mongoengine"
t.color = "green"

Menda commented Jul 20, 2011

@wpjunior It's pythonic, but maybe too easy for the developer to make a mistake and create non desirable fields?

Menda closed this Jul 20, 2011

Menda reopened this Jul 20, 2011

you can use this:

DynamicDocument and EmbeddedDynamicDocument attributes:

__delattr__(self, name): removes dinamic attribute
__setattr__(self, name, value)
: set a dinamic attribute
**getattr**(self, name)_: get a dinamic attribute

Simple?

see: http://docs.python.org/reference/datamodel.html#object.__delattr__

i like the sublcassing idea; not all documents should support dynamic fields. I do worry about how the dynamic fields are completely untyped in the example. it's just asking for a ton of weird ValueErrors later on when trying to get the fields from mongo back into python. Maybe the DynamicDocuments can do some sort of type guessing it could work:

class Tags(DynamicDocument):
    name = StringField()

t = Tags(name="test")
t.created_by = "mongoengine" # Guessed to be a StringField
t.color = StringField("green") # Explicitly a StringField

...but explicit is always better than implicit.

Also the google example is using BigTable which has got to be radically different than MongoDB when it comes to field typing so im not sure if it translates.

Regardless, an expando implementation for mongoengine would be amazing in any form.

rozza commented Jul 21, 2011

Guys theres also a thread in google groups about this - please contribute :)

https://groups.google.com/d/topic/mongoengine-dev/YgAC5AZTL58/discussion

btw I think the implementation is the easy part :)

tutuca commented Sep 8, 2011

Mongoid suports a similar convention. It's really handy. I Thought it was present in mongoengine (Asked this in irc and SO).

I would really like to see something like this in mongoengine.

dcrosta commented Sep 8, 2011

Why not just add __getattr__, __setattr__, and __delattr__ to DictField? You would lose validation, but you would still get back correct types, just as you would in the underlying dictionary given by pymongo. I can roll a patch if anyone's interested to see how it would look.

rozza commented Sep 9, 2011

@dcrosta go for it!

rozza closed this in a7edd86 Sep 28, 2011

rozza commented Sep 28, 2011

Added new classes DynamicDocument and DynamicEmbeddedDocument that handle dynamic data in an expando style.

See: a7edd86 for more information.

Usage example:

class Tags(DynamicDocument):
    name = StringField()

t = Tags(name="test")
t.created_by = "mongoengine"
t.color = "green"
t.save()

tkloc commented Oct 10, 2011

hey,

why DynamincDocument doesn't support boolean dynamic fields?

In [1]: from schemaless.models import *

In [2]: a = Schemaless(ident='a')

In [3]: a.save()

In [4]: a.test_int = 1

In [5]: a.test_char = 'abc'

In [6]: a.test_bool = False

In [7]: a.save()
> db.schemaless.find()                                                                                                                                                                                                                                                   
{ "_cls" : "Schemaless", "_id" : ObjectId("4e92afd36b669338b1000000"), "_types" : [ "Schemaless" ], "ident" : "a", "test_char" : "abc", "test_int" : 1 }                                                                                                                 

rozza commented Oct 10, 2011

@tkloc - added a new ticket: #311

tkloc commented Nov 29, 2011

Is there any way to delete dynamic field?

rozza commented Nov 29, 2011

From your model definitions? or from the database? You can do an update and unset ?

tkloc commented Nov 29, 2011

i tried :
del(a.cf_rtv_i_agdcf_float5me)
a._dynamic_fields.pop('cf_rtv_i_agdcf_float5me')
a._data.pop('cf_rtv_i_agdcf_float5me')
a.delattr('cf_rtv_i_agdcf_float5me')
delattr(a, 'cf_rtv_i_agdcf_float5me')

where 'a' is DynamicDocument object and 'cf_rtv_i_agdcf_float5me' is dynamic field name.

but after refreshing object field is still there:

In [32]: a.save()
In [33]: a = Auction.objects.get(id='4ecba3656b66935656000000')
In [34]: a.cf_rtv_i_agdcf_float5me
Out[34]: 321.0

i want to delete this field from database

rozza commented Nov 29, 2011

OK will have to check that out - can you open a new ticket?

Ross

On Tue, Nov 29, 2011 at 5:09 PM, tkloc <
reply@reply.github.com

wrote:

i tried :
del(a.cf_rtv_i_agdcf_float5me)
a._dynamic_fields.pop('cf_rtv_i_agdcf_float5me')
a._data.pop('cf_rtv_i_agdcf_float5me')
a.delattr('cf_rtv_i_agdcf_float5me')
delattr(a, 'cf_rtv_i_agdcf_float5me')

where 'a' is DynamicDocument object and 'cf_rtv_i_agdcf_float5me' is
dynamic field name.

but after refreshing object field is still there:

In [32]: a.save()
In [33]: a = Auction.objects.get(id='4ecba3656b66935656000000')
In [34]: a.cf_rtv_i_agdcf_float5me
Out[34]: 321.0

i want to delete this field from database


Reply to this email directly or view it on GitHub:
#112 (comment)

tkloc commented Nov 29, 2011

sure, #374

and if you have a little more time, please look at my last comment in #282 . It is also related to dynamic fields

rozza commented Nov 29, 2011

@tkloc thanks for your patience - a fix should be incoming tomorrow or
thurs.

On Tue, Nov 29, 2011 at 5:32 PM, tkloc <
reply@reply.github.com

wrote:

sure, #374

and if you have a little more time, please look at my last comment in #282
. It is also related to dynamic fields


Reply to this email directly or view it on GitHub:
#112 (comment)

Hi folks, I' need to use the mongodbforms for mongoengine DynamicDocuments. It's not implemented.
On my workaround I need to create a model called DynamicFields [1] to store all dynamic fields that will be render by form. With a patch on mongodbforms/documents [2]..

So, with these modification, at all Models I want to add dynamic_fields I have to
declare the Meta [3] to prepare mongodbforms parse the dynamic_fields.
Now if you have a extra field in your db.person, like age = 10, on models.DynamicFields you need to add:
DynamicFields(refer='Person', name='age', typo='intfield')

Finally at forms.py the PersonForm(DocumentForm) will render the fields+dynamic_fields. It's run..

My problem now is change the values of dynamic_fields. For declared fields it's ok, but dynamic no..

Any point me!? Thanks!

1 - myapp/models.py

class DynamicFields(Document):
    refer = StringField(max_length=120, required=True) 
    name = StringField(max_length=120, required=True)
    typo = StringField(required=True)
    max_length = IntField(required=False)
    choices = ListField(StringField(), required=False)

2 - mongodbforms/document.py

    # NOTE: Wrapper for dynamic_fields
    from mongoengine import *
    WRAPPER_GENERATE = dict(stringfield=StringField,
            emailfield = EmailField,
            urlfield = URLField,
            intfield = IntField,
            ...
            )

    def def fields_for_document(...):
        ...
        # NOTE: Workaround for dynamic_fields
        dynamic_fields = {}
        if hasattr(document, "Meta") and hasattr(document.Meta, "dynamic_fields"):
            for df in document.Meta.dynamic_fields:
                dynamic_fields[df.name] = WRAPPER_GENERATE[df.typo]()
                dynamic_fields[df.name].name = df.name
                dynamic_fields[df.name].max_length = df.max_length
        document._fields.update(dynamic_fields)

        sorted_fields = sorted(document._fields.values(), key=lambda field: field.__hash__())

3 - myapp/models.py - Example

class Person(DynamicDocument):
    name = StringField(required=True)
    class Meta:
        dynamic_fields = DynamicFields.objects.all()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment