ListField containing ReferenceField's

[prydie]

The current multiple choice widget for ListField(ReferenceField(Foo)) doesn't provide a very good user experience and our team has written jQuery to create an interface that posts data in the same format as a ListField(StringField()).

Our current processing of this data is far from optimal and I wonder how it should be approached?

Thanks,
Andrew

[jschrewe]

To be honest I don't really know what you mean by "data in the same format as a ListField(StringField())".

I do know however that in the admin you can use the filter_horizontal option for ListField(ReferenceField(Foo)). So, if I wanted to make a nicer UI for those fields I would start there. The advantage would be that you only provide a bit of javascript niceness that falls back to multiple selects (which actually is the right widget albeit ugly as hell) without javascript. And you keep the already working validation provided by DocumentMultipleChoiceField.

If you want to use any other widget, you should be fine if you continue to use the DocumentMultipleChoiceField. The clean function will happily take a list of values that have primary keys in them. So for the standard mongoengine object ids it would look like [u'51db16d357a0200423af880e', u'51db16dc57a0200423af880f']. I'm not exactly sure, but I think a list like that is also passed to the widget's init.

In general the DocumentMultipleChoiceField should behave very much like Django's ModelMultipleChoiceField, so anything that works there should work for documents too (if primary keys other then integers are handled correctly).

Hope that helps.

[prydie]

By "data in the same format as a ListField(StringField())" I mean that ListField(StringField())'s default widget produces a <input type="text" name="field_name_0"/> and any additional <input>s with names in the format field_name_n (n = 1 .. n) are also added to the ListField on POST.

We have reused some of our jQuery and are generating <select name="field_name_n"> elements for the ListField(ReferenceField(Foo)) which are populated with Foo.objects.all(). I'd like to know if there is a nice way of getting the data together and populating the ListField rather than iterating over request.POST.

Thanks,
Andrew

[jschrewe]

Yeah there should be a way to do that. What you want to do is use a mongodbforms.fields.ListField as your formfield and pass in mongodbforms.fields.ReferenceField as field type into it. That should give you exactly the same construct as you get with ListField(StringField()).

So, your form class should look like this. You will need the newest mongodbforms commit though, because until a few minutes ago we only allowed field classes to be passed, not objects. Note: This seems to be buggy. Just though I'd let you know how it's supposed to work (and will work soon).

class ReferenceForm(DocumentForm):
    references = ListField(ReferenceField(queryset=your_queryset))
    ...

You should also be able to copy the first generated select if you want to insert more references with jQuery. Just make sure that you increment the "counter" in the id and name attribute in the select.

And validation is probably quite inefficient because every ReferenceField will run db queries to se if it's valid. If that is a problem, you will need to write a more specialised field (which should be doable if you combine the validate() of the ListField with the validate() from DocumentMultipleChoiceField.)

[jschrewe]

Okay, so using the following document and form I get something that should be what you need. This is actual test code I use, so it has some attributes just for testing stuff.

# documents.py
class Author(Document):
    name = StringField(min_length=2)

    def __unicode__(self):
        return self.name

class Book(Document):
    title = StringField(max_length=200, required=True, verbose_name="title", unique_with=['authors', 'language'])
    authors = ListField(ReferenceField(Author, dbref=False, reverse_delete_rule=CASCADE), required=True, verbose_name="authors")
    language = StringField(max_length=2, required=True, choices=LANGUAGES, verbose_name="language")

# forms.py
from mongodbforms import ListField, ReferenceField
from .documents import Book, Author

class BookForm(DocumentForm):
    authors = ListField(ReferenceField(queryset=Author.objects()))

    class Meta:
        document = Book

[prydie]

Thanks a lot. This seems to be working great as far as saving the posted variables. Were you envisioning the <select> being populated from the queryset in this commit? It appears not to be currently (in my case at least). This also means that passing an instance to the form to enable editing isn't working as far as I can see.

Edit: I was replying to your previous message. Just trying this out now

[jschrewe]

I'm not sure I understand what you mean. But if you want to auto generate this from documents (and not provide a form for every ListField(ReferenceField()), you need to use your own field generator. The following should do it:

# fieldgenerator.py
class MyFieldGenerator(MongoDefaultFormFieldGenerator):
   def generate_listfield(self, field, **kwargs):
        field = super(MyFieldGenerator, self).generate_listfield(field, **kwargs)
        if isinstance(field.field, MongoReferenceField):
            field = ListField(ReferenceField(queryset=field.field.document_type.objects.clone())
        return field

# settings.py
# set the fieldgeneretor for the whole application
MONGODBFORMS_FIELDGENERATOR = 'myproject.fieldgenerator.MyFieldGenerator'

After that you should get lots of selects for every ListField(ReferenceField()).

I hope I did mention that you need the newest version from github in order for this to work.

Hope that helps and is what you asked for.

[prydie]

The generator is a good idea although I've already got the form in place so keeping it in the form is fine for this project.

One bug is that it throws 'NoneType' object is not iterable when there isn't isn't already any fields assigned to the document.

Environment:
Request Method: GET
Request URL: http://127.0.0.1:8000/staff/templates/add/

Django Version: 1.5.2
Python Version: 2.7.2
Installed Applications:
('django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.sites',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.admin',
 'south',
 'djcelery',
 'mongodbforms',
 'bootstrapform',
 'post_office',
 'accounts',
 'staff',
 'project_name')
Installed Middleware:
('django.middleware.common.CommonMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware')


Template error:
In template /Users/andrew/Projects/project_name/site/templates/staff/add_product_template.html, error at line 49
   'NoneType' object is not iterable
   39 :               <span class="input-group-addon remove select"><i class="icon-remove icon-white"></i></span>
   40 :             </div>
   41 :           {% endif %}
   42 :           {% for error in form.fields.errors %}
   43 :             <span class="help-block">{{ error }}</span>
   44 :           {% endfor%}
   45 :         </div>
   46 :         <a class="btn btn-info choices-btn">Add Fields</a>
   47 :       </div>
   48 :     </div>
   49 :      {{ form.fields }} 
   50 :     <input class="btn btn-success" type="submit" value="Submit" />
   51 :     <a class="btn btn-default" href="{% url 'staff_index' %}">Cancel</a>
   52 :     <input type="hidden" name="next" value="{{ next }}" />
   53 :   </form>
   54 : {% endblock %}

Traceback:
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  115.                         response = callback(request, *callback_args, **callback_kwargs)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/views/generic/base.py" in view
  68.             return self.dispatch(request, *args, **kwargs)
File "/Users/andrew/Projects/project_name/site/project_name/staff/views.py" in dispatch
  23.         return super(CreateProductTemplateView, self).dispatch(request, *args, **kwargs)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/utils/decorators.py" in _wrapper
  25.             return bound_func(*args, **kwargs)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  25.                 return view_func(request, *args, **kwargs)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/utils/decorators.py" in bound_func
  21.                 return func(self, *args2, **kwargs2)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/utils/decorators.py" in _wrapper
  25.             return bound_func(*args, **kwargs)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  25.                 return view_func(request, *args, **kwargs)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/utils/decorators.py" in bound_func
  21.                 return func(self, *args2, **kwargs2)
File "/Users/andrew/Projects/project_name/site/project_name/accounts/views.py" in dispatch
  49.         return super(IsStaffMemberMixin, self).dispatch(request, *args, **kwargs)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/views/generic/base.py" in dispatch
  86.         return handler(request, *args, **kwargs)
File "/Users/andrew/Projects/project_name/site/project_name/staff/views.py" in get
  27.         return render(request, self.template_name, {'form': form, 'fields': self.product_fields})
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/shortcuts/__init__.py" in render
  53.     return HttpResponse(loader.render_to_string(*args, **kwargs),
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/template/loader.py" in render_to_string
  177.         return t.render(context_instance)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/template/base.py" in render
  140.             return self._render(context)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/template/base.py" in _render
  134.         return self.nodelist.render(context)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/template/base.py" in render
  830.                 bit = self.render_node(node, context)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/template/debug.py" in render_node
  74.             return node.render(context)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/template/loader_tags.py" in render
  124.         return compiled_parent._render(context)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/template/base.py" in _render
  134.         return self.nodelist.render(context)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/template/base.py" in render
  830.                 bit = self.render_node(node, context)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/template/debug.py" in render_node
  74.             return node.render(context)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/template/loader_tags.py" in render
  63.             result = block.nodelist.render(context)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/template/base.py" in render
  830.                 bit = self.render_node(node, context)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/template/debug.py" in render_node
  74.             return node.render(context)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/template/debug.py" in render
  87.             output = force_text(output)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/utils/encoding.py" in force_text
  99.                 s = s.__unicode__()
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/forms/forms.py" in __str__
  411.         return self.as_widget()
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/forms/forms.py" in as_widget
  458.         return widget.render(name, self.value(), attrs=attrs)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/django/forms/forms.py" in value
  496.         return self.field.prepare_value(data)
File "/Users/andrew/.virtualenvs/project_name/lib/python2.7/site-packages/mongodbforms/fields.py" in prepare_value
  235.         for v in value:

Exception Type: TypeError at /staff/templates/add/
Exception Value: 'NoneType' object is not iterable

[jschrewe]

Oh. Yeah I didn't test that. The last commit should handle that.

[prydie]

Yep, works like a charm. Thanks a lot for your help. If you're ever in the UK I owe you a beer or three.