GEO2D and ensure_index Support

[taylanpince]

I was wondering about GEO2D indexes and ensure_index support. As far as I can see, this is not supported in the current txmongo, but I am not sure how hard it would be to support. When I try to create_index a GEO2D index, I get the following exception:

sort_fields must be an instance of filter.sort

Trying to use ensure_index failed as well - this would be a nice to have.

Let me know if I can help in any way. I have a good knowledge of MongoDB but I am not that well-versed in Twisted so I didn't want to just fork start hacking, but I'll be happy to look at it if you point me in the right direction.

[fiorix]

Have you seen this example: https://github.com/fiorix/mongo-async-python-driver/blob/master/examples/index.py

[zman0900]

This is causing problems for me as well. I have seen the example, but geospatial indices need to be set up in a different way. They have to be specified like:
db.places.ensureIndex( { loc : "2d" } )
or, if you are not using the normal coordinate range, like:
db.places.ensureIndex( { loc : "2d" } , { min : -500 , max : 500 } )

I have spent some time studying the code, but I don't see an easy way to go about doing that without changing the code to txmongo. Check out this page for more info: http://www.mongodb.org/display/DOCS/Geospatial+Indexing

[kientzlecatch]

Here's a standalone _create_index() that should do what you need. It accepts sort lists (like pymongo) in addition to filter.sort objects and passes through additional kwargs, which should allow it to be used to create geo indexes, as well as support background=True and other modifiers. Not yet extensively tested.

from   bson.son               import SON
from   pymongo                import helpers
from   txmongo                import filter

"""Usage: _create_index(db.coll, [('key1', ASCENDING)], background=True) """

def _create_index(collection, sort_fields, unique=False, dropDups=False, **kwargs):
    def wrapper(result, name):
        return name

    if isinstance(sort_fields, filter.sort):
        keys = list(sort_fields["orderby"])
    else:
        keys = helpers._index_list(sort_fields)

    name = u"_".join([u"%s_%s" % item for item in keys])
    index = dict(
        ns=str(collection),
        name=name,
        key=helpers._index_document(keys))
        unique=unique,
        dropDups=dropDups,
    )
    index.update(kwargs)

    d = collection._database.system.indexes.insert(SON(index), safe=True)
    d.addCallback(wrapper, name)
    return d

[doncatnip]

Thx kientzlecatch, works quite well after fixing imports and syntax :)

But it's still a dirty workaround, depending on pymongo. So, I'd like to see a proper ensure_index implemented. Especially since geospacial indexing is a damn cool feature of the mongo.

[fiorix]

I wish I could do it but I'm quite busy with other stuff. So, I'll kindly wait for a pull request.

[dynamicgl]

Hi..I want to know GEO2D index has been available or not?

Thanks

[fiorix]

No. I haven't had the time to work on it.

[rnz0]

It seems that not only GEO2D indices need implementation but also the new GEOHAYSTACK (http://www.mongodb.org/display/DOCS/Geospatial+Haystack+Indexing). Use of unique and dropUps as parameters has been deprecated in pymongo. Geo adds more parameters, namely min, max and bucketSize, so the list may keep growing and makes sense to move all to **kwargs. I can propose a fix.

[gleicon]

@rnz0 supplied a pull request that I'm willing to merge. But as I dont really have a grasp on geo stuff using mongodb, I'd like to everyone in this ticket help testing so I can have material to write the docs (and learn more about the ins and outs). Please let me know who is testing so I can merge the pull request today if possible. Thanks

[psi29a]

@kientzlecatch Would be so kind as to make a pull request? :)

[psi29a]

@rnz0 Do you still have that pull request?

@gleicon I can also review and test this if the PR is there.

[gleicon]

thanks @psi29a I don't see the PR anymore. In any case if you want to work on it let me know how I can help.

[psi29a]

This looks to be partially enabled, we have GEO2D support but the ensure_index is just a wrapper around create_index.

As of pymongo 3.0, ensure_index is deprecated. So as far as I'm concerned, this can be closed.
http://api.mongodb.org/python/current/api/pymongo/collection.html#pymongo.collection.Collection.ensure_index

Any objections or comments?

[gleicon]

Ok for me. I've proceeded with some tests at the time this was opened but didn't had the full grasp of GEO + mongo to submit something meaningful. Ping me if you need any help. Thanks.

[fiorix]

+1