Storing the location information from Mongo to ElasticSearch

[raviten]

I have latitude and longitude information as field but when i try to move it to elastic search it is considered as number rather than location.
please refer this question
http://stackoverflow.com/questions/29033305/geo-location-from-mongodb-to-elasticsearch

I have tried to change the data type to location using below post call.

http://localhost:9200/database/database.table/_mapping

{
"database": {
"mappings": {
"database.table": {
"properties": {
"geoip.location": {
"type": "geo_point"
}
}
}
}
}
}

[llvtt]

@black-star
Can you please provide:

• An example of a document in MongoDB containing location information that's being pushed to Elasticsearch.
• The command you're using to run mongo-connector.

[raviten]

@llvtt

example doc for containing location information:
{
"_id" : ObjectId("5502dff37659c667fddb1d92"),
"DATE" : ISODate("2015-03-13T13:02:43.594Z"),
"LOCATION_DETAILS": {
"area_code": "0",
"asn": "AS17488",
"continent_code": "AS",
"country": "India",
"country_code": "IN",
"country_code3": "IND",
"dma_code": "0",
"ip": "115.99.231.106",
"isp": "Hathway IP Over Cable Internet",
"latitude": 20,
"longitude": 77,
"offset": "5",
"timezone": "Asia/Kolkata"
},
"USERID": 37081164580880570,
"VALUE": 121,
"geoip": {
"location": [
-77.03653,
38.897676
],
"type": "Point"
}
}

Command for mongo-connector
mongo-connector -m localhost:27017 -t localhost:9200 -d elastic_doc_manager

[llvtt]

@black-star
I got this to work by doing the following:

1. Create an index manually on Elasticsearch before running Mongo Connector:

   curl -XPOST :9200/estest -d'{
       "mappings": {
         "geo": {
           "properties": {
             "location": {"type": "geo_point"}
           }
         }
       }
   }'
   

2. Start mongo-connector:

   mongo-connector -t localhost:9200 -d elastic_doc_manager -n estest.geo
   

3. Insert some documents into MongoDB:

   use estest
   db.geo.insert({"location": [100, 10]})
   

4. Make a query to Elasticsearch:

   curl :9200/estest/geo/_search -d'{
     "query": {
       "filtered": {
         "filter": {
           "geo_bounding_box": {
             "location": {
               "top_left":[80.0, 20.0] ,
               "bottom_right": [120.0, 0.0]
               }
             }
           }
         }
       }
   }'
   

I'm not sure if it's important or not to create the index manually on Elasticsearch before running the connector, but the geo-point documentation states that a dynamic mapping, which Mongo Connector creates, is not sufficient for Elasticsearch to detect points. Creating the index manually on Elasticsearch is a necessary step. I'm also not sure if and when changing the type of a field within a mapping creates the necessary index on that field for geo queries, so you might need to reload your data after you've created the appropriate mapping.

[raviten]

@llvtt
Thanks .,
It is helpful.

As index is created when database is created.

Is it possible to insert custom mappings along with
creation of index using mongo-connector config?

[llvtt]

Mongo Connector only creates a default, dynamic mapping when the first document is inserted into an index. Anything more complicated than that should be done manually before running Mongo Connector.

[aguyinmontreal]

Longitude must be stored first in MongoDB and latitude must be first in ElasticSearch. How do you deal with that?

[aguyinmontreal]

Nevermind, I just noticed the recent versions of ElasticSearch allow longitude first when an array is provided
(see https://www.elastic.co/guide/en/elasticsearch/reference/5.0/geo-point.html )