data being indexed on elasticsearch does not get pushed to mongodb #32

Closed
forste opened this Issue Aug 23, 2012 · 6 comments

Comments

Projects
None yet
4 participants
@forste

forste commented Aug 23, 2012

I initialized the river as documented in the wiki. Replica set is set up, but I only have a single replication server.

When I insert data in mongodb it gets pushed to elasticsearch:

mongo
PRIMARY> use DBNAME
PRIMARY> entry = {    "user" : "phil",
...     "post_date" : "2009-11-15T14:12:12",
...     "message" : "trying out Elastic Search"}
{
    "user" : "phil",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elastic Search"
}

db.contacts.insert(entry)
$ curl -XGET 'http://localhost:9200/contacts/_search?pretty=true&size=5000' -d '
> { 
>     "query" : { 
>         "matchAll" : {} 
>     } 
> }'
{
  "took" : 62,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "contacts",
      "_type" : "contact",
      "_id" : "1",
      "_score" : 1.0, "_source" : {
    "user" : "phil",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elastic Search"
}
    } ]
  }

However, when I index data to elasticsearch issuing a PUT request, the data does not show up in mongodb (data from above has been cleared before executing the following sample)

$ curl -XPUT 'http://localhost:9200/contacts/contact/1' -d '{
>     "user" : "kimchy",
>     "post_date" : "2009-11-15T14:12:12",
>     "message" : "trying out Elastic Search"
> }'
{"ok":true,"_index":"contacts","_type":"contact","_id":"1","_version":1}forste@machine:~/opt$ 
$ url -XGET 'http://localhost:9200/contacts/_search?pretty=true&size=5000' -d '
{                     
    "query" : {                         
        "matchAll" : {} 
    } 
}'
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "contacts",
      "_type" : "contact",
      "_id" : "1",
      "_score" : 1.0, "_source" : {
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elastic Search"
}
    } ]
  }
}
$ mongo
PRIMARY> use DBNAME
PRIMARY> db.contacts.find()
PRIMARY> 

Does the river work both ways?

@richardwilly98

This comment has been minimized.

Show comment Hide comment
@richardwilly98

richardwilly98 Aug 23, 2012

Owner

Hi,

The river only import the data from MongoDB in ElasticSearch.

Thanks,
Richard.

On Thu, Aug 23, 2012 at 10:40 AM, Stephan notifications@github.com wrote:

I initialized the river as documented in the wiki. Replica set is set up,
but I only have a single replication server.

When I insert data in mongodb it gets pushed to elasticsearch:

mongo
use DBNAME
PRIMARY> entry = { "user" : "phil",
... "post_date" : "2009-11-15T14:12:12",
... "message" : "trying out Elastic Search"}
{
"user" : "phil",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elastic Search"
}

db.contacts.insert(entry)

$ curl -XGET 'http://localhost:9200/contacts/_search?pretty=true&size=5000' -d '

{
"query" : {
"matchAll" : {}
}
}'
{
"took" : 62,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [ {
"_index" : "contacts",
"_type" : "contact",
"_id" : "1",
"_score" : 1.0, "_source" : {
"user" : "phil",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elastic Search"
}
} ]
}

However, when I index data to elasticsearch via PUSH, the data does not
show up in mongodb

$ curl -XPUT 'http://localhost:9200/contacts/contact/1' -d '{

"user" : "kimchy",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elastic Search"

}'
{"ok":true,"_index":"contacts","_type":"contact","_id":"1","_version":1}forste@machine:~/opt$
$ url -XGET 'http://localhost:9200/contacts/_search?pretty=true&size=5000' -d '
{
"query" : {
"matchAll" : {}
}
}'
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [ {
"_index" : "contacts",
"_type" : "contact",
"_id" : "1",
"_score" : 1.0, "_source" : {
"user" : "kimchy",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elastic Search"
}
} ]
}
}

PRIMARY> db.contacts.find()
PRIMARY>

Does the river work both ways?


Reply to this email directly or view it on GitHubhttps://github.com/richardwilly98/elasticsearch-river-mongodb/issues/32.

Owner

richardwilly98 commented Aug 23, 2012

Hi,

The river only import the data from MongoDB in ElasticSearch.

Thanks,
Richard.

On Thu, Aug 23, 2012 at 10:40 AM, Stephan notifications@github.com wrote:

I initialized the river as documented in the wiki. Replica set is set up,
but I only have a single replication server.

When I insert data in mongodb it gets pushed to elasticsearch:

mongo
use DBNAME
PRIMARY> entry = { "user" : "phil",
... "post_date" : "2009-11-15T14:12:12",
... "message" : "trying out Elastic Search"}
{
"user" : "phil",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elastic Search"
}

db.contacts.insert(entry)

$ curl -XGET 'http://localhost:9200/contacts/_search?pretty=true&size=5000' -d '

{
"query" : {
"matchAll" : {}
}
}'
{
"took" : 62,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [ {
"_index" : "contacts",
"_type" : "contact",
"_id" : "1",
"_score" : 1.0, "_source" : {
"user" : "phil",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elastic Search"
}
} ]
}

However, when I index data to elasticsearch via PUSH, the data does not
show up in mongodb

$ curl -XPUT 'http://localhost:9200/contacts/contact/1' -d '{

"user" : "kimchy",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elastic Search"

}'
{"ok":true,"_index":"contacts","_type":"contact","_id":"1","_version":1}forste@machine:~/opt$
$ url -XGET 'http://localhost:9200/contacts/_search?pretty=true&size=5000' -d '
{
"query" : {
"matchAll" : {}
}
}'
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [ {
"_index" : "contacts",
"_type" : "contact",
"_id" : "1",
"_score" : 1.0, "_source" : {
"user" : "kimchy",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elastic Search"
}
} ]
}
}

PRIMARY> db.contacts.find()
PRIMARY>

Does the river work both ways?


Reply to this email directly or view it on GitHubhttps://github.com/richardwilly98/elasticsearch-river-mongodb/issues/32.

@karansingh1000

This comment has been minimized.

Show comment Hide comment
@karansingh1000

karansingh1000 Sep 11, 2012

Hi Richard,

Your plugin/river is superb.

I needed some help.
How do i make your river index multiple collections of MongoDB at the same time.

I want the river to index two collections at the same time instead of only one.
Is there any way to do that...

Thanks
Jaskaran

Hi Richard,

Your plugin/river is superb.

I needed some help.
How do i make your river index multiple collections of MongoDB at the same time.

I want the river to index two collections at the same time instead of only one.
Is there any way to do that...

Thanks
Jaskaran

@richardwilly98

This comment has been minimized.

Show comment Hide comment
@richardwilly98

richardwilly98 Sep 11, 2012

Owner

Hi,

In the current version one river setting will only monitor one MongoDB collection so you could create one river for each MongoDB collection to monitor.

curl -XPUT "localhost:9200/_river/mongodb/_meta" ....
curl -XPUT "localhost:9200/_river/mongodb2/_meta" ....

Thanks,
Richard.

Owner

richardwilly98 commented Sep 11, 2012

Hi,

In the current version one river setting will only monitor one MongoDB collection so you could create one river for each MongoDB collection to monitor.

curl -XPUT "localhost:9200/_river/mongodb/_meta" ....
curl -XPUT "localhost:9200/_river/mongodb2/_meta" ....

Thanks,
Richard.

@karansingh1000

This comment has been minimized.

Show comment Hide comment
@karansingh1000

karansingh1000 Sep 12, 2012

Hi Richard,

Thanks a lot for your help.
It works now :-)
Great Thank you very much....

Thanks
Jaskaran

Hi Richard,

Thanks a lot for your help.
It works now :-)
Great Thank you very much....

Thanks
Jaskaran

@nishtha-singh

This comment has been minimized.

Show comment Hide comment
@nishtha-singh

nishtha-singh Mar 13, 2014

Hi Richard,
I followed your blog and had set up mongodb river plugin for elastic search.
I followed all the steps in sequence and in addition to them I had successfully setup replica sets for Mongodb and inserted data into the collection person of database testmongo but I'm getting the same output again and again which is
{"took":2,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}
I am specifying the version as well
Mongodb - 2.4.9
ElasticSearch 1.0.1
elastic search mapper attachments 1.6.0
river mongodb plugin 1.6.1
Also I tried several solutions but none of them worked for me. Any suggestion would be of great help.
Thanks,
Nishtha

Hi Richard,
I followed your blog and had set up mongodb river plugin for elastic search.
I followed all the steps in sequence and in addition to them I had successfully setup replica sets for Mongodb and inserted data into the collection person of database testmongo but I'm getting the same output again and again which is
{"took":2,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}
I am specifying the version as well
Mongodb - 2.4.9
ElasticSearch 1.0.1
elastic search mapper attachments 1.6.0
river mongodb plugin 1.6.1
Also I tried several solutions but none of them worked for me. Any suggestion would be of great help.
Thanks,
Nishtha

@richardwilly98

This comment has been minimized.

Show comment Hide comment
@richardwilly98

richardwilly98 Mar 20, 2014

Owner

@nishtha-singh with ES 1.0.X you should use the river release 2.0.0:

bin/plugin --install com.github.richardwilly98.elasticsearch/elasticsearch-river-mongodb/2.0.0

Owner

richardwilly98 commented Mar 20, 2014

@nishtha-singh with ES 1.0.X you should use the river release 2.0.0:

bin/plugin --install com.github.richardwilly98.elasticsearch/elasticsearch-river-mongodb/2.0.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment