Deprecating Rivers #10345

Closed
kimchy opened this Issue Mar 31, 2015 · 19 comments

Comments

Projects
None yet
@kimchy
Member

kimchy commented Mar 31, 2015

The reasoning behind it is captured in this blog post: https://www.elastic.co/blog/deprecating_rivers

We need to @Deprecate the River classes, and deprecate them in the docs, as well as add a deprecation notice in the rivers that are under elastic.

@clintongormley

This comment has been minimized.

Show comment
Hide comment
@clintongormley

clintongormley Apr 9, 2015

Member

Docs have been added to indicate that rivers are deprecated. Still need to add deprecation annotations to the code

Member

clintongormley commented Apr 9, 2015

Docs have been added to indicate that rivers are deprecated. Still need to add deprecation annotations to the code

@clintongormley clintongormley added v1.5.2 and removed v1.5.2 labels Apr 9, 2015

@javanna javanna removed the v1.5.1 label Apr 9, 2015

@dharshanr

This comment has been minimized.

Show comment
Hide comment
@dharshanr

dharshanr Apr 13, 2015

Is there a place to have a discussion on this? Looks like the problem has been punted rather than fixed. From a customer perspective it doesn't matter if the import code runs inside elastic or not - its more about importing data.

If stability was really an issue then a better solution would be to provide a standalone daemon that will be able to able to host the rivers and allow customers to import the data instead of leaving them to fend for themselves.

Is there a place to have a discussion on this? Looks like the problem has been punted rather than fixed. From a customer perspective it doesn't matter if the import code runs inside elastic or not - its more about importing data.

If stability was really an issue then a better solution would be to provide a standalone daemon that will be able to able to host the rivers and allow customers to import the data instead of leaving them to fend for themselves.

@carljohnstone

This comment has been minimized.

Show comment
Hide comment
@carljohnstone

carljohnstone Apr 15, 2015

Sorry I'm not hugely familiar with the code-base itself, but would it be possible to write a non-river plugin, that pulled in data from an external source than indexed it into Elasticsearch using the bulk-data APIs? Would you expect that to be any more stable than a plugin coded against the River classes? Eg is the actual problem the fact that these are plugins running within the ES server itself, rather than rivers.

Sorry I'm not hugely familiar with the code-base itself, but would it be possible to write a non-river plugin, that pulled in data from an external source than indexed it into Elasticsearch using the bulk-data APIs? Would you expect that to be any more stable than a plugin coded against the River classes? Eg is the actual problem the fact that these are plugins running within the ES server itself, rather than rivers.

@kimchy

This comment has been minimized.

Show comment
Hide comment
@kimchy

kimchy Apr 18, 2015

Member

@dharshanr / @carljohnstone in those cases, there really isn't any benefit in "Elasticsearch" itself having a framework for it, and its better left to be done by projects that are fully geared towards ETL such workloads, with native integration to Elasticsearch (which is where I think spending time would provide the best value). Logstash is a great example, but there are many more.

Member

kimchy commented Apr 18, 2015

@dharshanr / @carljohnstone in those cases, there really isn't any benefit in "Elasticsearch" itself having a framework for it, and its better left to be done by projects that are fully geared towards ETL such workloads, with native integration to Elasticsearch (which is where I think spending time would provide the best value). Logstash is a great example, but there are many more.

@nkavian

This comment has been minimized.

Show comment
Hide comment
@nkavian

nkavian Apr 28, 2015

I agree with @dharshanr this is punting the problem. We were enjoying using the Couchbase plugin to replicate into ES bypassing the app layer. Now what I'm understanding is the app layer now needs to copy into both places..

nkavian commented Apr 28, 2015

I agree with @dharshanr this is punting the problem. We were enjoying using the Couchbase plugin to replicate into ES bypassing the app layer. Now what I'm understanding is the app layer now needs to copy into both places..

@dadoonet

This comment has been minimized.

Show comment
Hide comment
@dadoonet

dadoonet Apr 28, 2015

Member

Couchbase plugin is a transport plugin not a river plugin so not concerned by this change.

Member

dadoonet commented Apr 28, 2015

Couchbase plugin is a transport plugin not a river plugin so not concerned by this change.

@nik9000

This comment has been minimized.

Show comment
Hide comment
@nik9000

nik9000 Apr 28, 2015

Contributor

@dharshanr / @carljohnstone in those cases, there really isn't any benefit in "Elasticsearch" itself having a framework for it

I figure the advantage of rivers is that you're already deploying Elasticsearch and they are already written. As much trouble as they are, they exist.

I'm still +1 on deprecating them.

Contributor

nik9000 commented Apr 28, 2015

@dharshanr / @carljohnstone in those cases, there really isn't any benefit in "Elasticsearch" itself having a framework for it

I figure the advantage of rivers is that you're already deploying Elasticsearch and they are already written. As much trouble as they are, they exist.

I'm still +1 on deprecating them.

@bgiromini

This comment has been minimized.

Show comment
Hide comment
@bgiromini

bgiromini May 12, 2015

The biggest hole in my opinion is getting data from a relational database into ES. I know that some work is being down on the logstash front but would be nice if it was officially supported by Elastic.

The biggest hole in my opinion is getting data from a relational database into ES. I know that some work is being down on the logstash front but would be nice if it was officially supported by Elastic.

@dadoonet

This comment has been minimized.

Show comment
Hide comment
Member

dadoonet commented May 12, 2015

@hash-include

This comment has been minimized.

Show comment
Hide comment
@hash-include

hash-include May 21, 2015

One quick question, If I am using river-mongodb and most of the company code is in Java. Can I push data into elasticsearch from Java API removing the river?

One quick question, If I am using river-mongodb and most of the company code is in Java. Can I push data into elasticsearch from Java API removing the river?

@dadoonet

This comment has been minimized.

Show comment
Hide comment
@dadoonet

dadoonet May 21, 2015

Member

@hash-include This is indeed my favorite way for doing that! Pushing from the application directly :)
I wrote a blog post about it though it's based on RDBMS and not NoSQL db: http://david.pilato.fr/blog/2015/05/09/advanced-search-for-your-legacy-application/

Member

dadoonet commented May 21, 2015

@hash-include This is indeed my favorite way for doing that! Pushing from the application directly :)
I wrote a blog post about it though it's based on RDBMS and not NoSQL db: http://david.pilato.fr/blog/2015/05/09/advanced-search-for-your-legacy-application/

@s1monw

This comment has been minimized.

Show comment
Hide comment
@s1monw

s1monw Jun 3, 2015

Contributor

@dadoonet do you know what is missing here to close this?

Contributor

s1monw commented Jun 3, 2015

@dadoonet do you know what is missing here to close this?

@dadoonet

This comment has been minimized.

Show comment
Hide comment
@dadoonet

dadoonet Jun 3, 2015

Member

@s1monw I think we can close it. It has been deprecated in 1.6.0 in docs and in 2.0.0 in code.
Note that we are not using the deprecation logger in master though: https://github.com/elastic/elasticsearch/blob/master/src/main/java/org/elasticsearch/river/RiversService.java#L129

Closing. If I'm missing anything, feel free to reopen.

Member

dadoonet commented Jun 3, 2015

@s1monw I think we can close it. It has been deprecated in 1.6.0 in docs and in 2.0.0 in code.
Note that we are not using the deprecation logger in master though: https://github.com/elastic/elasticsearch/blob/master/src/main/java/org/elasticsearch/river/RiversService.java#L129

Closing. If I'm missing anything, feel free to reopen.

@dadoonet dadoonet closed this Jun 3, 2015

@tchcxp

This comment has been minimized.

Show comment
Hide comment
@tchcxp

tchcxp Jun 22, 2015

@kimchy

I have been using mongodb river for a while. So the alternative solution is to update data in the application or other services with elastic API calls?

Thanks.

tchcxp commented Jun 22, 2015

@kimchy

I have been using mongodb river for a while. So the alternative solution is to update data in the application or other services with elastic API calls?

Thanks.

@dadoonet

This comment has been minimized.

Show comment
Hide comment
@dadoonet

dadoonet Jun 22, 2015

Member

@tchcxp I answered the same question a month ago: #10345 (comment)

HTH

Member

dadoonet commented Jun 22, 2015

@tchcxp I answered the same question a month ago: #10345 (comment)

HTH

@tchcxp

This comment has been minimized.

Show comment
Hide comment
@tchcxp

tchcxp Jun 22, 2015

@dadoonet

Cool! Great help. Thanks. 👍

tchcxp commented Jun 22, 2015

@dadoonet

Cool! Great help. Thanks. 👍

@syllogismos

This comment has been minimized.

Show comment
Hide comment
@syllogismos

syllogismos Nov 9, 2015

@tchcxp how well did updating data in elasticsearch from app layer work, instead of relying on mongodb river.

@tchcxp how well did updating data in elasticsearch from app layer work, instead of relying on mongodb river.

@monikamaheshwari

This comment has been minimized.

Show comment
Hide comment
@monikamaheshwari

monikamaheshwari May 31, 2017

@kimchy I want to replicate same set of data from mongoDB server to elasticSearch server and vice
versa. What should i use?

@kimchy I want to replicate same set of data from mongoDB server to elasticSearch server and vice
versa. What should i use?

@dadoonet

This comment has been minimized.

Show comment
Hide comment
@dadoonet

dadoonet May 31, 2017

Member

@monikamaheshwari please ask your question on discuss.elastic.co. I'm sure many users will be able to share their experience with you. Or better, search on discuss for previous discussions regarding this.

Member

dadoonet commented May 31, 2017

@monikamaheshwari please ask your question on discuss.elastic.co. I'm sure many users will be able to share their experience with you. Or better, search on discuss for previous discussions regarding this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment