Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecating Rivers #10345

Closed
kimchy opened this issue Mar 31, 2015 · 19 comments
Closed

Deprecating Rivers #10345

kimchy opened this issue Mar 31, 2015 · 19 comments

Comments

@kimchy
Copy link
Member

kimchy commented Mar 31, 2015

The reasoning behind it is captured in this blog post: https://www.elastic.co/blog/deprecating_rivers

We need to @Deprecate the River classes, and deprecate them in the docs, as well as add a deprecation notice in the rivers that are under elastic.

@clintongormley
Copy link

Docs have been added to indicate that rivers are deprecated. Still need to add deprecation annotations to the code

@javanna javanna removed the v1.5.1 label Apr 9, 2015
@dharshanr
Copy link

Is there a place to have a discussion on this? Looks like the problem has been punted rather than fixed. From a customer perspective it doesn't matter if the import code runs inside elastic or not - its more about importing data.

If stability was really an issue then a better solution would be to provide a standalone daemon that will be able to able to host the rivers and allow customers to import the data instead of leaving them to fend for themselves.

@carljohnstone
Copy link

Sorry I'm not hugely familiar with the code-base itself, but would it be possible to write a non-river plugin, that pulled in data from an external source than indexed it into Elasticsearch using the bulk-data APIs? Would you expect that to be any more stable than a plugin coded against the River classes? Eg is the actual problem the fact that these are plugins running within the ES server itself, rather than rivers.

@kimchy
Copy link
Member Author

kimchy commented Apr 18, 2015

@dharshanr / @carljohnstone in those cases, there really isn't any benefit in "Elasticsearch" itself having a framework for it, and its better left to be done by projects that are fully geared towards ETL such workloads, with native integration to Elasticsearch (which is where I think spending time would provide the best value). Logstash is a great example, but there are many more.

@nkavian
Copy link

nkavian commented Apr 28, 2015

I agree with @dharshanr this is punting the problem. We were enjoying using the Couchbase plugin to replicate into ES bypassing the app layer. Now what I'm understanding is the app layer now needs to copy into both places..

@dadoonet
Copy link
Member

Couchbase plugin is a transport plugin not a river plugin so not concerned by this change.

@nik9000
Copy link
Member

nik9000 commented Apr 28, 2015

@dharshanr / @carljohnstone in those cases, there really isn't any benefit in "Elasticsearch" itself having a framework for it

I figure the advantage of rivers is that you're already deploying Elasticsearch and they are already written. As much trouble as they are, they exist.

I'm still +1 on deprecating them.

@bgiromini
Copy link

The biggest hole in my opinion is getting data from a relational database into ES. I know that some work is being down on the logstash front but would be nice if it was officially supported by Elastic.

@dadoonet
Copy link
Member

@bgiromini in case you missed it, have a look at https://github.com/logstash-plugins/logstash-input-jdbc

@hash-include
Copy link

One quick question, If I am using river-mongodb and most of the company code is in Java. Can I push data into elasticsearch from Java API removing the river?

@dadoonet
Copy link
Member

@hash-include This is indeed my favorite way for doing that! Pushing from the application directly :)
I wrote a blog post about it though it's based on RDBMS and not NoSQL db: http://david.pilato.fr/blog/2015/05/09/advanced-search-for-your-legacy-application/

@s1monw
Copy link
Contributor

s1monw commented Jun 3, 2015

@dadoonet do you know what is missing here to close this?

@dadoonet
Copy link
Member

dadoonet commented Jun 3, 2015

@s1monw I think we can close it. It has been deprecated in 1.6.0 in docs and in 2.0.0 in code.
Note that we are not using the deprecation logger in master though: https://github.com/elastic/elasticsearch/blob/master/src/main/java/org/elasticsearch/river/RiversService.java#L129

Closing. If I'm missing anything, feel free to reopen.

@dadoonet dadoonet closed this as completed Jun 3, 2015
@tchcxp
Copy link

tchcxp commented Jun 22, 2015

@kimchy

I have been using mongodb river for a while. So the alternative solution is to update data in the application or other services with elastic API calls?

Thanks.

@dadoonet
Copy link
Member

@tchcxp I answered the same question a month ago: #10345 (comment)

HTH

@tchcxp
Copy link

tchcxp commented Jun 22, 2015

@dadoonet

Cool! Great help. Thanks. 👍

@syllogismos
Copy link

@tchcxp how well did updating data in elasticsearch from app layer work, instead of relying on mongodb river.

@ghost
Copy link

ghost commented May 31, 2017

@kimchy I want to replicate same set of data from mongoDB server to elasticSearch server and vice
versa. What should i use?

@dadoonet
Copy link
Member

@monikamaheshwari please ask your question on discuss.elastic.co. I'm sure many users will be able to share their experience with you. Or better, search on discuss for previous discussions regarding this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests