-
Notifications
You must be signed in to change notification settings - Fork 986
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Elasticsearch+Hadoop read and write #45
Comments
@marcelopaesrech this looks the same issue as #26. IMO there is no technical reason prohibiting this case. Furthermore i've implemented a simple fix that adds ES_QUERY in addition to the ES_RESOURCE and it works fine. |
Yes, I want the input of Map is /radio/artists/_search?q=me* and the Reducers write to /radio/statistics (or maybe another index). Any intermediate data that Hadoop generates might be stored by hdfs, I don't care. But the final result I want to store in a index on ES. I read your issue and is the same thing I think. |
@tzolov @marcelopaesrech hey, finally got around fixing this. Cascading/Hive and Pig set the read/write automatically - in case of MapReduce jobs one can use the |
Improve conf to allow for dedicated read and write resource as oppose to a single, unified resource used for both. This allows for different ES indices to be used in the same index, one as a source and the other as a sink. 'es.resource' is still supported and used as a fall back. Higher level abstractions, such as Cascading, Hive and Pig, set the proper property automatically. fix #156 fix #45 fix #26
Hi, I want to use Elasticsearch+Hadoop MapReduce and I want to read from elasticsearch and write to elasticsearch in the same MapReduce task. In the sample only one way is showed (or read or write because exists only one es.resource). What I want is something like follows:
Read from:
/radio/artists/_search?q=me*
Write to :
/radio/statistics
Best regards.
The text was updated successfully, but these errors were encountered: