Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elasticsearch+Hadoop read and write #45

Closed
marcelopaesrech opened this Issue May 13, 2013 · 3 comments

Comments

Projects
None yet
3 participants
@marcelopaesrech
Copy link

marcelopaesrech commented May 13, 2013

Hi, I want to use Elasticsearch+Hadoop MapReduce and I want to read from elasticsearch and write to elasticsearch in the same MapReduce task. In the sample only one way is showed (or read or write because exists only one es.resource). What I want is something like follows:

Read from:
/radio/artists/_search?q=me*

Write to :
/radio/statistics

Best regards.

@tzolov

This comment has been minimized.

Copy link

tzolov commented May 13, 2013

@marcelopaesrech this looks the same issue as #26. IMO there is no technical reason prohibiting this case.

Furthermore i've implemented a simple fix that adds ES_QUERY in addition to the ES_RESOURCE and it works fine.

@marcelopaesrech

This comment has been minimized.

Copy link
Author

marcelopaesrech commented May 13, 2013

Yes, I want the input of Map is /radio/artists/_search?q=me* and the Reducers write to /radio/statistics (or maybe another index). Any intermediate data that Hadoop generates might be stored by hdfs, I don't care. But the final result I want to store in a index on ES.

I read your issue and is the same thing I think.

@costin costin removed the v1.3.0.M2 label Feb 6, 2014

@costin costin closed this in 68cd50e Mar 10, 2014

@costin

This comment has been minimized.

Copy link
Member

costin commented Mar 10, 2014

@tzolov @marcelopaesrech hey, finally got around fixing this. Cascading/Hive and Pig set the read/write automatically - in case of MapReduce jobs one can use the es.resource.read and es.resource.write properties. es.resource is still supported and used as fall-back if the aforementioned properties are not defined.

costin added a commit that referenced this issue Apr 8, 2014

Split global resource into read/write targets
Improve conf to allow for dedicated read and write resource as oppose to
a single, unified resource used for both. This allows for different ES
indices to be used in the same index, one as a source and the other as
a sink.

'es.resource' is still supported and used as a fall back.
Higher level abstractions, such as Cascading, Hive and Pig, set the
proper property automatically.

fix #156
fix #45
fix #26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.