Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HashMap[String,String] and elastic search type mapping is not kicking in to map String to Integer #372

Closed
tzhang101 opened this issue Feb 4, 2015 · 3 comments

Comments

@tzhang101
Copy link

Hi, I am generating HashMap[String,String] and then send to Elastic Search with this line of code. rddAsMap.saveAsHadoopDataset(jobConf)
I also have type mapping defined on elastic search for my index and type properly.
But somehow elastic search kept all fields as string. Why is the mapping not kicking in?

Thanks.

Tian

@costin
Copy link
Member

costin commented Feb 5, 2015

I don't follow. Can you maybe post some snippets of code of what happens vs what your expectations are? You can verify the mapping of a document directly in Elasticsearch through its REST API. Most likely the mapping doesn't apply since maps are often mapped dynamically - you should be able to check that at ES level.

@tzhang101
Copy link
Author

Costin, sorry my fault, it turns out that the mapping indeed worked, it is just I am looking at the _source data which is indeed string. But I have another question about your package. Currently we generate HashMap[String,String] in order to call saveAsHadoopDataset. Is it possible to generate more generic HashMap[String, Object] and allow me to pass proper type of data directly to Elastic Search w/o ES to do the mapping? Thanks.

@costin
Copy link
Member

costin commented Feb 6, 2015

@tzhang101 I'm not sure what sure what you mean - you can send whatever object (whether it's a map or not) to es-hadoop. You haven't posted any code (typically it makes a big difference) so I can only infer that you are configuring OutputFormat to accept a Map[String, String] - simply configure it differently. Plus you don't have to use saveAsHadoopDataset()- simply use the native es-spark Java/Spark API.

Both approaches are explained in the docs so please review them.

In the future, for questions, please use the mailing list or the IRC (as mentioned in the resources) - the issue tracker is not the appropriate place. Thank you.

@costin costin closed this as completed Feb 6, 2015
@costin costin added the invalid label Feb 6, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants