Support Unicode Index and Type Names

[jbaiera]

Currently, ES-Hadoop does not have total support for index or type names that are Unicode.

The ES-Hadoop REST Client is capable of interacting with Elasticsearch using Unicode names as long as they are property encoded for URL usage:

val indexName = "בְּדִיקָה"
val indexNameEncoded = java.net.URLEncoder.encode(indexName, "UTF-8")
println(indexNameEncoded) //%D7%91%D6%B0%D6%BC%D7%93%D6%B4%D7%99%D7%A7%D6%B8%D7%94

When ES-Hadoop queries for which shards to run against during the sampling phase of setting up the number of input splits/slices, it receives index/type names back from Elasticsearch that may be Unicode. The library does no further encoding of this information before using it. Without the encoding, the ES-Hadoop REST Client cannot successfully continue communicating with Elasticsearch.

I'm not sure where else this may pop up at, but it needs to be handled uniformly across the entire library. Unicode index/type names are allowed, and even if a user encodes the values before handing them to us, we will break on encoding anyway. If we add the encoding support for internal operations, we might as well fully support receiving Unicode index/types and do the encoding for the user to keep things uniform.

[rtrujill007]

@jbaiera I was just wondering if we can move the priority on this issue? We (Esri) opened a support ticket with ES a while back ago to help fix this issue.

Is there anything holding this issue back from being fixed? Thanks!