es.field.read.as.array.include only works one level deep

[jeffsteinmetz]

I just ran into a real world situation, where there is a list of arrays, inside another array.

val sc = new SparkContext(conf)
val cfg = collection.mutable.Map("es.field.read.as.array.include" -> "nested.bar,foo,nested.bar.scores")
val json = """{"foo" : [5,6], "nested": { "bar" : [{"date":"2015-01-01", "scores":[1,2]},{"date":"2015-01-01", "scores":[3,4]}], "what": "now" } }"""
sc.makeRDD(Seq(json)).saveJsonToEs("spark/mappingtest")
val df = new SQLContext(sc).read
      .options(cfg)
      .format("org.elasticsearch.spark.sql")
      .load("spark/mappingtest")
println(df.collect().toList)

addin nested.bar.scores to es.field.read.as.array.include seems to not help ES hadoop with a hint that there is an array at this level.

throws:
EsHadoopIllegalStateException: Field 'nested.bar.scores' not found; typically this occurs with arrays which are not mapped as single value

[costin]

@jeffsteinmetz Hi,
I've pushed a fix in master which addressed this (snapshot is currently building and should be in Maven shortly); please try it out.

[jeffsteinmetz]

Great, I'll take a look.
On Jan 7, 2016 5:59 PM, "Costin Leau" notifications@github.com wrote:

@jeffsteinmetz https://github.com/jeffsteinmetz Hi,
I've pushed a fix in master which addressed this (snapshot is currently
building and should be in Maven shortly); please try it out.

—
Reply to this email directly or view it on GitHub
#589 (comment)
.

[jeffsteinmetz]

I am still getting
org.elasticsearch.hadoop.EsHadoopIllegalStateException: Field 'nested.bar.scores' not found; typically this occurs with arrays which are not mapped as single value using latest snapshot build.

[costin]

I've incorporated your example in the integration tests which passes just fine.
Can you please take a look and let me know what's the difference?

[costin]

Actually look at the test above and the difference is in the option - I've changed that form es.field.read.. to es.read.field.as.array to make it more consistent.
Changing the field strings or using ConfigurationOptions should fix the problem.

[jeffsteinmetz]

using ES_READ_FIELD_AS_ARRAY_INCLUDE fixed the issue! Thanks man.

[costin]

@jeffsteinmetz Cheers. Thanks for the feedback. And your patience. And for continuing to report issues on this project - much appreciated!