org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Field [_col1] needs to be a primitive; found [array<map<string,string>>]

[cynosureabu]

I defined the external table as:
CREATE EXTERNAL TABLE IF NOT EXISTS ${staging_table}(
customer_id STRING,
store_purchase array<map<string,string>>)
ROW FORMAT SERDE 'org.elasticsearch.hadoop.hive.EsSerDe'
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES (
'es.nodes'='localhost:9200',
'es.resource'='user_activity/store',
'es.mapping.id'='customer_id',
'es.input.json'='false',
'es.write.operation'='upsert',
'es.update.script'='ctx._source.store_purchase += purchase',
'es.update.script.params'='purchase:store_purchase'
) ;"

but it throws EsHadoopIllegalArgumentException: Field [_col1] needs to be a primitive; found [array<map<string,string>>]. Is array<map<string,string>> supported yet? If not, how can I get around this issue?

[jbaiera]

The issue here that is tripping up the table creation is that the array<map<string,string>> column is being used as the script parameter. For the time being, we only support using primitives in the script params for Hive.

[matthenricks]

HI jbaiera, is there a recommended way to circumvent this issue? We have an update batch job where we try to update a variety of fields - based on a condition being passed. A couple of these fields are nested objects.

Thanks, Matt Henricks

[jbaiera]

@matthenricks sadly there is no work around right now. The logic that performs serializing records from different integrations into JSON is different between rendering script parameters and rendering the final JSON document. We'll update here as new updates become available.