From my experience, Pig tends to be quite picky about tuples and bags when reading them directly from the source - I'm not sure why, maybe this is something that we could improve in ES but again, I'm not sure how.
Back to your gist, in ES you have defined an array (not a bag which is a collection of tuples). That's why it works reading as tuple(). Function COUNT fails since it works on bags not tuples - however you can create a bag from that tuple (for example by using TOBAG).

Regarding the initial error (the outofmemory one) I'm not sure why that occurs, but based on the stack trace it seems to be caused by Pig; I wasn't able to reproduce locally (using Pig in local mode).

My advice going forward is to try to read the data as a basic tuple and then create your structures by hand. From what I've seen this seems to be the recommendation on the Pig mailing list as well.

Hope this helps,

I tried TOBAG, but was not successful:

es_read = LOAD 'hread/doc' USING org.elasticsearch.hadoop.pig.EsStorage('') AS ( my_id: chararray, my_array:( ) );
the_gen = FOREACH es_read GENERATE my_id AS the_id, TOBAG(my_array) AS the_bag;
DUMP the_gen;

gives me the same OOM exception:

Unexpected System Error Occured: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:130)
    at org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:191)
...
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
...
Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2367)
    at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)

For some reason I can't reproduce this - maybe it's hadoop2 vs hadoop1. However I would imagine the crux of the problem is the usage of an array inside a bag.
Can you enable logging in Pig and report back? I'm still unsure why an array (created by es-hadoop or otherwise) would cause this issue in Pig - especially when only 2 items are contained within.

As an alternative you could load the array as individual items and then manually create the bag through TOBAG.

Rescheduling this for 1.3 RC1 potentially by adding more documentation on this type of mapping.

Hi,

This should be fixed in master - though I was not able to reproduce your issue, I bumped into one that had similar behaviour (and it turned out it depended on the JSON being read). It would be great if you could try the latest dev builds and let us know whether they work for you or not.

Cheers,