Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Invalid jackson fallback on MapR distro #215

Closed
costin opened this Issue Jun 15, 2014 · 11 comments

Comments

Projects
None yet
4 participants
@costin
Copy link
Member

costin commented Jun 15, 2014

Pre 1.7 jackson versions are not properly detected resulting in an invalid code path being executed. This typically occurs on MapR distros.

@costin costin added bug labels Jun 15, 2014

costin added a commit that referenced this issue Jun 15, 2014

@beljun

This comment has been minimized.

Copy link

beljun commented Jun 17, 2014

Hi. I grabbed the nightly build elasticsearch-hadoop-2.1.0.BUILD-20140617.012002-21.jar and tried to write to ES from Pig. Still getting a NoSuchMethodError when referencing the Jackson lib (MapR distro, Jackson 1.5.2). Might it be related to this same issue even after the fix?

2014-06-17 19:08:08,491 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.NoSuchMethodError: org.codehaus.jackson.map.ObjectMapper.reader(Ljava/lang/Class;)Lorg/codehaus/jackson/map/ObjectReader;
    at org.elasticsearch.hadoop.rest.RestClient.retryFailedEntries(RestClient.java:163)
    at org.elasticsearch.hadoop.rest.RestClient.bulk(RestClient.java:156)
    at org.elasticsearch.hadoop.rest.RestRepository.sendBatch(RestRepository.java:170)
    at org.elasticsearch.hadoop.rest.RestRepository.close(RestRepository.java:189)
    at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.doClose(EsOutputFormat.java:294)
    at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.close(EsOutputFormat.java:276)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.close(PigOutputFormat.java:149)
    at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:561)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:670)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1126)
    at org.apache.hadoop.mapred.Child.main(Child.java:264)
@costin

This comment has been minimized.

Copy link
Member Author

costin commented Jun 17, 2014

Try the 2.0.1.BUILD not the 2.1.0. Specifically https://oss.sonatype.org/content/repositories/snapshots/org/elasticsearch/elasticsearch-hadoop/2.0.1.BUILD-SNAPSHOT/

Can you also confirm the MapR version used?

Thanks,

@charlesflynn

This comment has been minimized.

Copy link

charlesflynn commented Jun 17, 2014

Hey there,

I'm also experiencing this problem. The MapR version is 2.1.3.20987.GA. I just downloaded elasticsearch-hadoop-2.0.1.BUILD-20140617.021905-22.zip and reran my Hive and Pig jobs using elasticsearch-hadoop-2.0.1.BUILD-SNAPSHOT.jar. Both returned:

2014-06-17 14:05:20,506 FATAL ExecMapper: java.lang.NoSuchMethodError: org.codehaus.jackson.map.ObjectMapper.reader(Ljava/lang/Class;)Lorg/codehaus/jackson/map/ObjectReader;
@charlesflynn

This comment has been minimized.

Copy link

charlesflynn commented Jun 17, 2014

If it helps, here are the jar versions:

$HADOOP_HOME/lib:
jackson-core-asl-1.5.2.jar
jackson-mapper-asl-1.5.2.jar

$HIVE_HOME/lib:
jackson-core-asl-1.8.8.jar
jackson-jaxrs-1.8.8.jar
jackson-mapper-asl-1.8.8.jar
jackson-xc-1.8.8.jar

$PIG_HOME/lib:
jackson-core-asl-1.8.8.jar
jackson-jaxrs-1.8.8.jar
jackson-mapper-asl-1.8.8.jar
jackson-xc-1.8.8.jar
@costin

This comment has been minimized.

Copy link
Member Author

costin commented Jun 17, 2014

@charlesflynn It does. Unfortunately the classpath is unreliable so on some systems the 1.8.8 version is picked up, on others 1.5.2. In your case the old version is picked up which results in the error above.
I'm working on a fix but it might take a bit longer - in the meantime you could try replacing the 1.5.2 versions with the 1.8.8 one.

@beljun

This comment has been minimized.

Copy link

beljun commented Jun 18, 2014

@costin, I'm on 2.0.1.15869.GA but everything else same as what @charlesflynn pasted above. Ok, I can afford to wait some for the fix. Looking forward to it. Thanks!

costin added a commit that referenced this issue Jun 18, 2014

Improve support of old/ancient Jackson version (1.5)
Backport ObjectReader from Jackson to 1.5 for easy, non-initializing
parsing of the json doc

fix #215
@costin

This comment has been minimized.

Copy link
Member Author

costin commented Jun 18, 2014

@beljun @charlesflynn guys, I've pushed a fix (and a nightly build) on the 2.0.x branch to address this issue. It works on my side and it be great if you can confirm this on your end as well.

Thanks!

@charlesflynn

This comment has been minimized.

Copy link

charlesflynn commented Jun 19, 2014

Awesome, thanks! Worked from both Hive and Pig, using elasticsearch-hadoop-2.0.1.BUILD-20140618.210714-25.jar

@beljun

This comment has been minimized.

Copy link

beljun commented Jun 19, 2014

Works on my end, too. Thanks a lot, @costin !

@costin costin closed this in 2c0e224 Jul 31, 2014

costin added a commit that referenced this issue Jul 31, 2014

Improve support of old/ancient Jackson version (1.5)
Backport ObjectReader from Jackson to 1.5 for easy, non-initializing
parsing of the json doc

fix #215

(cherry picked from commit c621b99)
@sibbsnb

This comment has been minimized.

Copy link

sibbsnb commented Apr 17, 2015

Is this issue taken care in latest release?

@costin

This comment has been minimized.

Copy link
Member Author

costin commented Apr 17, 2015

@sibbsnb This issue has been closed since Jul 2014. There are also two labels associated with it on the right. This means it has been addressed starting with these versions. If experience problems, considering raising a different issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.