Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Invalid jackson fallback on MapR distro #215

Closed
costin opened this issue Jun 15, 2014 · 11 comments
Closed

Invalid jackson fallback on MapR distro #215

costin opened this issue Jun 15, 2014 · 11 comments
Labels
bug v2.0.1 v2.1.0.Beta1

Comments

@costin
Copy link
Member

costin commented Jun 15, 2014

Pre 1.7 jackson versions are not properly detected resulting in an invalid code path being executed. This typically occurs on MapR distros.
costin added a commit that referenced this issue Jun 15, 2014
@costin
Improve detection of old Jackson versions
a9973cf 
Fix #215
@beljun
Copy link

beljun commented Jun 17, 2014

Hi. I grabbed the nightly build elasticsearch-hadoop-2.1.0.BUILD-20140617.012002-21.jar and tried to write to ES from Pig. Still getting a NoSuchMethodError when referencing the Jackson lib (MapR distro, Jackson 1.5.2). Might it be related to this same issue even after the fix?

2014-06-17 19:08:08,491 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.NoSuchMethodError: org.codehaus.jackson.map.ObjectMapper.reader(Ljava/lang/Class;)Lorg/codehaus/jackson/map/ObjectReader;
    at org.elasticsearch.hadoop.rest.RestClient.retryFailedEntries(RestClient.java:163)
    at org.elasticsearch.hadoop.rest.RestClient.bulk(RestClient.java:156)
    at org.elasticsearch.hadoop.rest.RestRepository.sendBatch(RestRepository.java:170)
    at org.elasticsearch.hadoop.rest.RestRepository.close(RestRepository.java:189)
    at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.doClose(EsOutputFormat.java:294)
    at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.close(EsOutputFormat.java:276)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.close(PigOutputFormat.java:149)
    at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:561)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:670)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1126)
    at org.apache.hadoop.mapred.Child.main(Child.java:264)

@costin
Copy link
Member Author

costin commented Jun 17, 2014

Try the 2.0.1.BUILD not the 2.1.0. Specifically https://oss.sonatype.org/content/repositories/snapshots/org/elasticsearch/elasticsearch-hadoop/2.0.1.BUILD-SNAPSHOT/

Can you also confirm the MapR version used?

Thanks,

@charlesflynn
Copy link

Hey there,

I'm also experiencing this problem. The MapR version is 2.1.3.20987.GA. I just downloaded elasticsearch-hadoop-2.0.1.BUILD-20140617.021905-22.zip and reran my Hive and Pig jobs using elasticsearch-hadoop-2.0.1.BUILD-SNAPSHOT.jar. Both returned:

2014-06-17 14:05:20,506 FATAL ExecMapper: java.lang.NoSuchMethodError: org.codehaus.jackson.map.ObjectMapper.reader(Ljava/lang/Class;)Lorg/codehaus/jackson/map/ObjectReader;

@charlesflynn
Copy link

If it helps, here are the jar versions:

$HADOOP_HOME/lib:
jackson-core-asl-1.5.2.jar
jackson-mapper-asl-1.5.2.jar

$HIVE_HOME/lib:
jackson-core-asl-1.8.8.jar
jackson-jaxrs-1.8.8.jar
jackson-mapper-asl-1.8.8.jar
jackson-xc-1.8.8.jar

$PIG_HOME/lib:
jackson-core-asl-1.8.8.jar
jackson-jaxrs-1.8.8.jar
jackson-mapper-asl-1.8.8.jar
jackson-xc-1.8.8.jar

@costin
Copy link
Member Author

costin commented Jun 17, 2014

@charlesflynn It does. Unfortunately the classpath is unreliable so on some systems the 1.8.8 version is picked up, on others 1.5.2. In your case the old version is picked up which results in the error above.
I'm working on a fix but it might take a bit longer - in the meantime you could try replacing the 1.5.2 versions with the 1.8.8 one.

@beljun
Copy link

beljun commented Jun 18, 2014

@costin, I'm on 2.0.1.15869.GA but everything else same as what @charlesflynn pasted above. Ok, I can afford to wait some for the fix. Looking forward to it. Thanks!

costin added a commit that referenced this issue Jun 18, 2014
@costin
Improve support of old/ancient Jackson version (1.5)
c621b99 
Backport ObjectReader from Jackson to 1.5 for easy, non-initializing
parsing of the json doc

fix #215
@costin
Copy link
Member Author

costin commented Jun 18, 2014

@beljun @charlesflynn guys, I've pushed a fix (and a nightly build) on the 2.0.x branch to address this issue. It works on my side and it be great if you can confirm this on your end as well.

Thanks!

@charlesflynn
Copy link

Awesome, thanks! Worked from both Hive and Pig, using elasticsearch-hadoop-2.0.1.BUILD-20140618.210714-25.jar

@beljun
Copy link

beljun commented Jun 19, 2014

Works on my end, too. Thanks a lot, @costin !

@costin costin closed this as completed in 2c0e224 Jul 31, 2014
costin added a commit that referenced this issue Jul 31, 2014
@costin
Improve support of old/ancient Jackson version (1.5)
b919c17 
Backport ObjectReader from Jackson to 1.5 for easy, non-initializing
parsing of the json doc

fix #215

(cherry picked from commit c621b99)
@sibbsnb
Copy link

sibbsnb commented Apr 17, 2015

Is this issue taken care in latest release?

@costin
Copy link
Member Author

costin commented Apr 17, 2015

@sibbsnb This issue has been closed since Jul 2014. There are also two labels associated with it on the right. This means it has been addressed starting with these versions. If experience problems, considering raising a different issue.

@sibbsnb sibbsnb mentioned this issue Apr 17, 2015
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug v2.0.1 v2.1.0.Beta1
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@costin @beljun @charlesflynn @sibbsnb