Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pig storage constructors only take strings #34

Closed
bwmeier opened this issue Apr 15, 2013 · 4 comments
Closed

Pig storage constructors only take strings #34

bwmeier opened this issue Apr 15, 2013 · 4 comments
Labels
v1.3.0.M1

Comments

@bwmeier
Copy link

bwmeier commented Apr 15, 2013

ESStorage has a constructor ESStorage(String, int) that is intended to set the host and port for the storage. Pig requires that the constructor only take strings. I changed my local code as follows:

--- a/src/main/java/org/elasticsearch/hadoop/pig/ESStorage.java
+++ b/src/main/java/org/elasticsearch/hadoop/pig/ESStorage.java
@@ -83,12 +83,12 @@ public class ESStorage extends LoadFunc implements StoreFuncInterface, StoreMeta
     private RecordWriter<Object, Object> writer;

     public ESStorage() {
-        this(null, 0);
+        this(null, "0");
     }

-    public ESStorage(String host, int port) {
+    public ESStorage(String host, String port) {
         this.host = host;
-        this.port = port;
+        this.port = Integer.parseInt(port);
     }

     @Override
@bwmeier
Copy link
Author

bwmeier commented Apr 15, 2013

Note, the same change needs to be made to ESLoader :-)

@bwmeier bwmeier closed this as completed Apr 15, 2013
@bwmeier bwmeier reopened this Apr 15, 2013
@costin
Copy link
Member

costin commented Apr 18, 2013

Where does this 'requirement' come from? We test again Pig (0.10.1) in local mode in our suite and it runs fine. Can you tell me how to reproduce the problem?

Thanks!

@bwmeier
Copy link
Author

bwmeier commented Apr 18, 2013

The root of the problem is when the elasticsearch server does not reside on localhost, so that you have to define the server and the port to connect to using the custom constructor.

This pig code generates a pig syntax error when run because of the 9200 port number being numeric.

REGISTER piggybank.jar;
REGISTER elasticsearch-hadoop-1.0.0.BUILD-SNAPSHOT.jar;

DEFINE ESStorage org.elasticsearch.hadoop.pig.ESStorage('es-server',9200);

A = LOAD '_all/_search?q=START' USING ESStorage() AS (id:chararray, data);
DUMP A;

If I change the 9200 to a string ('9200'), then the syntax error goes away, but the constructor cannot be resolved (because there is no (String, String) constructor. In order to make it work, I patched the ESStorage custom constructor to change it to take a string instead of an integer for the port number.

@costin costin closed this as completed in a1a7952 Apr 19, 2013
@costin
Copy link
Member

costin commented Apr 19, 2013

Found the problem - the tests were using the default constructor which called the non-string constructor internally, outside Pig.
Fixed this in master - thanks for taking the time to reply and comment on the issue!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
v1.3.0.M1
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@costin @bwmeier