Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proxy user authentication causes NPE #195

Closed
costin opened this Issue Apr 23, 2014 · 10 comments

Comments

Projects
None yet
3 participants
@costin
Copy link
Member

commented Apr 23, 2014

Using user authentication causes a NPE in CommonsHttpTransport. Disabling authentication works just fine.

@baskaranauc

This comment has been minimized.

Copy link

commented Apr 23, 2014

Hi Costin,

Thanks indeed.

Have you fixed NPE only ? With latest dev build I am getting unauthorized[401] error even proxy username and password provided.

Did I understand anything wrong ? Please find the script and stacktrace.

Script:

DEFINE EsStorage org.elasticsearch.hadoop.pig.EsStorage('es.nodes=IP-x:port-y'
,'es.net.proxy.http.host=IP-x'
,'es.net.proxy.http.port=port-y'
,'es.net.proxy.http.user=username'
,'es.net.proxy.http.password=password'
);

Stacktrace:

org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: [GET] on [_nodes/transport] failed; server[IP-x:port-y] returned [401|Unauthorized:]
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:300)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:266)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:270)
at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:108)
at org.elasticsearch.hadoop.rest.RestClient.discoverNodes(RestClient.java:90)
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverNodesIfNeeded(InitializationUtils.java:61)
at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.init(EsOutputFormat.java:181)
at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.write(EsOutputFormat.java:158)
at org.elasticsearch.hadoop.pig.EsStorage.putNext(EsStorage.java:191)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98)
at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:638)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:284)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:277)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

Thanks.

@costin

This comment has been minimized.

Copy link
Member Author

commented Apr 23, 2014

The fix addressed the NPE mainly not the authorization itself. Can you confirm whether it works without it and double check the user/pass?

Thank,

@costin

This comment has been minimized.

Copy link
Member Author

commented Apr 23, 2014

By the way, you should use:
es.net.proxy.http.pass NOT es.net.proxy.http.password.

costin added a commit that referenced this issue Apr 23, 2014

@baskaranauc

This comment has been minimized.

Copy link

commented Apr 25, 2014

Hi Costin,

Yes. The fix with dev build is working with proxy without user authentication.

I hope user authentication also will be fixed in RC1.

Thanks.

@costin

This comment has been minimized.

Copy link
Member Author

commented Apr 25, 2014

I'm not clear what is the current status - is authentication running or not for you with the latest build? What's your configuration again?

@baskaranauc

This comment has been minimized.

Copy link

commented Apr 25, 2014

It does not work with below configuration.

nginx config snippet:

server {

listen       8080;
server_name  localhost;

error_log   elasticsearch-errors.log;
access_log  elasticsearch.log;

location / {

  # Deny access to Cluster API
  if ($request_filename ~ "_cluster") {
    return 403;
    break;
  }

  # Pass requests to ElasticSearch
  proxy_pass http://localhost:9200;
  proxy_redirect off;

  #proxy_set_header  X-Real-IP  $remote_addr;
  #proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
  #proxy_set_header  Host $http_host;

  # Authorize access
  auth_basic           "Restricted";
  auth_basic_user_file  /var/www/a.htpasswd;

Pig script:

DEFINE EsStorage org.elasticsearch.hadoop.pig.EsStorage(
'es.nodes=localhost:8080'
,'es.net.proxy.http.host=localhost'
,'es.net.proxy.http.port=8080'
,'es.net.proxy.http.user={username}'
,'es.net.proxy.http.pass={password}');

Stacktrace:

org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: [GET] on [_nodes/transport] failed; server[localhost:8080] returned [401|Unauthorized:]
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:300)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:266)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:270)
at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:108)
at org.elasticsearch.hadoop.rest.RestClient.discoverNodes(RestClient.java:90)
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverNodesIfNeeded(InitializationUtils.java:61)
at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.init(EsOutputFormat.java:181)
at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.write(EsOutputFormat.java:158)
at org.elasticsearch.hadoop.pig.EsStorage.putNext(EsStorage.java:191)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98)
at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:638)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:284)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:277)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

@costin

This comment has been minimized.

Copy link
Member Author

commented Apr 25, 2014

I've pushed another dev build which works for me - can you please try it out?

@baskaranauc

This comment has been minimized.

Copy link

commented Apr 25, 2014

Hi Costin,

It works!!!! with latest dev build (elasticsearch-hadoop-1.3.0.BUILD-20140425.102702-404)

Thanks a lot.

costin added a commit that referenced this issue Apr 25, 2014

costin added a commit that referenced this issue Apr 25, 2014

@bass14

This comment has been minimized.

Copy link

commented May 9, 2014

Hi Constin
I used BUILD jar file.It does not support the Object Data datatype.Can check this problem?

@costin

This comment has been minimized.

Copy link
Member Author

commented May 9, 2014

@bass14 Please open a new issue, this one is closed. And please try to be more descriptive since I'm afraid I don't understand what the problem is. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.