Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Security Lab #8

Open
doddys opened this issue May 16, 2018 · 14 comments
Open

Security Lab #8

doddys opened this issue May 16, 2018 · 14 comments
Assignees
Labels
review This doesn't seem right
Milestone

Comments

@doddys
Copy link
Owner

doddys commented May 16, 2018

No description provided.

@doddys doddys self-assigned this May 16, 2018
@doddys doddys added this to the Labs milestone May 16, 2018
@doddys doddys added the started This issue or pull request already exists label May 16, 2018
@doddys
Copy link
Owner Author

doddys commented May 16, 2018

Setting up lab requirement

  • hue sample data

@doddys
Copy link
Owner Author

doddys commented May 16, 2018

Having problem with hue

Current value: http://ec2-13-250-20-43.ap-southeast-1.compute.amazonaws.com:50070/webhdfs/v1
Failed to access filesystem root

@doddys
Copy link
Owner Author

doddys commented May 16, 2018

Apparently hue was pointing to passive namenode. After changing to active namenode and restart it works

@doddys
Copy link
Owner Author

doddys commented May 16, 2018

having problem activating kerberos

  • due to firewall setting
    Need to open port
    TCP : 464, 88 , 749
    UDP :646, 88

  • JCE is not put in the correct folder JAVA_HOME/jre/lib/security

@doddys
Copy link
Owner Author

doddys commented May 16, 2018

Fixed the kerberos isu by updating executing the following command

modprinc -maxlife 1days -maxrenewlife 7days +allow_renewable krbtgt/EXPECC.COM@EXPECC.COM

but now got a different problem:

18/05/16 12:20:32 WARN security.UserGroupInformation: PriviledgedActionException as:hdfs (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
18/05/16 12:20:32 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
18/05/16 12:20:32 WARN security.UserGroupInformation: PriviledgedActionException as:hdfs (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "hadoop5.expecc.com/172.30.1.187"; destination host is: "hadoop5.expecc.com":8020; 

@doddys doddys added the question Further information is requested label May 16, 2018
@doddys
Copy link
Owner Author

doddys commented May 16, 2018

Apparently, this is an expected behaviour. In order to execute command as hdfs user, we need to use hdfs.keytab to acquire kerberos ticket

@doddys
Copy link
Owner Author

doddys commented May 16, 2018

Another Problem: Canary Service is not working.

This problem is resolve by opening port 1004.

@doddys
Copy link
Owner Author

doddys commented May 16, 2018

Next Problem : Try to run teragen
Error:

18/05/16 16:05:08 WARN security.UserGroupInformation: PriviledgedActionException as:doddys@EXPECC.COM (auth:KERBEROS) cause:java.io.IOException: Can't get Master Kerberos principal for use as renewer
java.io.IOException: Can't get Master Kerberos principal for use as renewer
	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:133)
	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)

@doddys
Copy link
Owner Author

doddys commented May 16, 2018

add gateway role to the node where I execute the error resolve "renewer" issue but

iagnostics: Application application_1526485969852_0001 initialization failed (exitCode=255) with output: main : command provided 0
main : run as user is doddys
main : requested yarn user is doddys
Can't create directory /yarn/nm/usercache/doddys/appcache/application_1526485969852_0001 - Permission denied
Did not create any app directories

Failing this attempt. Failing the application.
18/05/16 16:18:33 INFO mapreduce.Job: Counters: 0



@doddys
Copy link
Owner Author

doddys commented May 16, 2018

fixed yarn isu:

rm /yarn/nm/usercache/*

on all data nodes

@doddys doddys added review This doesn't seem right and removed started This issue or pull request already exists labels May 16, 2018
@doddys
Copy link
Owner Author

doddys commented May 16, 2018

final result submitted

@doddys
Copy link
Owner Author

doddys commented May 17, 2018

I missed the sentry lab.

Starting Sentry Lab now

@doddys
Copy link
Owner Author

doddys commented May 17, 2018

Sentry Lab is completed.

Please Review

@doddys doddys removed the question Further information is requested label May 17, 2018
@endlesscyl
Copy link

add gateway role to the node

hi,doddys:
I also have same error "Can't get Master Kerberos principal for use as renewer",how did you solve it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
review This doesn't seem right
Projects
None yet
Development

No branches or pull requests

3 participants