[ZEPPELIN-1149] %sh interpreter kerberos support #1163

rconline · 2016-07-11T14:41:36Z

What is this PR for?

Zeppelin Shell interpreter should work in a Kerberos environment

What type of PR is it?

[Feature]

What is the Jira issue?

ZEPPELIN-1149

How should this be tested?

In JDBC interpreter setting add following properties

shell.auth.type = KERBEROS
shell.principal = principal value
shell.keytab.location = keytab location

Now try and run any shell command (example: hdfs dfs -ls) it should return with valid results.

Screenshots (if appropriate)

Questions:

Does the licenses files need update? no
Is there breaking changes for older versions? no
Does this needs documentation? yes

prabhjyotsingh · 2016-07-11T17:09:16Z

LGTM, tested on ubuntu 14.04 server.

felixcheung · 2016-07-11T22:53:02Z

shell/src/main/java/org/apache/zeppelin/shell/security/ShellSecurityImpl.java

+        "auth.type to SIMPLE", properties.getProperty("shell.auth.type").trim()));
+      authType = SIMPLE;
+    }
+


extra newline

felixcheung · 2016-07-11T22:55:51Z

If each interpreter has a separate settings for Kerberos, eg. shell.keytab.location, would it be harder to manage?

…the project.

rconline · 2016-07-12T06:39:11Z

@felixcheung over a period of time, each interpreter will start to have their own Keytab's. I think we should have separate keytabs for different interpreters. We will ofcourse have to invest on getting a better management console for interpreters themselves.

jongyoul · 2016-07-12T09:33:29Z

@rconline What is kinit? I've tried to find a kinit in my CentOS, but I couldn't find it. Is it fine?

rconline · 2016-07-12T10:09:17Z

@jongyoul here are some reference - http://linux.die.net/man/1/kinit & http://www.tutorialspoint.com/unix_commands/kinit.htm

…erberos/Simple.

jongyoul · 2016-07-12T14:12:02Z

@rconline In case of not having kinit, there isn't a way to use kerberos, right?

rconline · 2016-07-12T17:00:57Z

@jongyoul no there isn't. Here's the snippet from UserGroupInformation, which is essentially from Hadoop security. In future we will implement multiple methods of security. At present for Kerberos security Kinit is a necessary step. I intend to generalize this across other interpreters as well subsequently.

public static enum AuthenticationMethod { SIMPLE, KERBEROS, TOKEN, CERTIFICATE, KERBEROS_SSL, PROXY;

# Conflicts: # docs/interpreter/shell.md

felixcheung · 2016-07-12T19:17:54Z

shell/src/main/java/org/apache/zeppelin/shell/security/ShellSecurityImpl.java

+
+  private static Logger LOGGER = LoggerFactory.getLogger(ShellSecurityImpl.class);
+
+  public static void createSecureCinfiguration(Properties properties, String shell) {


do you mean for this to be createSecureConfiguration?

rconline · 2016-07-14T06:21:58Z

@felixcheung as of now the interpreters are reusing zeppelin keytab. Typically this is case in the hadoop security world, where the keytab is generated at a process level, whereas the authorization is done through impersonation. This is then mapped to the user permissions setup on the underlying data system via posix users. Mature systems which implement full support for security + impersonation, can provide this ability. Right now with this change we get to the first stage which is security. To get to impersonation we will have to start the shell with the user's identity. I think that can be followup patch, in the next couple of weeks.

prabhjyotsingh · 2016-07-21T10:20:56Z

Merging this if no more discussion.

rconline · 2016-07-21T11:28:37Z

Re-trigger CI

zjffdu · 2016-07-22T01:51:10Z

agree with @felixcheung, use the zeppelin service keytab/principal might bring some other potential security problems. I think user A would expect to run shell script as himself, but actually it run as user zeppelin. And seems all the users are sharing the same principal if I understand correctly. Anyway, I don't have a better idea now, but seems zeppelin in secured cluster needs a lot work to do.

Besides that I have one question about shiro of zeppelin, does shiro will map the username to hadoop side ? Say I login zeppelin as user A, would that guaranteed there should also be a user A in the hadoop ecosystem?

zjffdu · 2016-07-22T02:37:49Z

Sorry, I mistake it as hdfs interpreter. For shell interpreter, do we must have kerberos support ? I think here kinit is just for the hadoop command, so actually it is for running hdfs command in secured cluster rather than running general shell command in secured cluster.

prabhjyotsingh · 2016-07-22T07:45:39Z

This will also, allow CLI/shell interaction to other data systems such hbase, falcon, hive, kafka, oozie, sqoop, etc. on secured clusters. Hence this is a need, particularly during data ingestion via a notebook.

zjffdu · 2016-07-22T08:38:06Z

@prabhjyotsingh Thanks for clarification.

prabhjyotsingh · 2016-07-23T04:33:22Z

CI fail is unrelated, which fails for Build #6048.2 with No output has been received in the last 10 minutes, this potentially indicates a stalled build or something wrong with the build itself.

Will merge this if no more discussion.

felixcheung · 2016-07-23T15:45:11Z

I think getting impersonation is an important next step that could come after.

### What is this PR for? Zeppelin Shell interpreter should work in a Kerberos environment ### What type of PR is it? [Feature] ### What is the Jira issue? * [ZEPPELIN-1149](https://issues.apache.org/jira/browse/ZEPPELIN-1149) ### How should this be tested? In JDBC interpreter setting add following properties - shell.auth.type = KERBEROS - shell.principal = principal value - shell.keytab.location = keytab location Now try and run any shell command (example: hdfs dfs -ls) it should return with valid results. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? yes Author: Rohit Choudhary <rconline@gmail.com> Closes apache#1163 from rconline/ZEPPELIN-1149 and squashes the following commits: f6af0f6 [Rohit Choudhary] typo fix 3648a1c [Rohit Choudhary] fix CI, resolve merge conflict 1924b99 [Rohit Choudhary] Merge remote-tracking branch 'origin/master' into ZEPPELIN-1149 9143b47 [Rohit Choudhary] Removed Hadoop common dependency and changed switch to String based Kerberos/Simple. 60e6d21 [Rohit Choudhary] Addressing feedback from felixcheung b0300e8 [Rohit Choudhary] changing property name to "zeppelin.*" to make it match with rest of the project. 0a77e80 [Rohit Choudhary] shell interpreter doc 5190791 [Rohit Choudhary] [ZEPPELIN-1149] %sh interpreter kerberos support

rconline added 2 commits July 11, 2016 20:05

[ZEPPELIN-1149] %sh interpreter kerberos support

5190791

shell interpreter doc

0a77e80

rconline force-pushed the ZEPPELIN-1149 branch from bc71051 to 0a77e80 Compare July 11, 2016 15:11

felixcheung reviewed Jul 11, 2016
View reviewed changes

rconline force-pushed the ZEPPELIN-1149 branch 2 times, most recently from 5bb5654 to 0a77e80 Compare July 12, 2016 05:57

rconline added 2 commits July 12, 2016 11:33

changing property name to "zeppelin.*" to make it match with rest of …

b0300e8

…the project.

Addressing feedback from felixcheung

60e6d21

Removed Hadoop common dependency and changed switch to String based K…

9143b47

…erberos/Simple.

rconline added 2 commits July 12, 2016 23:27

Merge remote-tracking branch 'origin/master' into ZEPPELIN-1149

1924b99

# Conflicts: # docs/interpreter/shell.md

fix CI, resolve merge conflict

3648a1c

felixcheung reviewed Jul 12, 2016
View reviewed changes

typo fix

f6af0f6

rconline closed this Jul 21, 2016

rconline reopened this Jul 21, 2016

rconline closed this Jul 21, 2016

rconline reopened this Jul 21, 2016

asfgit closed this in b86148c Jul 25, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ZEPPELIN-1149] %sh interpreter kerberos support #1163

[ZEPPELIN-1149] %sh interpreter kerberos support #1163

rconline commented Jul 11, 2016

prabhjyotsingh commented Jul 11, 2016

felixcheung Jul 11, 2016

felixcheung commented Jul 11, 2016

rconline commented Jul 12, 2016

jongyoul commented Jul 12, 2016

rconline commented Jul 12, 2016

jongyoul commented Jul 12, 2016

rconline commented Jul 12, 2016

felixcheung Jul 12, 2016

rconline commented Jul 14, 2016

prabhjyotsingh commented Jul 21, 2016

rconline commented Jul 21, 2016

zjffdu commented Jul 22, 2016 •

edited

zjffdu commented Jul 22, 2016

prabhjyotsingh commented Jul 22, 2016

zjffdu commented Jul 22, 2016

prabhjyotsingh commented Jul 23, 2016

felixcheung commented Jul 23, 2016


		private static Logger LOGGER = LoggerFactory.getLogger(ShellSecurityImpl.class);

		public static void createSecureCinfiguration(Properties properties, String shell) {

[ZEPPELIN-1149] %sh interpreter kerberos support #1163

[ZEPPELIN-1149] %sh interpreter kerberos support #1163

Conversation

rconline commented Jul 11, 2016

What is this PR for?

What type of PR is it?

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

prabhjyotsingh commented Jul 11, 2016

felixcheung Jul 11, 2016

Choose a reason for hiding this comment

felixcheung commented Jul 11, 2016

rconline commented Jul 12, 2016

jongyoul commented Jul 12, 2016

rconline commented Jul 12, 2016

jongyoul commented Jul 12, 2016

rconline commented Jul 12, 2016

felixcheung Jul 12, 2016

Choose a reason for hiding this comment

rconline commented Jul 14, 2016

prabhjyotsingh commented Jul 21, 2016

rconline commented Jul 21, 2016

zjffdu commented Jul 22, 2016 • edited

zjffdu commented Jul 22, 2016

prabhjyotsingh commented Jul 22, 2016

zjffdu commented Jul 22, 2016

prabhjyotsingh commented Jul 23, 2016

felixcheung commented Jul 23, 2016

zjffdu commented Jul 22, 2016 •

edited