[ZEPPELIN-828] Use individual user credentials for data source authentication #860

prasadwagle · 2016-04-27T21:18:32Z

What is this PR for?

The purpose of this pull request is to allow users to use their own credentials to authenticate with data sources.
It contains the following:

web UI and rest API to let users input their credentials
changes to AuthenticationInfo to add data source credentials
changes to Paragraph to set data source credentials in AuthenticationInfo
changes to HiveInterpreter to use the user credentials and simplification to use a new connection for every query. We made this change since we found an issue with closed vertica connections. Since we are using HiveInterpreter for analytics, the cost of creating a new connection is small compared to query execution. We don't have to merge this change into master.

What type of PR is it?

Feature

Todos

- Save user credentials in a file
- Do not persist AuthenticationInfo in paragraph since it contains sensitive information
- Add tests
- Add documentation

What is the Jira issue?

ZEPPELIN-828

How should this be tested?

In conf/shiro.ini, use /** = authc
Login to Zeppelin
Enter credentials for a data source
Check if the interpreter is using the correct credentials

Screenshots (if appropriate)

https://gfycat.com/LiquidGentleKatydid

Questions:

Does this needs documentation? Yes.

Leemoonsoo · 2016-04-30T05:48:53Z

Hi @prasadwagle. Thanks for sharing great work.

I think this is nice solution for http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Storing-Interpreter-Passwords-tp2773.html

One feedback is, about passing credential information to interpreter process.
Currently, single credential passed to interpreter process, when DATSOURCE_KEY matches to '%hive(DATASOURCE_KEY)'.

But currently, hive (jdbc) is only interpreter that leverages '%hive(DATASOURCE_KEY)'. So most of other interpreters are not able to leverage this new features.

So, how about passing all credential map (of login user) to interpreter process?
And let interpreter find appropriate credential from the map.
What do you think?

prasadwagle · 2016-05-10T19:17:22Z

@Leemoonsoo I agree with you and have implemented the change to pass all user credentials to the interpreter. I also updated the screencast. Please let me know your feedback.

Leemoonsoo · 2016-05-20T00:32:53Z

@prasadwagle Tried and looks good. I think separate issue can handle address HiveInterpreter changes. If you'd prefer.

One thing is, credential here not only can be credentials for 'datasource' but also credentials for anything, depends on how individual interpreter uses this information. right?

So how about use little bit more generic name such as 'Credential Id' or 'Credential Name', instead of 'Data source' on 'Credential' page?

prasadwagle · 2016-05-21T23:06:49Z

@Leemoonsoo I agree. I removed HiveInterpreter changes from this PR. What do you think about using Entity instead of Data source?

prasadwagle · 2016-05-22T03:56:14Z

The "TEST_SELENIUM=true" job in last build failed with the error below. I am going to close/reopen the PR to start the build.

12076 -------------------------------------------------------^M
12077 T E S T S^M
12078 -------------------------------------------------------^M
12079 Running org.apache.zeppelin.rest.ZeppelinSparkClusterTest^M
...
12390 17:25:22,650 INFO org.apache.zeppelin.notebook.Paragraph:235 - run paragraph 20160512-172522_1287820414 using pyspark org. apache.zeppelin.interpreter.LazyOpenInterpreter@14fda3ee^M
12391
12392
12393 No output has been received in the last 10 minutes, this potentially indicates a stalled build or something wrong with the build itself.
12394
12395 The build has been terminated

prasadwagle · 2016-05-22T20:23:32Z

The build finally succeeded. I have created a jira issue with flaky-test label. Is it ok to merge #860 and #856?

Leemoonsoo · 2016-05-23T16:56:36Z

Thanks @prasadwagle. It looks good to me!

felixcheung · 2016-05-24T00:06:55Z

zeppelin-interpreter/src/main/java/org/apache/zeppelin/user/Credentials.java

+      CredentialsInfoSaving info = gson.fromJson(json, CredentialsInfoSaving.class);
+      this.credentialsMap = info.credentialsMap;
+    } catch (IOException e) {
+      LOG.error("Error loading credentials file");


please use LOG.error("message", e); instead?

Thanks @felixcheung. I made the change. Unfortunately, the selenium flaky test is failing again. I am going to close/reopen the PR again. I created https://issues.apache.org/jira/browse/ZEPPELIN-878.

@prasadwagle Looks like there is still this guy left

@bzz Sorry I missed that. Fixed.

prasadwagle · 2016-05-25T00:14:39Z

The build failed again because of the flaky test. https://issues.apache.org/jira/browse/ZEPPELIN-878
@Leemoonsoo @bzz @felixcheung - what should we do about this?

prasadwagle · 2016-05-26T21:48:51Z

I just merged with the latest master and still got two build failures.
cc: @bzz @corneadoug @sagarkulkarni3592

https://travis-ci.org/apache/incubator-zeppelin/jobs/133223324
#4352.7 failed
Tests in error:
ZeppelinIT.testSparkInterpreterDependencyLoading:243 Â» NoSuchElement Unable to...

https://travis-ci.org/apache/incubator-zeppelin/jobs/133223320
#4352.3 errored
21:24:32,109 INFO org.apache.zeppelin.notebook.Paragraph:241 - run paragraph 20160526-212432_3024637 using pyspark org.apache.zeppelin.interpreter.LazyOpenInterpreter@49dfce80
No output has been received in the last 10 minutes, this potentially indicates a stalled build or something wrong with the build itself.

bzz · 2016-05-27T05:46:12Z

@prasadwagle I think it's better to actually rebase (may be even interactive, to get rid of last 2 commits) the whole branch on top of the latest pull from master, instead of just merging it in. Could you please try that and see if it solves the first issue?

prasadwagle · 2016-05-29T00:27:48Z

@bzz I tried your suggestion to rebase of the latest master but got the same error:
https://travis-ci.org/apache/incubator-zeppelin/jobs/133661149
Tests in error:
ZeppelinIT.testSparkInterpreterDependencyLoading:243 » NoSuchElement Unable to...

bzz · 2016-05-31T02:34:45Z

Let me look into that again and get back to you

bzz · 2016-05-31T09:01:12Z

It's different issue now - I have found the reason of CI failure and described at ZEPPELIN-878. Will fix it from there.

@prasadwagle I think it's good to merge now, as soon as you address felix's feedback

prasadwagle · 2016-05-31T19:09:42Z

Thanks @bzz. I think I addressed @felixcheung feedback. @felixcheung - can you please review?

bzz · 2016-06-01T00:04:00Z

@prasadwagle thank you! CI fix was merged, so may be we could try rebasing on master again and see if it makes CI happy here as well

prasadwagle · 2016-06-01T19:27:07Z

@bzz I rebased off the latest master and got the error below. Should I create a new jira issue or is this a duplicate?

https://travis-ci.org/apache/incubator-zeppelin/jobs/134535308

18:38:26,926 ERROR org.apache.zeppelin.AbstractZeppelinIT:153 - Exception in ZeppelinIT while testSparkInterpreterDependencyLoading
org.openqa.selenium.NoSuchElementException: Unable to locate element: {"method":"xpath","selector":"//div[@id='spark']//form//button[1]"}
Command duration or timeout: 30.08 seconds
...
testSparkInterpreterDependencyLoading(org.apache.zeppelin.integration.ZeppelinIT) Time elapsed: 82.819 sec <<< ERROR!
org.openqa.selenium.NoSuchElementException: Unable to locate element: {"method":"xpath","selector":"//div[@id='spark']//form//button[1]"}
Command duration or timeout: 30.08 seconds
...
Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 109.101 sec <<< FAILURE! - in org.apache.zeppelin.integration.ZeppelinIT
testSparkInterpreterDependencyLoading(org.apache.zeppelin.integration.ZeppelinIT) Time elapsed: 82.819 sec <<< ERROR!
org.openqa.selenium.NoSuchElementException: Unable to locate element: {"method":"xpath","selector":"//div[@id='spark']//form//button[1]"}
Command duration or timeout: 30.08 seconds
For documentation on this error, please visit: http://seleniumhq.org/exceptions/no_such_element.html

...
Results :

Tests in error:
ZeppelinIT.testSparkInterpreterDependencyLoading:243 Â» NoSuchElement Unable to...

Tests run: 14, Failures: 0, Errors: 1, Skipped: 0

bzz · 2016-06-02T08:49:05Z

@praagarw thanks for poining it out, it's a new one, I'm lookin into it under ZEPPELIN-942

It does not block merging this PR, so since all feedback was addressed - merge it, if there is no further discussion

### What is this PR for? There were several changes in Zeppelin UI after #860, #1006, #1013, #1081 so update screenshot of documents accordingly. ### What type of PR is it? Documentation ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: AhyoungRyu <fbdkdud93@hanmail.net> Author: Mina Lee <mina.hyeji.lee@gmail.com> Author: Mina Lee <minalee@apache.org> Closes #1089 from minahlee/doc/ZEPPELIN-1002 and squashes the following commits: b237caf [Mina Lee] Merge pull request #1 from AhyoungRyu/doc/ZEPPELIN-1002/again b18544a [AhyoungRyu] Update screenshot images in interpreters.md add97fb [AhyoungRyu] Update screenshot images in notebookashomepage.md cdaeb30 [AhyoungRyu] Update screenshot images in index.md b21444a [AhyoungRyu] Update screenshot images in notebook_authorization.md b23f7e4 [AhyoungRyu] Update screenshot images in dependencymanagement.md e7a85f3 [AhyoungRyu] Update screenshot images in lens.md cecd161 [AhyoungRyu] Update screenshot images in ignite.md 9f8cb71 [AhyoungRyu] Update screenshot images in elasticsearch.md 0c9a688 [AhyoungRyu] Hide dynamicinterpreterloading.md temporarily a17f31f [Mina Lee] Update doc image in Explore Zeppelin UI page

### What is this PR for? There were several changes in Zeppelin UI after #860, #1006, #1013, #1081 so update screenshot of documents accordingly. ### What type of PR is it? Documentation ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: AhyoungRyu <fbdkdud93@hanmail.net> Author: Mina Lee <mina.hyeji.lee@gmail.com> Author: Mina Lee <minalee@apache.org> Closes #1089 from minahlee/doc/ZEPPELIN-1002 and squashes the following commits: b237caf [Mina Lee] Merge pull request #1 from AhyoungRyu/doc/ZEPPELIN-1002/again b18544a [AhyoungRyu] Update screenshot images in interpreters.md add97fb [AhyoungRyu] Update screenshot images in notebookashomepage.md cdaeb30 [AhyoungRyu] Update screenshot images in index.md b21444a [AhyoungRyu] Update screenshot images in notebook_authorization.md b23f7e4 [AhyoungRyu] Update screenshot images in dependencymanagement.md e7a85f3 [AhyoungRyu] Update screenshot images in lens.md cecd161 [AhyoungRyu] Update screenshot images in ignite.md 9f8cb71 [AhyoungRyu] Update screenshot images in elasticsearch.md 0c9a688 [AhyoungRyu] Hide dynamicinterpreterloading.md temporarily a17f31f [Mina Lee] Update doc image in Explore Zeppelin UI page (cherry picked from commit 09f0ebd) Signed-off-by: Mina Lee <minalee@apache.org>

Leemoonsoo mentioned this pull request Apr 30, 2016

[ZEPPELIN-826] Run scheduled notes with a user #856

Closed

1 task

prasadwagle closed this May 22, 2016

prasadwagle reopened this May 22, 2016

prasadwagle closed this May 22, 2016

prasadwagle reopened this May 22, 2016

felixcheung reviewed May 24, 2016
View reviewed changes

prasadwagle closed this May 24, 2016

prasadwagle reopened this May 24, 2016

prasadwagle closed this May 25, 2016

prasadwagle reopened this May 25, 2016

prasadwagle closed this May 25, 2016

prasadwagle reopened this May 25, 2016

prasadwagle closed this May 26, 2016

prasadwagle reopened this May 26, 2016

prasadwagle force-pushed the ZEPPELIN-828 branch from 9a9824b to 46440ed Compare May 28, 2016 23:51

prasadwagle closed this May 29, 2016

prasadwagle reopened this May 29, 2016

bzz mentioned this pull request May 31, 2016

ZEPPELIN-928: fix flaky ZeppelinIT test for Spark interpreter #939

Closed

2 tasks

Prasad Wagle added 11 commits June 1, 2016 11:19

Use individual user credentials for data source authentication

90d546f

Pass all user credentials to the interpreter

cfe4c86

Persist user info instead of authentication info in paragraph

d9dae04

Persist credentials depending on conf setting

8b70f08

Fix checkstyle error, Update Data Source Authorization section in docs

d4e3cd1

Remove vertica dependency from hive/pom.xml

f2628e3

Revert changes to HiveInterpreter, Update tests

623d42a

Added conf/credentials.json to .gitignore

cbbb591

Use a more general term 'entity' instead of 'datasource'

c628099

Use LOG.error(msg, e)

62b64a0

Use LOG.error(msg, e)

f6d8de3

prasadwagle force-pushed the ZEPPELIN-828 branch from 0756962 to f6d8de3 Compare June 1, 2016 18:19

asfgit closed this in bdc78da Jun 2, 2016

AhyoungRyu mentioned this pull request Jun 4, 2016

[ZEPPELIN-939] Improve notebook authorization documentation #947

Closed

2 tasks

mwkang mentioned this pull request Jun 9, 2016

[ZEPPELIN-835]Don't persist authentication info into note.json #904

Closed

minahlee mentioned this pull request Jun 26, 2016

[DOC] Update doc image in Explore Zeppelin UI page #1089

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ZEPPELIN-828] Use individual user credentials for data source authentication #860

[ZEPPELIN-828] Use individual user credentials for data source authentication #860

prasadwagle commented Apr 27, 2016 •

edited

Leemoonsoo commented Apr 30, 2016

prasadwagle commented May 10, 2016

Leemoonsoo commented May 20, 2016

prasadwagle commented May 21, 2016

prasadwagle commented May 22, 2016

prasadwagle commented May 22, 2016

Leemoonsoo commented May 23, 2016

felixcheung May 24, 2016

prasadwagle May 24, 2016

bzz Jun 1, 2016 •

edited

prasadwagle Jun 1, 2016

prasadwagle commented May 25, 2016

prasadwagle commented May 26, 2016

bzz commented May 27, 2016 •

edited

prasadwagle commented May 29, 2016

bzz commented May 31, 2016

bzz commented May 31, 2016

prasadwagle commented May 31, 2016

bzz commented Jun 1, 2016

prasadwagle commented Jun 1, 2016

bzz commented Jun 2, 2016

[ZEPPELIN-828] Use individual user credentials for data source authentication #860

[ZEPPELIN-828] Use individual user credentials for data source authentication #860

Conversation

prasadwagle commented Apr 27, 2016 • edited

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

Leemoonsoo commented Apr 30, 2016

prasadwagle commented May 10, 2016

Leemoonsoo commented May 20, 2016

prasadwagle commented May 21, 2016

prasadwagle commented May 22, 2016

prasadwagle commented May 22, 2016

Leemoonsoo commented May 23, 2016

felixcheung May 24, 2016

Choose a reason for hiding this comment

prasadwagle May 24, 2016

Choose a reason for hiding this comment

bzz Jun 1, 2016 • edited

Choose a reason for hiding this comment

prasadwagle Jun 1, 2016

Choose a reason for hiding this comment

prasadwagle commented May 25, 2016

prasadwagle commented May 26, 2016

bzz commented May 27, 2016 • edited

prasadwagle commented May 29, 2016

bzz commented May 31, 2016

bzz commented May 31, 2016

prasadwagle commented May 31, 2016

bzz commented Jun 1, 2016

prasadwagle commented Jun 1, 2016

bzz commented Jun 2, 2016

prasadwagle commented Apr 27, 2016 •

edited

bzz Jun 1, 2016 •

edited

bzz commented May 27, 2016 •

edited