Add File Interpreter, HDFS Interpreter and Tests #276

raj-bains · 2015-09-03T07:00:49Z

This is the first version of File Interpreter that adds basic ls, cd and pwd functionality against WebHDFS. It addresses ZEPPELIN-198

bzz · 2015-09-03T07:47:11Z

file/src/test/java/org/apache/zeppelin/file/HDFSFileInterpreterTest.java

👍 for having tests!
Can you also please explain how Hive interpreter is involved here, as couple of other places mentions it too?

Typo, I started the test file by a copy, I'll remove the references.

bzz · 2015-09-03T08:14:56Z

Thank you for great contribution!
Can you please also provide the documentation for new interpreter (needs to be in a separate PR to gh-pages branch)?

I.e right now it looks like only a limited subset of commands is supported, so it's a good idea mention things like that.

We keep track of list of interpreters here https://github.com/apache/incubator-zeppelin/blob/gh-pages/docs/index.md and here https://github.com/apache/incubator-zeppelin/tree/gh-pages/docs/interpreter a new 'page' can be added.

bzz · 2015-09-03T08:16:44Z

file/src/main/java/org/apache/zeppelin/file/HDFSFileInterpreter.java

Are you sure that would be a good default setting for all the Zeppelin users?

This seems to be Hortonworks specific...

I'll change this to be localhost.

bzz · 2015-09-04T00:08:59Z

Most probably you also want to update conf/zeppelin-site.xml.template property zeppelin.interpreters with name of the new interpreter class

…c ls, cd and pwd functionality against WebHDFS. It addresses ZEPPELIN-198

Merge with master so that the documentation can be checked-in using single commit

raj-bains · 2015-09-10T20:46:54Z

The request is ready to merge. Made changes based on the comments including adding documentation.

felixcheung · 2015-09-10T21:47:11Z

docs/docs/index.md

shouldn't this be ../docs/ like the other ones?

Thanks for looking at the change again. The real ones are different from the missing ones (../docs/pleasecontribute.html), below is a larger snippet that should help:
edited - to show text

flink -../docs/pleasecontribute.html

hdfs - ./interpreter/hdfs.html

hive - ../docs/pleasecontribute.html)

ignite - ../docs/pleasecontribute.html)

lens - ../docs/pleasecontribute.html)

md - ../docs/pleasecontribute.html)

sh - ../docs/pleasecontribute.html)

spark - ./interpreter/spark.html)

tajo - ../docs/pleasecontribute.html)

postgresql, hawq - ./interpreter/postgresql.html)

geode - ./interpreter/geode.html)

ah, got it. thanks

bzz · 2016-01-05T10:17:00Z

@raj-bains could you please elaborate on status of this PR - do you want to update\add doc\merge it or close it unresolved?

raj-bains · 2016-01-10T04:19:05Z

I'm on vacation for 10 days, then I'll update the patch and check this in

felixcheung · 2016-02-02T06:46:06Z

@raj-bains any update?

felixcheung · 2016-02-21T21:03:25Z

Ping. Would anyone else like to pick this up if we don't hear back from @raj-bains ?

runyontr · 2016-02-26T03:22:57Z

@felixcheung I'll be happy to take this up. From what I can see there is the following left:

Change string creation to use StringBuilder
Throw exception up the chain in the isDirectory()
Update conf/zeppelin-site.xml.template with HdfsInterpretor

One I've seen is that the alignment of columns when using %hdfs ls -l when column sizes are drastically different cause a mis alignment in its display.

I had to change the POM file to get the interpreter for runtime execution as well, I'll include that change as well.

What other items would you like me to accomplish

What I've done is forked the apache/incubator-zeppelin git repo and merged raj-bains/incubator-zeppelin (fixed merge conflicts). Given that I can fix the list above, how do I update this pull request with my repository?

bzz · 2016-02-26T03:44:50Z

@runyontr thanks for standing up!

Given that I can fix the list above, how do I update this pull request with my repository?

It's not really possible, so what you may do instead is - submit a new PR following CONTRIBUTING guidelines , mentioning this work and linking new PR from here as well i.e by using Github #NNN mentions.

If you could use ZEPPELIN-198 issue name in your commit messages - it will be linked to the JIRA ticket as well.

### What is this PR for? This pull request is a follow of #276 started by raj-bains. The additional commits address comments from the pull request regarding string creation and error propagation for bad object requests. ### What type of PR is it? [Bug Fix | Improvement | Feature | Documentation | Hot Fix | Refactoring] Feature/Subtask ### Todos ### Is there a relevant Jira issue? [ZEPPELIN-198](https://issues.apache.org/jira/browse/ZEPPELIN-198) ### How should this be tested? Outline the steps to test the PR here. ### Screenshots (if appropriate) ![selection_001](https://cloud.githubusercontent.com/assets/9200575/13370345/87a922ac-dcd2-11e5-9217-66ec21c016de.png) ![selection_002](https://cloud.githubusercontent.com/assets/9200575/13370347/89a5603e-dcd2-11e5-9555-9c38167e8667.png) ![selection_003](https://cloud.githubusercontent.com/assets/9200575/13370359/b9bbadc8-dcd2-11e5-97bd-47d216f0c7da.png) ![selection_004](https://cloud.githubusercontent.com/assets/9200575/13370364/16ff9274-dcd3-11e5-9b4c-f324e7035c20.png) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Tom Runyon <runyontr@gmail.com> Author: Raj Bains <rajbains@Rajs-MacBook-Pro.local> Closes #752 from runyontr/master and squashes the following commits: f7bfef8 [Tom Runyon] ZEPPELIN-198 added transitive dependency lincense information af16ce0 [Tom Runyon] ZEPPELIN-198 removed GPL2 reference from license file 5e9e131 [Tom Runyon] ZEPPELIN-198 Updated ZeppelinConfiguration to hold HDFS interpreter 9832622 [Tom Runyon] ZEPPELIN-198 Changed group for hdfs interpreter c34a913 [Tom Runyon] ZEPPELIN-198 Updated licenses file to include org.glassfish.jersey.core 8d0ee3d [Tom Runyon] Merge https://github.com/apache/incubator-zeppelin 9f66514 [Tom Runyon] ZEPPELIN-198 Removed extra copy of configuration table and fixed formatting issues for hdfs.md 5938b0e [Tom Runyon] ZEPPELIN-198 Updated documentation to be consistent with naming between pom file and interpreter documentation. 67bbc5b [Tom Runyon] ZEPPELIN-198 Added navigation to hdfs interpreter 1c7a5c2 [Tom Runyon] ZEPPELIN-198 removed errant text in documentation. 56a5174 [Tom Runyon] ZEPPELIN-198 fixed logging to match standards 933c890 [Tom Runyon] MAINT Updated .gitignore file to remove zeppelin-server/derby.log 29540df [Tom Runyon] ZEPPELIN-198 removed zeppelin-server/derby.log 71d53d3 [Tom Runyon] ZEPPELIN-198 Changed pom name to Zeppelin File System Interpreters to match functionality aec0512 [Tom Runyon] ZEPPELIN-198 Fixed compile error for error logging. d24f4c0 [Tom Runyon] ZEPPELIN-198 Added error logging when returning error in interpet 227b815 [Tom Runyon] ZEPPELIN-198 Updated interpreter documentation. b505391 [Tom Runyon] ZEPPELIN-198 Added completion functionality to HDFSInterpreter 32ed7cb [Tom Runyon] ZEPPELIN-198 Added -h flag for human readable byte sizes. Updated string creation to StringBuilder. 797fd29 [Tom Runyon] Added org.glassfish.jersey.core to pom.xml file for hdfs intepretor 27e0438 [Tom Runyon] Modified string creation to use StringBuilder 79f0d90 [Tom Runyon] Merge branch 'master' of http://github.com/raj-bains/incubator-zeppelin 70507a8 [Raj Bains] Add Documentation and a missing dependency for HDFS File Browser 1239fe6 [Raj Bains] Merge remote-tracking branch 'upstream/master' 7d61e5f [Raj Bains] This is the first reviewed version of File Interpreter that adds basic ls, cd and pwd functionality against WebHDFS. It addresses ZEPPELIN-198 865e6ab [Raj Bains] Add File Interpreter, HDFS Interpreter and Tests

Force ace-angular version to 0.1.1

### What is this PR for? This pull request is a follow of apache#276 started by raj-bains. The additional commits address comments from the pull request regarding string creation and error propagation for bad object requests. ### What type of PR is it? [Bug Fix | Improvement | Feature | Documentation | Hot Fix | Refactoring] Feature/Subtask ### Todos ### Is there a relevant Jira issue? [ZEPPELIN-198](https://issues.apache.org/jira/browse/ZEPPELIN-198) ### How should this be tested? Outline the steps to test the PR here. ### Screenshots (if appropriate) ![selection_001](https://cloud.githubusercontent.com/assets/9200575/13370345/87a922ac-dcd2-11e5-9217-66ec21c016de.png) ![selection_002](https://cloud.githubusercontent.com/assets/9200575/13370347/89a5603e-dcd2-11e5-9555-9c38167e8667.png) ![selection_003](https://cloud.githubusercontent.com/assets/9200575/13370359/b9bbadc8-dcd2-11e5-97bd-47d216f0c7da.png) ![selection_004](https://cloud.githubusercontent.com/assets/9200575/13370364/16ff9274-dcd3-11e5-9b4c-f324e7035c20.png) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Tom Runyon <runyontr@gmail.com> Author: Raj Bains <rajbains@Rajs-MacBook-Pro.local> Closes apache#752 from runyontr/master and squashes the following commits: f7bfef8 [Tom Runyon] ZEPPELIN-198 added transitive dependency lincense information af16ce0 [Tom Runyon] ZEPPELIN-198 removed GPL2 reference from license file 5e9e131 [Tom Runyon] ZEPPELIN-198 Updated ZeppelinConfiguration to hold HDFS interpreter 9832622 [Tom Runyon] ZEPPELIN-198 Changed group for hdfs interpreter c34a913 [Tom Runyon] ZEPPELIN-198 Updated licenses file to include org.glassfish.jersey.core 8d0ee3d [Tom Runyon] Merge https://github.com/apache/incubator-zeppelin 9f66514 [Tom Runyon] ZEPPELIN-198 Removed extra copy of configuration table and fixed formatting issues for hdfs.md 5938b0e [Tom Runyon] ZEPPELIN-198 Updated documentation to be consistent with naming between pom file and interpreter documentation. 67bbc5b [Tom Runyon] ZEPPELIN-198 Added navigation to hdfs interpreter 1c7a5c2 [Tom Runyon] ZEPPELIN-198 removed errant text in documentation. 56a5174 [Tom Runyon] ZEPPELIN-198 fixed logging to match standards 933c890 [Tom Runyon] MAINT Updated .gitignore file to remove zeppelin-server/derby.log 29540df [Tom Runyon] ZEPPELIN-198 removed zeppelin-server/derby.log 71d53d3 [Tom Runyon] ZEPPELIN-198 Changed pom name to Zeppelin File System Interpreters to match functionality aec0512 [Tom Runyon] ZEPPELIN-198 Fixed compile error for error logging. d24f4c0 [Tom Runyon] ZEPPELIN-198 Added error logging when returning error in interpet 227b815 [Tom Runyon] ZEPPELIN-198 Updated interpreter documentation. b505391 [Tom Runyon] ZEPPELIN-198 Added completion functionality to HDFSInterpreter 32ed7cb [Tom Runyon] ZEPPELIN-198 Added -h flag for human readable byte sizes. Updated string creation to StringBuilder. 797fd29 [Tom Runyon] Added org.glassfish.jersey.core to pom.xml file for hdfs intepretor 27e0438 [Tom Runyon] Modified string creation to use StringBuilder 79f0d90 [Tom Runyon] Merge branch 'master' of http://github.com/raj-bains/incubator-zeppelin 70507a8 [Raj Bains] Add Documentation and a missing dependency for HDFS File Browser 1239fe6 [Raj Bains] Merge remote-tracking branch 'upstream/master' 7d61e5f [Raj Bains] This is the first reviewed version of File Interpreter that adds basic ls, cd and pwd functionality against WebHDFS. It addresses ZEPPELIN-198 865e6ab [Raj Bains] Add File Interpreter, HDFS Interpreter and Tests

Add File Interpreter, HDFS Interpreter and Tests

865e6ab

bzz reviewed Sep 3, 2015
View reviewed changes

Raj Bains added 3 commits September 4, 2015 21:29

This is the first reviewed version of File Interpreter that adds basi…

7d61e5f

…c ls, cd and pwd functionality against WebHDFS. It addresses ZEPPELIN-198

Merge remote-tracking branch 'upstream/master'

1239fe6

Merge with master so that the documentation can be checked-in using single commit

Add Documentation and a missing dependency for HDFS File Browser

70507a8

felixcheung reviewed Sep 10, 2015
View reviewed changes

runyontr mentioned this pull request Feb 27, 2016

ZEPPELIN-198 HDFS File Interpreter #752

Closed

lelou6666 pushed a commit to lelou6666/incubator-zeppelin that referenced this pull request Mar 25, 2016

Merge pull request apache#276 from NFLabs/fix/ace-angular

147ee9d

Force ace-angular version to 0.1.1

asfgit closed this in 0209bc3 Jun 26, 2016

Add File Interpreter, HDFS Interpreter and Tests #276

Add File Interpreter, HDFS Interpreter and Tests #276

Uh oh!

Conversation

raj-bains commented Sep 3, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bzz commented Sep 3, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bzz commented Sep 4, 2015

Uh oh!

raj-bains commented Sep 10, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bzz commented Jan 5, 2016

Uh oh!

raj-bains commented Jan 10, 2016

Uh oh!

felixcheung commented Feb 2, 2016

Uh oh!

felixcheung commented Feb 21, 2016

Uh oh!

runyontr commented Feb 26, 2016

Uh oh!

bzz commented Feb 26, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants