NF: rudimentary web interface #738

debanjum · 2016-08-29T17:44:43Z

yoh: supersedes original #541
since over 100 commits, buildbot tests were ran in #759 and passed

datalad ls -w results in filesystem traversal at mentioned location. stores data in list with a flat sublist of files, directories per git/annex submodule. can handle recursive case too. Next: format to json for web UI consumption

reduce code in fsmodel class by inheriting from dsmodel fix bugs in list flattening date not working, parent submodule incorrect after first, symlink unicode coercion error in non datalad modules

modify DsModel for usability in FsModel after inheritance work on issue datalad#520

prints json as it computes each directory node information. not storing json in variable/memory as expect a lot of files in annex. this json is to be consumed by the web UI for the datasets. refer issue datalad#520.

Add DOC-strings to _ls_web class, follow convention Based on comments on pull#541

use dicts to allow json.dumps to handle conversion to json refactor to traverse complete directory tree only once improve node names using leaf_node function remove redundant parent repo extraction code

…rface

Each subdirectory has a json containing the metadata on files etc available in that subdirectory only, no deeper standardizes json to simplify rendering dirs for website

adds search for file at each directory level, adds css styling based on file/folder type(i.e annex, git etc) uses datatables library for rendering json as table and dir traversal

no need to unwrap each submodule before passing to fs_traverse ignored index.html, ignore root's parent for web publish

handle GET parameters in URL to specify directory to load useful for adding permalink feature remove local assets, caching should take care of asset load speed

update browser url to reflect current directory being visualized use the dir argument passed in url to find and load dir metadata json allows accessing previous location in dataset with browser back button

passing file, display, delete option explicitly prevents users from inadvertently populating their dataset with json metadata files minor bug fix in column rendering of html

automatically update dataset metadata post-update scp web interface html from local datalad repository while initialising publication target

hardcoded path to datalad virtualenv on datalad.org server, so just works for our use-case currently, fix will require post-update hook to know location of datalad on server automagically/etc

datatable sorted by node.type, datatable doesn't support custom order need sorter to create custom ordering with directories at the top

…rface

Removes '../' parent row from top dataset in html instead of datalad ls.

…rely in the future We might not bother with those relative paths until we are actually printing those models and they might all get "relpath" property if they are provided "toppath"

yarikoptic · 2016-09-06T22:11:26Z

Thank you @aqw in advance. Indeed PR is somewhat sizeable although majority is just 3rd party pieces, but yes, there is dune python and JavaScript ( with tests) to consider.

aqw · 2016-09-07T07:29:27Z

datalad/resources/website/index.html

+		<span style="font-weight:bold">Javascript is not enabled!</span>
+	    </div>
+	</noscript>
+	<script type="text/javascript" charset="utf8" src=".git/datalad/web/assets/js/jquery-1.12.3.js"></script>


These should be charset="utf-8" to validate properly.

aqw · 2016-09-07T08:03:25Z

@debanjum Just got done taking a look at the web bits of the PR. Overall, it looks good. Most of my comments came from simply running the page through the HTML validator.

Is there any reason in particular that you're choosing to use jQuery 1.x? I know it's still supported, but it seems like an odd choice for a new project. Data Tables works with both 1.x and 2.x. The only difference I can see is for IE6-8 support, but if that's the case, I'm all for bumping to 2.x, dropping those browsers, and saving ~10KB in the process. The next version in 1.x drops < IE8 anyway, so it seems there's little to gain here.

jQuery 1.x isn't a show stopper for me, but it does merit consideration.

@yarikoptic Once the small changes I suggested are addressed, it LGTM.

---Alex

debanjum · 2016-09-07T15:19:16Z

@aqw Hey, thanks a ton for the review of the HTML and JS stuff! It really needed it ;)
I'll enable some kind of HTML validator on my setup to help out. The JS linter proved really helpful while coding the js up.

No particular reason that I can remember of for using jQuery 1.x so i'll move it to 2.x and hopefully everything should work fine.

aqw · 2016-09-07T16:16:52Z

@debanjum Glad I could be of some help.

Nice work!

---Alex

coveralls · 2016-09-07T17:33:09Z

Coverage decreased (-1.9%) to 87.043% when pulling 7d952fc on debanjum:bk-webinterface-bisect-ssh-test-fail into 5ff2165 on datalad:master.

coveralls · 2016-09-07T17:33:09Z

Coverage decreased (-1.9%) to 87.043% when pulling 7d952fc on debanjum:bk-webinterface-bisect-ssh-test-fail into 5ff2165 on datalad:master.

resolves datalad#754

To handle datasets with spaces

coveralls · 2016-09-07T19:23:51Z

Coverage decreased (-1.9%) to 87.043% when pulling 7f3d54a on debanjum:bk-webinterface-bisect-ssh-test-fail into 5ff2165 on datalad:master.

refer datalad#754

coveralls · 2016-09-07T20:48:36Z

Coverage decreased (-1.9%) to 87.047% when pulling 6e9ca7d on debanjum:bk-webinterface-bisect-ssh-test-fail into 5ff2165 on datalad:master.

coveralls · 2016-09-07T21:31:53Z

Coverage decreased (-1.9%) to 87.047% when pulling 6e9ca7d on debanjum:bk-webinterface-bisect-ssh-test-fail into 5ff2165 on datalad:master.

yarikoptic · 2016-09-08T19:46:04Z

please merge current master and push! watchout for conflicts

…ace-bisect-ssh-test-fail Conflicts: .travis.yml Resolved: Combined diffs

coveralls · 2016-09-08T20:55:57Z

Coverage decreased (-1.9%) to 87.031% when pulling 64b5381 on debanjum:bk-webinterface-bisect-ssh-test-fail into 511f3be on datalad:master.

yarikoptic · 2016-09-08T22:24:11Z

datalad/distribution/create_publication_target_sshwebserver.py

+        hook_content = '\n'.join(['#!/bin/bash', 'git update-server-info', json_command])
+
+        # make datalad logs directory
+        ssh(['mkdir', '-p', logs_remote_target])


You could've added this to hook itself... Make a separate PR since this I've hopefully will get merged!:-)

debanjum and others added 30 commits June 14, 2016 01:28

WiP: NF: Adds filesystem traversal to datalad ls

be16bfb

datalad ls -w results in filesystem traversal at mentioned location. stores data in list with a flat sublist of files, directories per git/annex submodule. can handle recursive case too. Next: format to json for web UI consumption

NF: create filesystem model for issue datalad#520

e8d69ff

reduce code in fsmodel class by inheriting from dsmodel fix bugs in list flattening date not working, parent submodule incorrect after first, symlink unicode coercion error in non datalad modules

NF: fix link, date and size bugs in FsModel for web UI

2dc0b68

modify DsModel for usability in FsModel after inheritance work on issue datalad#520

NF: dataset-web-interface: format dir info as json

99d6483

prints json as it computes each directory node information. not storing json in variable/memory as expect a lot of files in annex. this json is to be consumed by the web UI for the datasets. refer issue datalad#520.

NF: Attach single repo object to each repo subnode

c8a29b6

Add DOC-strings to _ls_web class, follow convention Based on comments on pull#541

RF:Maintain fs hierarchy in json,improve traversal

fc79919

use dicts to allow json.dumps to handle conversion to json refactor to traverse complete directory tree only once improve node names using leaf_node function remove redundant parent repo extraction code

Merge remote-tracking branch 'origin/master' into nf-dataset-web-inte…

8ba2ee6

…rface

Merge remote-tracking branch 'origin/master' into nf-dataset-web-inte…

c6457dd

…rface

RF: Maintain directory metadata in each subdir

184ce28

Each subdirectory has a json containing the metadata on files etc available in that subdirectory only, no deeper standardizes json to simplify rendering dirs for website

NF: creates website for viewing datasets

31127a5

adds search for file at each directory level, adds css styling based on file/folder type(i.e annex, git etc) uses datatables library for rendering json as table and dir traversal

RF: fs_traverse handles complete dict creation

c597ecd

no need to unwrap each submodule before passing to fs_traverse ignored index.html, ignore root's parent for web publish

NF+RF: handle GET parameters, load assets from web

138128c

handle GET parameters in URL to specify directory to load useful for adding permalink feature remove local assets, caching should take care of asset load speed

NF: Permalink, stores traversal in browser history

32c4499

update browser url to reflect current directory being visualized use the dir argument passed in url to find and load dir metadata json allows accessing previous location in dataset with browser back button

RF: removed duplication and added comments

4d0e70c

RF+NF: adds display, delete json. explicit options to json flag

e065723

passing file, display, delete option explicitly prevents users from inadvertently populating their dataset with json metadata files minor bug fix in column rendering of html

NF: Initialise Web UI on create_publication_target

9babe36

automatically update dataset metadata post-update scp web interface html from local datalad repository while initialising publication target

TST: add tests for datalad ls_json and its helpers

1c05af2

RF: cleanup, output valid json on ls json=display

2c5b742

BF: publish works now but only for datalad.org

6dc3df6

hardcoded path to datalad virtualenv on datalad.org server, so just works for our use-case currently, fix will require post-update hook to know location of datalad on server automagically/etc

BF/NF: open files in browser. add, order columns

5d76894

datatable sorted by node.type, datatable doesn't support custom order need sorter to create custom ordering with directories at the top

NF+RF: more robust node sizes and UI node sorting

593d6cf

NF: Add HTML styling, title, columns, icon etc

c6f44f2

Merge remote-tracking branch 'origin/master' into nf-dataset-web-inte…

d7a6a8b

…rface

BF: create hook locally before uploading to remote

3f2d6ed

Merge branch 'master' into nf-dataset-web-interface

21eb75d

BF: Handle parent data row better, minor refactor

2b65a3b

Removes '../' parent row from top dataset in html instead of datalad ls.

RF: sum size of all children to compute node size

33e7200

RF(BK): some minor changes to FsModel within ls

dd8d2b7

RF: DsModel pretty much became RepoBasedModel and might dissolve enti…

1b32f16

…rely in the future We might not bother with those relative paths until we are actually printing those models and they might all get "relpath" property if they are provided "toppath"

BF: fix print statement to pass python3 build test

77d1aa0

aqw reviewed Sep 7, 2016
View reviewed changes

debanjum added 2 commits September 7, 2016 12:51

RF: Improve styling consistency

939f208

RF: Update jquery to 2.x, use version in asset names

7d952fc

debanjum added 3 commits September 7, 2016 14:04

BF: Post-update hook should handle spaces in datasets

02cfadc

resolves datalad#754

TST: Ensure dataset path wrapped in quotes in postupdate hook

7f3d54a

To handle datasets with spaces

TST: Test publishing web-interface only to rootds

1705939

ENH: Log post_update logs in the dataset itself

6e9ca7d

refer datalad#754

debanjum force-pushed the bk-webinterface-bisect-ssh-test-fail branch from 6e9ca7d to 40318e8 Compare September 7, 2016 21:04

debanjum closed this Sep 7, 2016

debanjum force-pushed the bk-webinterface-bisect-ssh-test-fail branch from 40318e8 to 5bbbe2f Compare September 7, 2016 21:05

debanjum reopened this Sep 7, 2016

Merge branch 'master' of github.com:datalad/datalad into bk-webinterf…

64b5381

…ace-bisect-ssh-test-fail Conflicts: .travis.yml Resolved: Combined diffs

yarikoptic reviewed Sep 8, 2016
View reviewed changes

yarikoptic merged commit fa6e6af into datalad:master Sep 8, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NF: rudimentary web interface #738

NF: rudimentary web interface #738

debanjum commented Aug 29, 2016 •

edited by yarikoptic

Loading

yarikoptic commented Sep 6, 2016

aqw Sep 7, 2016

aqw commented Sep 7, 2016

debanjum commented Sep 7, 2016 •

edited

Loading

aqw commented Sep 7, 2016

coveralls commented Sep 7, 2016

coveralls commented Sep 7, 2016

coveralls commented Sep 7, 2016 •

edited

Loading

coveralls commented Sep 7, 2016

coveralls commented Sep 7, 2016 •

edited

Loading

yarikoptic commented Sep 8, 2016

coveralls commented Sep 8, 2016 •

edited

Loading

yarikoptic Sep 8, 2016

NF: rudimentary web interface #738

NF: rudimentary web interface #738

Conversation

debanjum commented Aug 29, 2016 • edited by yarikoptic Loading

yarikoptic commented Sep 6, 2016

aqw Sep 7, 2016

Choose a reason for hiding this comment

aqw commented Sep 7, 2016

debanjum commented Sep 7, 2016 • edited Loading

aqw commented Sep 7, 2016

coveralls commented Sep 7, 2016

coveralls commented Sep 7, 2016

coveralls commented Sep 7, 2016 • edited Loading

coveralls commented Sep 7, 2016

coveralls commented Sep 7, 2016 • edited Loading

yarikoptic commented Sep 8, 2016

coveralls commented Sep 8, 2016 • edited Loading

yarikoptic Sep 8, 2016

Choose a reason for hiding this comment

debanjum commented Aug 29, 2016 •

edited by yarikoptic

Loading

debanjum commented Sep 7, 2016 •

edited

Loading

coveralls commented Sep 7, 2016 •

edited

Loading

coveralls commented Sep 7, 2016 •

edited

Loading

coveralls commented Sep 8, 2016 •

edited

Loading