Merge pull request #23 from asmacdo/structural-improvements

Structural improvements

CONTRIBUTING.md

@@ -48,8 +48,7 @@ Every project on GitHub uses [issues](https://github.com/dbic/handbook/issues) s
 The following outlines how DBIC developers think about communicating through issues.
 
 **Issues** are individual pieces of work that need to be completed or decisions that need to be made to move the project forwards.
-A general guideline: if you find yourself tempted to write a great big issue that
-is difficult to describe as one unit of work, please consider splitting it into two or more issues.
+A general guideline: if you find yourself tempted to write a great big issue that is difficult to describe as one unit of work, please consider splitting it into two or more issues.
 
 Issues are assigned [labels](#issue-labels) which explain how they relate to the overall project's goals and immediate next steps.
 
@@ -76,38 +75,44 @@ The current list of labels are [here](https://github.com/dbic/handbook/labels) a
 
 The handbook documents follow the [Markdown Style Guide](http://www.cirosantilli.com/markdown-style-guide/).
 
-You
-can validate your changes against the guide using [remark](https://github.com/remarkjs/remark-lint) which works as a
-[standalone command line tool](https://github.com/remarkjs/remark/tree/master/packages/remark-cli) as well as [a plugin for various text editors](https://github.com/remarkjs/remark-lint#editor-integrations). Remark preserves consistent markdown styling across the contributions. Please ensure before submitting a contribution that you do not have any linter errors in your text editor.
+You can validate your changes against the guide using [remark](https://github.com/remarkjs/remark-lint) which works as a [standalone command line tool](https://github.com/remarkjs/remark/tree/master/packages/remark-cli) as well as [a plugin for various text editors](https://github.com/remarkjs/remark-lint#editor-integrations).
+Remark preserves consistent markdown styling across the contributions.
+Please ensure before submitting a contribution that you do not have any linter errors in your text editor.
 You can also use [prettier](https://github.com/prettier/prettier) to automatically correct some of the style issues that might be found in the proposed changes.
 
-We have deployed a continuous integrator ([circle CI](https://circleci.com/)) to further allow for integrating changes continuously. The CI is testing that the changes are inline with our standard styling.
+We have deployed a continuous integrator ([circle CI](https://circleci.com/)) to further allow for integrating changes continuously.
+The CI is testing that the changes are inline with our standard styling.
 
 GitHub has a helpful page on [getting started with writing and formatting on GitHub](https://help.github.com/articles/getting-started-with-writing-and-formatting-on-github).
 
 ## Building the handbook using mkdocs
 
-We are using mkdocs to render our handbook. Please follow these instructions if you would like to build the handbook locally.
+We are using mkdocs to render our handbook. 
+Please follow these instructions if you would like to build the handbook locally.
 
 #### 1. Install mkdocs
 
 To begin please follow [this link](https://www.mkdocs.org/#installation) to install mkdocs locally.
 
-#### 2. Download the DBIC Handbook [repository](https://github.com/dbic/handbook/tree/master) onto your computer
+#### 2. Download the DBIC Handbook [repository](https://github.com/dbic/handbook/tree/master)
 
-This can be done by clicking the green button on the right titled "Clone or download"
+Cloning the repository with `git`(TODO link to docs). You will need `git` to contribute the changes you make later.
 
 #### 3. Install our theme
 
-Please go [here](https://squidfunk.github.io/mkdocs-material/) and install our theme - material. The terminal command is `pip install mkdocs-material`
+`pip install mkdocs-material`
+See [theme website](https://squidfunk.github.io/mkdocs-material/) for more info.
 
 #### 4. In the terminal (command line) navigate to your local version of the handbook
 
 This location will have the same files you see on our [main handbook page](https://github.com/dbic/handbook). Note: A finder window may not show the hidden files (those that start with a period i.e. .remarkrc)
 
 #### 5. Ready to build!
 
-Using the terminal (command line) please enter `mkdocs serve`. This will allow you to see a local version of the handbook. The local address will be `http://127.0.0.1:8000`. You may enter that into your browser and this will bring up the handbook!
+Using the terminal (command line) please enter `mkdocs serve`.
+This will allow you to see a local version of the handbook.
+The local address will be `http://127.0.0.1:8000` or `localhost:8000`.
+You may enter that into your browser and this will bring up the handbook!
 
 ## Making a change with a pull request
 
@@ -117,40 +122,39 @@ We appreciate all contributions to the DBIC Handbook. **THANK YOU** for helping
 
 This allows other members of the DBIC Handbook team to confirm that you aren't overlapping with work that's currently underway and that everyone is on the same page with the goal of the work you're going to carry out.
 
-
 #### 2. [Fork](https://help.github.com/articles/fork-a-repo/) [this repository](https://github.com/dbic/handbook) to your profile
 
-This is now your own unique copy of the DBIC Handbook. Changes here won't affect anyone else's work, so it's a safe space to explore edits to the handbook!
+This is now your own unique copy of the DBIC Handbook.
+Changes here won't affect anyone else's work, so it's a safe space to explore edits to the handbook!
 
 Make sure to [keep your fork up to date](https://help.github.com/articles/syncing-a-fork/) with the master repository, otherwise you can end up with lots of dreaded [merge conflicts](https://help.github.com/articles/about-merge-conflicts/).
 
 #### 3. Make the changes you've discussed
 
-Try to keep the changes focused. If you submit a large amount of work in all in one go it will be much more work for whomever is reviewing your pull request. Please detail the changes you are attempting to make.
+Try to keep the changes focused.
+If you submit a large amount of work in all in one go it will be much more work for whomever is reviewing your pull request.
+Please detail the changes you are attempting to make.
 
 #### 4. Submit a [pull request](https://help.github.com/articles/about-pull-requests/)
 
-Please keep the title of your pull request short but informative - it will
-appear in the [changelog](CHANGES.md).
+Please keep the title of your pull request short but informative - it will appear in the [changelog](CHANGES.md).
 
 Use one of the following prefixes in the title of your pull request:
-  - `[ENH]` - enhancement of the handbook that adds a new feature or
-    support for a new data type
+  - `[ENH]` - enhancement of the handbook that adds a new feature or support for a new data type
   - `[FIX]` - fix of a typo or language clarification
-  - `[INFRA]` - changes to the infrastructure automating the handbook
-    release (for example building HTML docs etc.)
-  - `[MISC]` - everything else including changes to the file listing
-    contributors
+  - `[INFRA]` - changes to the infrastructure automating the handbook release (for example building HTML docs etc.)
+  - `[MISC]` - everything else including changes to the file listing contributors
 
-If you are opening a pull request to obtain early feedback, but the changes
-are not ready to be merged (a.k.a. Work in Progress pull request) please
-use a [draft pull request](https://github.blog/2019-02-14-introducing-draft-pull-requests/).
+If you are opening a pull request to obtain early feedback, but the changes are not ready to be merged (a.k.a. Work in Progress pull request) please use a [draft pull request](https://github.blog/2019-02-14-introducing-draft-pull-requests/).
 
 A member of the DBIC Handbook team will review your changes to confirm that they can be merged into the main codebase.
 
-A [review](https://help.github.com/articles/about-pull-request-reviews/) will probably consist of a few questions to help clarify the work you've done. Keep an eye on your GitHub notifications and be prepared to join in that conversation.
+A [review](https://help.github.com/articles/about-pull-request-reviews/)
+will probably consist of a few questions to help clarify the work you've done.
+Keep an eye on your GitHub notifications and be prepared to join in that conversation.
 
-You can update your [fork](https://help.github.com/articles/about-forks/) of the DBIC Handbook and the pull request will automatically update with those commits. You don't need to submit a new pull request when you make a change in response to a review.
+You can update your [fork](https://help.github.com/articles/about-forks/) of the DBIC Handbook and the pull request will automatically update with those commits.
+You don't need to submit a new pull request when you make a change in response to a review.
 
 GitHub has a [nice introduction](https://help.github.com/articles/github-flow/) to the pull request workflow, but please [get in touch](https://github.com/dbic/handbook/issues/new) if you have any questions.
 
@@ -160,22 +164,18 @@ GitHub has a [nice introduction](https://help.github.com/articles/github-flow/)
 
 ## Fixing Travis Remark errors
 
-We use a linter called [Remarkjs](https://github.com/remarkjs/remark-lint) to ensure all of
-our Markdown documents are consistent and well-styled.
-This commonly produces errors, which are flagged by [Travis CI](https://travis-ci.org/),
-a continuous integration service.
+We use a linter called [Remarkjs](https://github.com/remarkjs/remark-lint) to ensure all of our Markdown documents are consistent and well-styled.
+This commonly produces errors, which are flagged by [Travis CI](https://travis-ci.org/), a continuous integration service.
 When Travis returns an error, use the following process to resolve the issue:
 
 #### 1. Install NodeJS / npm
 
 We use a markdown linter written in Javascript.
-To run command Javascript tools on the command line, please [download and
-install](https://nodejs.org/en/download/) NodeJS.
+To run command Javascript tools on the command line, please [download and install](https://nodejs.org/en/download/) NodeJS.
 
 #### 2. Install Remark-CLI and our style guide
 
-Remark-CLI can be installed via [npm](https://www.npmjs.com/), which is part of
-the NodeJS distribution.
+Remark-CLI can be installed via [npm](https://www.npmjs.com/), which is part of the NodeJS distribution.
 
 To install the packages we use for our style guide, the following command will work on most command lines:
 
@@ -205,8 +205,7 @@ remark flagged_file_fixed.md --frail
 ```
 
 This command will indicate whether this file now conforms to the style guide.
-If it passes, replace `flagged_file.md` with the contents of `flagged_file_fixed.md`,
-add and commit the change:
+If it passes, replace `flagged_file.md` with the contents of `flagged_file_fixed.md`, add and commit the change:
 
 ```
 mv flagged_file_fixed.md flagged_file.md
@@ -216,8 +215,7 @@ git commit -m 'STY: Fixed Markdown style'
 
 ## How the decision to merge a pull request is made?
 
-TODO: decide on either we should establish a formal procedure.  yoh thinks it might not be needed to be that formal for starters.
-
+Everyone use your best judgement, it's worked fine so far.
 
 ## Recognizing contributions
 

mkdocs.yml

@@ -17,12 +17,11 @@ plugins:
 docs_dir: 'src'
 use_directory_urls: false
 nav:
-    - Introduction: 01-introduction.md
-    - Useful basics:
-      - Shell: TODO.md
+    - Fundamentals:
+      - Shell: basics/shell.md
       - SSH: TODO.md
-      - Containers: containers.md
-      - DataLad: datalad.md
+      - Containers: basics/containers.md
+      - DataLad: basics/datalad.md
       - PyBIDS: TODO.md
     - A new study:
         - Policies & procedures: https://www.dartmouth.edu/dbic/
@@ -44,16 +43,14 @@ nav:
     - EEG: TODO.md
     - MRS: TODO.md
     - Computing resources:
-      - Discovery HPC: discovery.md
+      - Discovery HPC: computing/discovery.md
     - Publishing:
         - Datasets: TODO.md
-        - Data papers: data-paper.md
+        - Data papers: publishing/data-paper.md
         - Study papers: TODO.md
     - Appendix:
         - Contributors: 99-appendices/01-contributors.md
         - Licenses: 99-appendices/02-licenses.md
     - Additional resources: additional_resources.md
     - Changelog: CHANGES.md
     - Contributing: CONTRIBUTING.md
-    - DBIC website: http://dartmouth.edu/dbic
-    - DBIC GitHub: http://github.com/dbic

src/99-appendices/01-contributors.md

@@ -31,3 +31,4 @@ ecosystem (in alphabetical order). If you contributed to the DBIC ecosystem and
 your name is not listed, please add it.
 
 -   Yaroslav O. Halchenko 📖📢🔧💬🐛👀
+-   Austin Macdonald📖🔧💬🐛👀

src/additional_resources.md

@@ -1,14 +1,19 @@
 # Additional resources
 
-## MRI collection/analysis
+## MISC links
 
-- [DartBrains](https://dartbrains.org) - the Psyc60 (Dartmouth fMRI Analysis)
-  course by [Prof. Luke Chang (COSANLab)](http://cosanlab.com), with accent
-  on Python
+- [DBIC website](http://dartmouth.edu/dbic)
+- [DBIC GitHub](http://github.com/dbic)
 
-- [DataLad Handbook](http://handbook.datalad.org) - all there is to know to use DataLad
+## MRI collection/analysis
 
+- [DartBrains](https://dartbrains.org) - the Psyc60 (Dartmouth fMRI Analysis) course by [Prof. Luke Chang (COSANLab)](http://cosanlab.com), with accent on Python
+- [DataLad Handbook](http://handbook.datalad.org) - all there is to know to use DataLad
 - [The Princeton Handbook for Reproducible Neuroimaging](https://brainhack-princeton.github.io/handbook/) - another great handbook.
-  Many questions not clarified here, could find answers in that resource.  ReproIn is also
-  used as a naming convention at the scanner.
+  Many questions not clarified here, could find answers in that resource.
+  ReproIn is also used as a naming convention at the scanner.
+
+## con/catenate
 
+The Center for Open Neuroscience has a collection of resources that maybe useful to you.
+[con/catenate](https://github.com/con/catenate)

src/basics/containers.md

@@ -0,0 +1,14 @@
+## ReproNim/containers
+
+We highly recommend (re)using collection of Singularity containers prepared by ReproNim project, and ideally using datalad-container extension.
+See https://github.com/ReproNim/containers#a-typical-yoda-workflow for more information on how it should typically be used.
+
+Besides using that collection via datalad, you can use them directly for the images/versions we already prefetched (if any is missing, please let us know).
+That might be preferable over manual fetching of container copies to not waste more space on discovery (some containers are large). E.g. to validate your BIDS dataset using bids-validator, do:
+
+    cd path/to/my/bids/dataset
+    singularity run -B $PWD:$PWD /dartfs/rc/lab/D/DBIC/DBIC/archive/containers/images/bids/bids-validator--1.11.0.sing $PWD
+
+## NeuroDocker
+## ReproZip
+## ReproMan 

src/datalad.md → src/basics/datalad.md

@@ -1,11 +1,10 @@
 # DataLad
 
-The DataLad Handbook at [http://handbook.datalad.org](http://handbook.datalad.org)
-is the ultimate introduction into DataLad. 
+The DataLad Handbook at [http://handbook.datalad.org](http://handbook.datalad.org) is the ultimate introduction into DataLad. 
 
 ## An example of sample analysis using DataLad & containers
 
-This is a WiP example on how to organize your study as DataLad dataset and
+This is an example on how to organize your study as DataLad dataset and
 use [datalad-container](http://docs.datalad.org/projects/container/en/latest/index.html)
 extension to facilitate efficient and reproducible research:
 
@@ -23,8 +22,6 @@ datalad install -d . -s rolando.cns.dartmouth.edu:/inbox/BIDS/Haxby/Sam/1021_act
 datalad get bids/sub-sid000005
 ```
 
-TODO: finish
-
 Meanwhile, have a look at 
 
 - https://github.com/ReproNim/containers/#a-typical-workflow 
@@ -38,13 +35,45 @@ Meanwhile, have a look at
 
 **The problem**: web browser de-references symlinks, which leads `.html` into a `.git/annex/objects` subfolder and thus makes it impossible to see the images.
 
-![datalad-fmriprep-brokenview_min.jpg](images/datalad-fmriprep-brokenview_min.jpg)
+![datalad-fmriprep-brokenview_min.jpg](../images/datalad-fmriprep-brokenview_min.jpg)
 
 To overcome this, start a simple web server, e.g. as provided by Python itself, and navigate to the file of interest:
 
 - Run this line of code on local directory (e.g., where you have mounted discovery directory, or have a local clone with data fetched): `python -m http.server 8081`
 - Copy the URL it prints out (typically  http://0.0.0.0:8081/) into your web browser address bar
 - Now you should be able to access HTML outputs with embedded images at that address -- just navigate to .html file of interest and open it.
 
-![datalad-fmriprep-goodview-terminal_min.jpg](images/datalad-fmriprep-goodview-terminal_min.jpg)
-![datalad-fmriprep-goodview_min.jpg](images/datalad-fmriprep-goodview_min.jpg)
+### DataLad way
+
+Make sure you have git configured, in that it knows about you.
+If you have output to running `git config user.name` command, then most likely you are all set.
+If it comes out empty, please do
+
+
+    git config --global user.name "First Last"
+    git config --global user.email "someemail@address"
+
+while replacing those values with your name and email.
+
+If you are doing it to discovery HPC, please first checkout the next section of this documentation below (Discovery filesystem) on how to configure your git for Discovery's ACL filesystem, unless you know that it is pure POSIX.
+
+Then it is recommended to create a directory for your study first, e.g. `mkdir ID_name` where `ID_name` are the same as on rolando, e.g. `0001_dbic-animals`, and `cd` into it, e.g.:
+
+    mkdir 0001_dbic-animals
+    cd 0001_dbic-animals 
+
+Use [datalad install](https://docs.datalad.org/en/stable/generated/man/datalad-install.html) command
+to obtain dataset, and then [datalad get](http://docs.datalad.org/en/stable/generated/man/datalad-get.html) to 
+obtain specific files.  If you are greedy, add `-r` to get full hierarchy of datasets, and/or `-g`
+to immediately also fetch all data files
+
+    datalad install -s your-login-on-rolando@rolando.cns.dartmouth.edu:/inbox/BIDS/dbic/0001_dbic-animals dbic
+    cd dbic
+    datalad get -J4 sub-*  # to get only converted data, without tarballs etc
+
+Later upgrades to fetch new data (subjects etc) could be done via 
+
+    datalad update --how merge -r 
+
+![datalad-fmriprep-goodview-terminal_min.jpg](../images/datalad-fmriprep-goodview-terminal_min.jpg)
+![datalad-fmriprep-goodview_min.jpg](../images/datalad-fmriprep-goodview_min.jpg)

src/basics/shell.md

@@ -0,0 +1,4 @@
+## Shell / Terminal / Command Line
+
+The rest of this guide assumes basic knowledge of the `terminal`.
+The [Datalad Handbook](https://handbook.datalad.org/en/latest/intro/howto.html) covers most what you'll need to know.