Contributing documentation

docs/contributing_tutorials/adding_a_new_template.rst

@@ -0,0 +1,113 @@
+Creating a new template space
+###############################################
+
+Who is this tutorial for
+============================
+
+First, this is intended for those wishing to add templates to TemplateFlow.
+Second, this is for people who want to add a template directory that does not already exists.
+TemplateFlow consists of multiple templates sorted by the space the template is in.
+This tutorial tells you how to add a new template space.
+
+If the space for you template already exists, then you should follow the tutorial: :ref: `uploading_to_existing_templates`.
+
+This tutorial assumes you have done all the steps in the preceding tutorial: :ref: `prerequisites_to_contributing`.
+
+**Note** at present, this tutorial will require writing access to the TemplateFlow repo.
+If you do not have access here, it may be best to open up an issue asking for a template space to be created.
+
+Step 1: create a new dataset
+=============================
+
+First make sure you are in your local templateflow directory.
+If you do not have a local templateflow copy, run:
+
+.. code-block:: bash
+
+    git clone https://github.com/templateflow/templateflow/
+    cd templateflow
+
+Now set the variable ``TEMPLATENAME`` to whatever your template will be called.
+Also set your Github username to the variable ``GITHUBUSERNAME``.
+Finally, write a description of your template.
+
+.. code-block:: bash
+
+    TEMPALTENAME='tpl-test'
+    GITHUBUSERNAME='yourusername'
+    TEMPALTEDESCRIPTION="This is a test template"
+    GITHUBREPO='templateflow'
+
+At the moment, always keep templateflow as GITHUBREPO, this may be changed in the future.
+
+With these variables set you can then run the following code with no modifications:
+
+.. code-block:: bash
+
+    datalad create -d . -D "$TEMPALTEDESCRIPTION" $TEMPALTENAME
+    cd $TEMPALTENAME
+    datalad create-sibling-github --github-organization $GITHUBREPO --github-login $GITHUBUSERNAME --access-protocol ssh $TEMPALTENAME
+    cd ..
+    sed -i -e "s/url = .\/$TEMPALTENAME/url = https:\/\/github.com\/$GITHUBREPO\/$TEMPLATENAME/g" .gitmodules
+    datalad save -m "set the github repo url for new template ``$TEMPLATENAME``"
+    datalad publish
+
+You will be asked to enter your github username and password while running this code.
+
+Explanation of above code
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+After running, this code will create an empty datalad folder called ``tpl-test`` (or whatever TEMPLATENAME is set to).
+It will then change into that directory, upload the template to github.
+It will then return to the templateflow directory.
+
+The 5th line in the above code edits the file .gitmodules to replace:
+
+.. code-block::
+
+    [submodule "tpl-test"]
+            path = tpl-test
+            url = ./tpl-test
+
+with:
+
+.. code-block::
+
+    [submodule "tpl-test"]
+            path = tpl-test
+            url = https://github.com/templateflow/tpl-test
+
+I.e. it adds a full url link to the.
+The final two lines upload this change.
+
+Step 2: Add a template_description.json
+========================================
+
+Within this directory we place a template_description.json which is needed in all templates.
+The json file contains the following:
+
+.. code-block:: json
+
+    {
+        "Authors": ["Noone"],
+        "Acknowledgements": "Curated and added to TemplateFlow by Thompson WH",
+        "BIDSVersion": "1.1.0",
+        "HowToAcknowledge": "You should not use this template",
+        "Identifier": "test",
+        "License": "See LICENSE file",
+        "Name": "A test template to for testing.",
+        "RRID": "SCR_002823",
+        "ReferencesAndLinks": [
+            "Link to article if there is one"],
+        "TemplateFlowVersion": "1.0.0",
+        "res": {
+            "01": {
+            "origin": [-91.0, -126.0, -72.0],
+            "shape": [182, 218, 182],
+            "zooms": [1.0, 1.0, 1.0]
+            }
+        }
+    }
+
+Add all the necessary information into the .json file.
+Then open a pull request on github to submit this information.

docs/contributing_tutorials/prerequisites_to_contributing.rst

@@ -0,0 +1,78 @@
+
+Prerequisites to contributing templates to TemplateFlow
+############################################################
+
+Who is this tutorial for?
+=================================
+
+Have a template that you would like to see on TemplateFlow? Great!
+Contributions are welcome.
+This document goes through some of the prerequisites needed to submit a template.
+Once you have these prerequisites achieved.
+
+Are you allowed to share the template?
+==========================================
+
+Templates have a licence which specifies the terms that they can be shared.
+TemplateFlow can only include templates that allow for redistribution.
+It is okay if the template requires attribution, but you need to make sure to add the attribution information.
+
+What type of contribution are you making?
+============================================
+
+There are three different types of contributions you can make to TemplateFlow.
+
+**A new template space**.
+This contribution involves adding a new space that does not currently exist.
+Let us say you have made a new pediatric space that you transform your images to; this would be a new template space.
+All the different MNI templates are each considered their own template space.
+Currently this requires writing permissions to the TemplateFlow repo.
+For now, if you do not have access open up an issue in the templateflow repo to say which template spaces should be added.
+
+**Nifti images within an existing template space**.
+This contribution involves adding to a template space that currently exists.
+An example of this would be adding a nifti file that is an atlas.
+You need to know which template space your atlas is in (Note: there are multiple MNI spaces).
+
+**Meta information**.
+This contribution involves additional information about existing templates.
+These will generally be in .json or .tsv files.
+There are also transform files which help translate between templates.
+
+There are tutorials for each of these three different types of contributions.
+
+Prerequisites for uploading nifti files
+=====================================================
+
+Aside from typical requirements (python3, git, pip), some prerequisites need addressing before you can upload a template (i.e. any image file).
+
+1. Getting access to the OSF repository. Create an issue stating you would like to access to the OSF repo. You need an account at: `osf.io <https://osf.io>`_
+2. Download the OSF client: ``pip install osfclient``.
+3. Install `DataLad <https://www.datalad.org>`_. See the `installation page <https://www.datalad.org/get_DataLad.html>`_ for more details.
+4. You also need TemplateFlows DataLad/osf tools. To install these type in a terminal: ``pip install git+https://github.com/TemplateFlow/DataLad-osf``.
+
+After installing these tools, you are ready to upload a template.
+
+If you only plan to contribute metadata (e.g. json files and tsv files), then these three steps are not needed.
+
+Initializing the TemplateFlow OSF directory
+==================================================
+
+Once you have the prerequisites set up, you can initialize the OSF directory onto your computer.
+
+In a new directory type:
+
+.. code-block:: bash
+
+    osf init
+
+This will prompt you for your username and TemplateFlow project number.
+This project number is ``ue5gx``.
+
+You can check that your directory has been correctly set up by typing:
+
+.. code-block:: bash
+
+    osf ls
+
+You should see the contents of the OSF project folder appear in the console.

docs/contributing_tutorials/uploading_to_existing_template.rst

@@ -0,0 +1,170 @@
+Uploading files to an existing template space
+###############################################
+
+Who is this tutorial for
+============================
+
+First, this is intended for those wishing to upload templates to TemplateFlow.
+Second, this is for people who want to add to a template directory that already exists.
+TemplateFlow consists of multiple templates sorted by the space the template is in.
+
+If the space for you template does not exist, then you should follow this tutorial: :ref: `adding_a_new_template`.
+
+This tutorial assumes you have done all the steps in the preceding tutorial: :ref: `prerequisites_to_contributing`.
+
+Big picture
+===============
+
+Image files (e.g. nifti files) are hosted on OSF.
+All other information (e.g. tsv, json files) will be hosted on github as regular files.
+
+When you use TemplateFlow, the images are only downloaded when they are needed.
+This can be done with DataLad, DataLad tracks a dataset and only downloads files when they are needed.
+You can read more about DataLad `here <https://www.datalad.org>`_.
+
+Thus, when uploading to an existing template, there are two different steps:
+
+1. You need to place them on the OSF server.
+2. you need to tell DataLad about the new items so the Github repo can track the new files.
+
+Step 1: Placing the template on the OSF server
+================================================
+
+Go to the TemplateFlow directory you previously initialized.
+Place the file you want to upload in directory and give them appropriate .
+
+Let us say you want to upload:
+`tpl-test/tpl-test_atlas-test1_dseg.nii.gz`, `tpl-test/tpl-test_atlas-test2_dseg.nii.gz`.
+
+.. code-block:: bash
+
+    export OSF_PASSWORD='<some-password>'
+    osf upload tpl-test_atlas-test1_dseg.nii.gz tpl-test_atlas-test1_dseg.nii.gz
+
+If you are overwriting an existing image, use the -f flag.
+
+If you want to upload multiple images (let us say we have multiple atlases), use a for bash loop:
+
+.. code-block:: bash
+
+    ls tpl-test/*_atlas-test*_dseg.nii.gz | parallel -j8 'osf upload {} {}'
+
+And this will iteratively upload all instances of the atlas to OSF.
+
+Step 2: Telling Datalad where the files are
+================================================
+
+Github is using Datalad to track the files.
+You need to tell Datalad where the new files are.
+
+To do this, run the following python script, changing the tqo required variables:
+
+.. code-block:: python
+
+    from pathlib import Path
+    from datalad_osf.utils import url_from_key, json_from_url,
+                                  get_osf_recursive, osf_to_csv
+
+    ## VARIABLES THAT NEED TO BE CHANGED
+
+    # THe template you are adding too
+    subset = 'tpl-test'
+    # A string that identifies your new nifit files
+    # Leave as blank if you want all files added
+    strinfile = 'atlas-test'
+
+    ## OPTIONAL VARIABLES
+
+    # Name of output file
+    output_filename = 'new_files.csv'
+    # templateflow project name
+    key = 'ue5gx'
+
+    ## Rest of script
+
+    toplevel = json_from_url(url_from_key(key))
+    for folder in toplevel['data']:
+        if folder['attributes']['name'] == subset:
+            url = folder['links']['move']
+            break
+
+    data = json_from_url(url)
+    hits = ['name,link']
+    for item in data['data']:
+        name = item['attributes']['name']
+        ext = ''.join(Path(name).suffixes)
+        if strinfile in name and '.nii' in ext:
+            print(item)
+            link = item['links']['download']
+            path = item['attributes']['materialized']
+            hits.append(','.join((name, link)))
+    Path(output_filename).write_text('\n'.join(hits))
+
+Either run this script interactively or save the script as a python file (e.g. 'get_datalad_urls.py')
+then run the file with `python get_datalad_urls.py`.
+
+Note, the above script assumes there are no subdirectories within the template folder.
+See end of tutorial for an example script when there are subdirectories within the template folder.
+
+This script will produce a file called new_files.csv.
+
+Finally, the contents of new_files.csv need to be uploaded via DataLad.
+
+To do this first, move the local image file into a tmp folder.
+
+.. code-block:: bash
+
+    mv tpl-test/*_atlas-test*.nii.gz ~/tmp/
+
+Then you add the new urls to DataLad. Add a message
+
+.. code-block:: bash
+
+    datalad addurls new_files.csv '{link}' '{name}' --message 'My test atlases'
+    datalad publish
+
+Example script when subdirectories are presents
+================================================
+
+.. code-block:: python
+
+    from pathlib import Path
+    from datalad_osf.utils import url_from_key, json_from_url, get_osf_recursive, osf_to_csv
+
+    ## VARIABLES THAT NEED TO BE CHANGED
+
+    # THe template you are adding too
+    subset = 'tpl-test'
+    # A string that identifies your new files
+    strinfile = 'atlas-test'
+
+    ## OPTIONAL VARIABLES
+
+    # Name of output file
+    output_filename = 'new_files.csv'
+    # templateflow project name
+    key = 'ue5gx'
+
+    ## REST OF SCRIPT
+
+    toplevel = json_from_url(url_from_key(key))
+    for folder in toplevel['data']:
+        if folder['attributes']['name'] == subset:
+            url = folder['links']['move']
+            break
+
+    data = json_from_url(url)
+    hits = ['name,link']
+    for item in data['data']:
+        if item['attributes']['kind'] == 'folder':
+            subdata = json_from_url(item['links']['move'])
+            for subitem in subdata['data']:
+                if subitem['attributes']['kind'] == 'file':
+                    name = subitem['attributes']['name']
+                    ext = ''.join(Path(name).suffixes)
+                    if strinfile in name and '.nii' in ext:
+                        print(name)
+                        link = subitem['links']['download']
+                        path = subitem['attributes']['materialized']
+                        hits.append(','.join((name, link)))
+    Path(output_filename).write_text('\n'.join(hits))