From 7c2f652c30117139c34d5144a001d15bfc3f0dde Mon Sep 17 00:00:00 2001 From: wiheto Date: Tue, 26 Nov 2019 10:50:59 -0800 Subject: [PATCH 1/4] draft 1 of documentation --- .../adding_a_new_template.rst | 116 +++++++++++++ .../prerequisites_to_contributing.rst | 72 ++++++++ .../uploading_to_existing_template.rst | 162 ++++++++++++++++++ docs/templateflow_naming.rst | 57 ++++++ 4 files changed, 407 insertions(+) create mode 100644 docs/contributing_tutorials/adding_a_new_template.rst create mode 100644 docs/contributing_tutorials/prerequisites_to_contributing.rst create mode 100644 docs/contributing_tutorials/uploading_to_existing_template.rst create mode 100644 docs/templateflow_naming.rst diff --git a/docs/contributing_tutorials/adding_a_new_template.rst b/docs/contributing_tutorials/adding_a_new_template.rst new file mode 100644 index 00000000..a6ac8dfd --- /dev/null +++ b/docs/contributing_tutorials/adding_a_new_template.rst @@ -0,0 +1,116 @@ +Creating a new template space +############################################### + +Who is this tutorial for +============================ + +First, this is intended for those wishing to add templates to TemplateFlow. +Second, this is for people who want to add a template directory that does not already exists. +TemplateFlow consists of multiple templates sorted by the space the template is in. +This tutorial tells you how to add a new template space. + +If the space for you template already exists, then you should follow the tutorial: :ref: `uploading_to_existing_templates`. + +This tutorial assumes you have done all the steps in the preceding tutorial: :ref: `prerequisites_to_contributing`. + +__Note__ at present, this tutorial will require writing access to the TemplateFlow repo. +If you do not have access here, it may be best to open up an issue asking for a template space to be created. + +Big picture +=============== + +Image files (e.g. nifti files) are hosted on OSF and linked to Github. + +You will need to publish the directory to OSF and create a link to it to Github. +This link can be done with DataLad, DataLad tracks a dataset and only downloads files when they are needed. +You can read more about DataLad [`Link `_]. + +Step 1: create a new dataset +============================= + +First make sure you are in your local templateflow directory. +If you do not have a local templateflow copy, run: + +::. + git clone https://github.com/templateflow/templateflow/ + cd templateflow + +Now set the variable ``TEMPLATENAME`` to whatever your template will be called. +Also set your Github username to the variable ``GITHUBUSERNAME``. +Finally, write a description of your template. + +::. + TEMPALTENAME='tpl-test' + GITHUBUSERNAME='yourusername' + TEMPALTEDESCRIPTION="This is a test template" + GITHUBREPO='templateflow' + +At the moment, always keep templateflow as GITHUBREPO, this may be changed in the future + +With these variables set you can then run the following code with no modifications: + +::. + datalad create -d . -D "$TEMPALTEDESCRIPTION" $TEMPALTENAME + cd $TEMPALTENAME + datalad create-sibling-github --github-organization $GITHUBREPO --github-login $GITHUBUSERNAME --access-protocol ssh $TEMPALTENAME + cd .. + sed -i -e "s/url = .\/$TEMPALTENAME/url = https:\/\/github.com\/$GITHUBREPO\/$TEMPLATENAME/g" .gitmodules + datalad save -m "set the github repo url for new template ``$TEMPLATENAME``" + datalad publish + +You will be asked to enter your github username and password while running this code. + +Explanation of above code +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +After running, this code will create an empty datalad folder called ``tpl-test`` (or whatever TEMPLATENAME is set to). +It will then change into that directory, upload the template to github. +It will then return to the templateflow directory. + +The 5th line in the above code edits the file .gitmodules to replace: + +::. + [submodule "tpl-test"] + path = tpl-test + url = ./tpl-test + +with: + +::. + [submodule "tpl-test"] + path = tpl-test + url = https://github.com/templateflow/tpl-test + +I.e. it adds a full url link to the. +The final two lines upload this change. + +Step 2: Add a template_description.json +======================================== + +Within this directory we place a template_description.json which is needed in all templates. +The json file contains the following: + +::. + { + "Authors": ["Noone"], + "Acknowledgements": "Curated and added to TemplateFlow by Thompson WH", + "BIDSVersion": "1.1.0", + "HowToAcknowledge": "You should not use this template", + "Identifier": "test", + "License": "See LICENSE file", + "Name": "A test template to for testing.", + "RRID": "SCR_002823", + "ReferencesAndLinks": [ + "Link to article if there is one"], + "TemplateFlowVersion": "1.0.0", + "res": { + "01": { + "origin": [-91.0, -126.0, -72.0], + "shape": [182, 218, 182], + "zooms": [1.0, 1.0, 1.0] + } + } + } + +Add all the necessary information into the .json file. +Then open a pull request on github to submit this information. diff --git a/docs/contributing_tutorials/prerequisites_to_contributing.rst b/docs/contributing_tutorials/prerequisites_to_contributing.rst new file mode 100644 index 00000000..13ffed06 --- /dev/null +++ b/docs/contributing_tutorials/prerequisites_to_contributing.rst @@ -0,0 +1,72 @@ + +Prerequisites to contributing templates to TemplateFlow +############################################################ + +Who is this tutorial for? +================================= + +Have a template that you would like to see on TemplateFlow? Great! +Contributions are welcome. +This document goes through some of the prerequisites needed to submit a template. +Once you have these prerequisites achieved. + +Are you allowed to share the template? +========================================== + +Templates have a licence which specifies the terms that they can be shared. +TemplateFlow can only include templates that allow for redistribution. +It is okay if the template requires attribution, but you need to make sure to add the attribution information. + +What type of contribution are you making? +============================================ + +There are three different types of contributions you can make to TemplateFlow. + +**A new template space**. +This contribution involves adding a new space that does not currently exist. +Let us say you have made a new pediatric space that you transform your images to; this would be a new template space. +All the different MNI templates are each considered their own template space. + +**Nifti images within an existing template space**. +This contribution involves adding to a template space that currently exists. +An example of this would be adding a nifti file that is an atlas. +You need to know which template space your atlas is in (Note: there are multiple MNI spaces). + +**Meta information**. +This contribution involves additional information about existing templates. +These will generally be in .json or .tsv files. +There are also transform files which help translate between templates. + +There are tutorials for each of these three different types of contributions. + +Prerequisites for uploading nifti files +===================================================== + +Aside from typical requirements (python3, git, pip), some prerequisites need addressing before you can upload a template (i.e. any image file). + +1. Getting access to the OSF repository. Create an issue stating you would like to access to the OSF repo. You need an account at: `osf.io `_ +2. Download the OSF client: ``pip install osfclient``. +3. Install `DataLad `_. See the `installation page `_ for more details. +4. You also need TemplateFlows DataLad/osf tools. To install these type in a terminal: ``pip install git+https://github.com/TemplateFlow/DataLad-osf``. + +After installing these tools, you are ready to upload a template. + +If you only plan to contribute metadata (e.g. json files and tsv files), then these three steps are not needed. + +Initializing the TemplateFlow OSF directory +================================================== + +Once you have the prerequisites set up, you can initialize the OSF directory onto your computer. + +In a new directory type: + +> osf init + +This will prompt you for your username and TemplateFlow project number. +This project number is ``ue5gx``. + +You can check that your directory has been correctly set up by typing: + +> osf ls + +You should see the contents of the OSF project folder appear in the console. diff --git a/docs/contributing_tutorials/uploading_to_existing_template.rst b/docs/contributing_tutorials/uploading_to_existing_template.rst new file mode 100644 index 00000000..0b5a59c5 --- /dev/null +++ b/docs/contributing_tutorials/uploading_to_existing_template.rst @@ -0,0 +1,162 @@ +Uploading files to an existing template space +############################################### + +Who is this tutorial for +============================ + +First, this is intended for those wishing to upload templates to TemplateFlow. +Second, this is for people who want to add to a template directory that already exists. +TemplateFlow consists of multiple templates sorted by the space the template is in. + +If the space for you template does not exist, then you should follow this tutorial: :ref: `adding_a_new_template`. + +This tutorial assumes you have done all the steps in the preceding tutorial: :ref: `prerequisites_to_contributing`. + +Big picture +=============== + +Image files (e.g. nifti files) are hosted on OSF. +All other information (e.g. tsv, json files) will be hosted on github as regular files. + +When you use TemplateFlow, the images are only downloaded when they are needed. +This can be done with DataLad, DataLad tracks a dataset and only downloads files when they are needed. +You can read more about DataLad [`Link `_]. + +Thus, when uploading to an existing template, there are two different steps: + +1. You need to place them on the OSF server. +2. you need to tell DataLad about the new items so the Github repo can track the new files. + +Step 1: Placing the template on the OSF server +================================================ + +Go to the TemplateFlow directory you previously initialized. +Place the file you want to upload in directory and give them appropriate . + +Let us say you want to upload: +`tpl-test/tpl-test_atlas-test1_dseg.nii.gz`, `tpl-test/tpl-test_atlas-test2_dseg.nii.gz`. + +>>> export OSF_PASSWORD='' +>>> osf upload tpl-test_atlas-test1_dseg.nii.gz tpl-test_atlas-test1_dseg.nii.gz + +If you are overwriting an existing image, use the -f flag. + +If you want to upload multiple images (let us say we have multiple atlases), use a for bash loop: + +>>> ls tpl-test/*_atlas-test*_dseg.nii.gz | parallel -j8 'osf upload {} {}' + +And this will iteratively upload all instances of the atlas to OSF. + +Step 2: Telling Datalad where the files are +================================================ + +Github is using Datalad to track the files. +You need to tell Datalad where the new files are. + +To do this, run the following python script, changing the tqo required variables: + +.. code-block:: python + + from pathlib import Path + from datalad_osf.utils import url_from_key, json_from_url, + get_osf_recursive, osf_to_csv + + ## VARIABLES THAT NEED TO BE CHANGED + + # THe template you are adding too + subset = 'tpl-test' + # A string that identifies your new nifit files + # Leave as blank if you want all files added + strinfile = 'atlas-test' + + ## OPTIONAL VARIABLES + + # Name of output file + output_filename = 'new_files.csv' + # templateflow project name + key = 'ue5gx' + + ## Rest of script + + toplevel = json_from_url(url_from_key(key)) + for folder in toplevel['data']: + if folder['attributes']['name'] == subset: + url = folder['links']['move'] + break + + data = json_from_url(url) + hits = ['name,link'] + for item in data['data']: + name = item['attributes']['name'] + ext = ''.join(Path(name).suffixes) + if strinfile in name and '.nii' in ext: + print(item) + link = item['links']['download'] + path = item['attributes']['materialized'] + hits.append(','.join((name, link))) + Path(output_filename).write_text('\n'.join(hits)) + +Either run this script interactively or save the script as a python file (e.g. 'get_datalad_urls.py') +then run the file with `python get_datalad_urls.py`. + +Note, the above script assumes there are no subdirectories within the template folder. +See end of tutorial for an example script when there are subdirectories within the template folder. + +This script will produce a file called new_files.csv. + +Finally, the contents of new_files.csv need to be uploaded via DataLad. + +To do this first, move the local image file into a tmp folder. + +> mv tpl-test/*_atlas-test*.nii.gz ~/tmp/ + +Then you add the new urls to DataLad. Add a message + +> datalad addurls new_files.csv '{link}' '{name}' --message 'My test atlases' +> datalad publish + +Example script when subdirectories are presents +================================================ + +.. code-block:: python + + from pathlib import Path + from datalad_osf.utils import url_from_key, json_from_url, get_osf_recursive, osf_to_csv + + ## VARIABLES THAT NEED TO BE CHANGED + + # THe template you are adding too + subset = 'tpl-test' + # A string that identifies your new files + strinfile = 'atlas-test' + + ## OPTIONAL VARIABLES + + # Name of output file + output_filename = 'new_files.csv' + # templateflow project name + key = 'ue5gx' + + ## REST OF SCRIPT + + toplevel = json_from_url(url_from_key(key)) + for folder in toplevel['data']: + if folder['attributes']['name'] == subset: + url = folder['links']['move'] + break + + data = json_from_url(url) + hits = ['name,link'] + for item in data['data']: + if item['attributes']['kind'] == 'folder': + subdata = json_from_url(item['links']['move']) + for subitem in subdata['data']: + if subitem['attributes']['kind'] == 'file': + name = subitem['attributes']['name'] + ext = ''.join(Path(name).suffixes) + if strinfile in name and '.nii' in ext: + print(name) + link = subitem['links']['download'] + path = subitem['attributes']['materialized'] + hits.append(','.join((name, link))) + Path(output_filename).write_text('\n'.join(hits)) \ No newline at end of file diff --git a/docs/templateflow_naming.rst b/docs/templateflow_naming.rst new file mode 100644 index 00000000..90d1395b --- /dev/null +++ b/docs/templateflow_naming.rst @@ -0,0 +1,57 @@ + +TemplateFlow Naming +=========================== + +Who is this tutorial for? +================================= + +Anyone trying to understand the naming structure in TemplateFlow. + +Naming information +=========================== + +TemplateFlow generally follows the `BIDS naming structure `_, but has a few deviations (e.g. the tpl key). +Here we outline the most common names that are found in TemplateFlow. + +Common key names using in TemplateFlow: + +===== ===== +Key Description +===== ===== +tpl The space of the template. +res Resolution index. See ``template_description.json`` within each template for more information about what the index specifies. +atlas Name of an atlas. +desc Additional information about the file to differentiate it from other files. +===== ===== + +Common suffixes used in TemplateFlow: + +======= ============ +Suffix Description +======= ============ +dseg discrete segmentation +pseg probability segmentation +mask binary mask +xfm transform file +T2w T2 weighted image +T1w T1 weighted image +======= ============ + +Common file-formats used in TemplateFlow: + +========== ======= +Extension Description +========== ======= +.nii.gz Image +.tsv Tabular information +.json Meta-information +.h5 Transform file +========== ======= + +Thus a template with the following name: + +``tpl-test_res-01_atlas-myatlas_desc-200nodes_dseg.nii.gz`` + +This would be a nifti image containing discrete segmentation of "myatlas" which contains 200nodes. +And it is in the template space "test". +The resolution information will be found in the dataset_description.json file under '01'. From fc389711efd0a8db9a81a5195f49d4b53f9c808b Mon Sep 17 00:00:00 2001 From: wiheto Date: Tue, 26 Nov 2019 10:59:23 -0800 Subject: [PATCH 2/4] formatting --- .../adding_a_new_template.rst | 21 ++++++++++++------- .../prerequisites_to_contributing.rst | 10 +++++++-- .../uploading_to_existing_template.rst | 10 ++++++--- 3 files changed, 28 insertions(+), 13 deletions(-) diff --git a/docs/contributing_tutorials/adding_a_new_template.rst b/docs/contributing_tutorials/adding_a_new_template.rst index a6ac8dfd..efc67e1c 100644 --- a/docs/contributing_tutorials/adding_a_new_template.rst +++ b/docs/contributing_tutorials/adding_a_new_template.rst @@ -13,7 +13,7 @@ If the space for you template already exists, then you should follow the tutoria This tutorial assumes you have done all the steps in the preceding tutorial: :ref: `prerequisites_to_contributing`. -__Note__ at present, this tutorial will require writing access to the TemplateFlow repo. +**Note** at present, this tutorial will require writing access to the TemplateFlow repo. If you do not have access here, it may be best to open up an issue asking for a template space to be created. Big picture @@ -31,7 +31,8 @@ Step 1: create a new dataset First make sure you are in your local templateflow directory. If you do not have a local templateflow copy, run: -::. +.. code-block:: bash + git clone https://github.com/templateflow/templateflow/ cd templateflow @@ -39,17 +40,19 @@ Now set the variable ``TEMPLATENAME`` to whatever your template will be called. Also set your Github username to the variable ``GITHUBUSERNAME``. Finally, write a description of your template. -::. +.. code-block:: bash + TEMPALTENAME='tpl-test' GITHUBUSERNAME='yourusername' TEMPALTEDESCRIPTION="This is a test template" GITHUBREPO='templateflow' -At the moment, always keep templateflow as GITHUBREPO, this may be changed in the future +At the moment, always keep templateflow as GITHUBREPO, this may be changed in the future. With these variables set you can then run the following code with no modifications: -::. +.. code-block:: bash + datalad create -d . -D "$TEMPALTEDESCRIPTION" $TEMPALTENAME cd $TEMPALTENAME datalad create-sibling-github --github-organization $GITHUBREPO --github-login $GITHUBUSERNAME --access-protocol ssh $TEMPALTENAME @@ -69,14 +72,16 @@ It will then return to the templateflow directory. The 5th line in the above code edits the file .gitmodules to replace: -::. +.. code-block:: + [submodule "tpl-test"] path = tpl-test url = ./tpl-test with: -::. +.. code-block:: + [submodule "tpl-test"] path = tpl-test url = https://github.com/templateflow/tpl-test @@ -90,7 +95,7 @@ Step 2: Add a template_description.json Within this directory we place a template_description.json which is needed in all templates. The json file contains the following: -::. +.. code-block:: json { "Authors": ["Noone"], "Acknowledgements": "Curated and added to TemplateFlow by Thompson WH", diff --git a/docs/contributing_tutorials/prerequisites_to_contributing.rst b/docs/contributing_tutorials/prerequisites_to_contributing.rst index 13ffed06..2e539d0f 100644 --- a/docs/contributing_tutorials/prerequisites_to_contributing.rst +++ b/docs/contributing_tutorials/prerequisites_to_contributing.rst @@ -26,6 +26,8 @@ There are three different types of contributions you can make to TemplateFlow. This contribution involves adding a new space that does not currently exist. Let us say you have made a new pediatric space that you transform your images to; this would be a new template space. All the different MNI templates are each considered their own template space. +Currently this requires writing permissions to the TemplateFlow repo. +For now, if you do not have access open up an issue in the templateflow repo to say which template spaces should be added. **Nifti images within an existing template space**. This contribution involves adding to a template space that currently exists. @@ -60,13 +62,17 @@ Once you have the prerequisites set up, you can initialize the OSF directory ont In a new directory type: -> osf init +.. code-clock:: bash + + osf init This will prompt you for your username and TemplateFlow project number. This project number is ``ue5gx``. You can check that your directory has been correctly set up by typing: -> osf ls +.. code-clock:: bash + + osf ls You should see the contents of the OSF project folder appear in the console. diff --git a/docs/contributing_tutorials/uploading_to_existing_template.rst b/docs/contributing_tutorials/uploading_to_existing_template.rst index 0b5a59c5..216eeea4 100644 --- a/docs/contributing_tutorials/uploading_to_existing_template.rst +++ b/docs/contributing_tutorials/uploading_to_existing_template.rst @@ -108,12 +108,16 @@ Finally, the contents of new_files.csv need to be uploaded via DataLad. To do this first, move the local image file into a tmp folder. -> mv tpl-test/*_atlas-test*.nii.gz ~/tmp/ +.. code-block:: bash + + mv tpl-test/*_atlas-test*.nii.gz ~/tmp/ Then you add the new urls to DataLad. Add a message -> datalad addurls new_files.csv '{link}' '{name}' --message 'My test atlases' -> datalad publish +.. code-block:: bash + + datalad addurls new_files.csv '{link}' '{name}' --message 'My test atlases' + datalad publish Example script when subdirectories are presents ================================================ From c31f9ca9a9c3898b478045102029c2019a8ec5a5 Mon Sep 17 00:00:00 2001 From: wiheto Date: Tue, 26 Nov 2019 11:04:47 -0800 Subject: [PATCH 3/4] improving codeblocks --- .../contributing_tutorials/adding_a_new_template.rst | 10 +--------- .../uploading_to_existing_template.rst | 12 ++++++++---- 2 files changed, 9 insertions(+), 13 deletions(-) diff --git a/docs/contributing_tutorials/adding_a_new_template.rst b/docs/contributing_tutorials/adding_a_new_template.rst index efc67e1c..80b3effd 100644 --- a/docs/contributing_tutorials/adding_a_new_template.rst +++ b/docs/contributing_tutorials/adding_a_new_template.rst @@ -16,15 +16,6 @@ This tutorial assumes you have done all the steps in the preceding tutorial: :re **Note** at present, this tutorial will require writing access to the TemplateFlow repo. If you do not have access here, it may be best to open up an issue asking for a template space to be created. -Big picture -=============== - -Image files (e.g. nifti files) are hosted on OSF and linked to Github. - -You will need to publish the directory to OSF and create a link to it to Github. -This link can be done with DataLad, DataLad tracks a dataset and only downloads files when they are needed. -You can read more about DataLad [`Link `_]. - Step 1: create a new dataset ============================= @@ -96,6 +87,7 @@ Within this directory we place a template_description.json which is needed in al The json file contains the following: .. code-block:: json + { "Authors": ["Noone"], "Acknowledgements": "Curated and added to TemplateFlow by Thompson WH", diff --git a/docs/contributing_tutorials/uploading_to_existing_template.rst b/docs/contributing_tutorials/uploading_to_existing_template.rst index 216eeea4..c2d34ea0 100644 --- a/docs/contributing_tutorials/uploading_to_existing_template.rst +++ b/docs/contributing_tutorials/uploading_to_existing_template.rst @@ -20,7 +20,7 @@ All other information (e.g. tsv, json files) will be hosted on github as regular When you use TemplateFlow, the images are only downloaded when they are needed. This can be done with DataLad, DataLad tracks a dataset and only downloads files when they are needed. -You can read more about DataLad [`Link `_]. +You can read more about DataLad `here `_. Thus, when uploading to an existing template, there are two different steps: @@ -36,14 +36,18 @@ Place the file you want to upload in directory and give them appropriate . Let us say you want to upload: `tpl-test/tpl-test_atlas-test1_dseg.nii.gz`, `tpl-test/tpl-test_atlas-test2_dseg.nii.gz`. ->>> export OSF_PASSWORD='' ->>> osf upload tpl-test_atlas-test1_dseg.nii.gz tpl-test_atlas-test1_dseg.nii.gz +.. code-block:: bash + + export OSF_PASSWORD='' + osf upload tpl-test_atlas-test1_dseg.nii.gz tpl-test_atlas-test1_dseg.nii.gz If you are overwriting an existing image, use the -f flag. If you want to upload multiple images (let us say we have multiple atlases), use a for bash loop: ->>> ls tpl-test/*_atlas-test*_dseg.nii.gz | parallel -j8 'osf upload {} {}' +.. code-block:: bash + + ls tpl-test/*_atlas-test*_dseg.nii.gz | parallel -j8 'osf upload {} {}' And this will iteratively upload all instances of the atlas to OSF. From 5461e58e443e8c4379094ea241880a07fc89aa74 Mon Sep 17 00:00:00 2001 From: wiheto Date: Tue, 26 Nov 2019 11:06:10 -0800 Subject: [PATCH 4/4] fixing misspelt codeblock --- docs/contributing_tutorials/prerequisites_to_contributing.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/contributing_tutorials/prerequisites_to_contributing.rst b/docs/contributing_tutorials/prerequisites_to_contributing.rst index 2e539d0f..287c5490 100644 --- a/docs/contributing_tutorials/prerequisites_to_contributing.rst +++ b/docs/contributing_tutorials/prerequisites_to_contributing.rst @@ -62,7 +62,7 @@ Once you have the prerequisites set up, you can initialize the OSF directory ont In a new directory type: -.. code-clock:: bash +.. code-block:: bash osf init @@ -71,7 +71,7 @@ This project number is ``ue5gx``. You can check that your directory has been correctly set up by typing: -.. code-clock:: bash +.. code-block:: bash osf ls