diff --git a/docs/contributing_tutorials/adding_a_new_template.rst b/docs/contributing_tutorials/adding_a_new_template.rst new file mode 100644 index 00000000..80b3effd --- /dev/null +++ b/docs/contributing_tutorials/adding_a_new_template.rst @@ -0,0 +1,113 @@ +Creating a new template space +############################################### + +Who is this tutorial for +============================ + +First, this is intended for those wishing to add templates to TemplateFlow. +Second, this is for people who want to add a template directory that does not already exists. +TemplateFlow consists of multiple templates sorted by the space the template is in. +This tutorial tells you how to add a new template space. + +If the space for you template already exists, then you should follow the tutorial: :ref: `uploading_to_existing_templates`. + +This tutorial assumes you have done all the steps in the preceding tutorial: :ref: `prerequisites_to_contributing`. + +**Note** at present, this tutorial will require writing access to the TemplateFlow repo. +If you do not have access here, it may be best to open up an issue asking for a template space to be created. + +Step 1: create a new dataset +============================= + +First make sure you are in your local templateflow directory. +If you do not have a local templateflow copy, run: + +.. code-block:: bash + + git clone https://github.com/templateflow/templateflow/ + cd templateflow + +Now set the variable ``TEMPLATENAME`` to whatever your template will be called. +Also set your Github username to the variable ``GITHUBUSERNAME``. +Finally, write a description of your template. + +.. code-block:: bash + + TEMPALTENAME='tpl-test' + GITHUBUSERNAME='yourusername' + TEMPALTEDESCRIPTION="This is a test template" + GITHUBREPO='templateflow' + +At the moment, always keep templateflow as GITHUBREPO, this may be changed in the future. + +With these variables set you can then run the following code with no modifications: + +.. code-block:: bash + + datalad create -d . -D "$TEMPALTEDESCRIPTION" $TEMPALTENAME + cd $TEMPALTENAME + datalad create-sibling-github --github-organization $GITHUBREPO --github-login $GITHUBUSERNAME --access-protocol ssh $TEMPALTENAME + cd .. + sed -i -e "s/url = .\/$TEMPALTENAME/url = https:\/\/github.com\/$GITHUBREPO\/$TEMPLATENAME/g" .gitmodules + datalad save -m "set the github repo url for new template ``$TEMPLATENAME``" + datalad publish + +You will be asked to enter your github username and password while running this code. + +Explanation of above code +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +After running, this code will create an empty datalad folder called ``tpl-test`` (or whatever TEMPLATENAME is set to). +It will then change into that directory, upload the template to github. +It will then return to the templateflow directory. + +The 5th line in the above code edits the file .gitmodules to replace: + +.. code-block:: + + [submodule "tpl-test"] + path = tpl-test + url = ./tpl-test + +with: + +.. code-block:: + + [submodule "tpl-test"] + path = tpl-test + url = https://github.com/templateflow/tpl-test + +I.e. it adds a full url link to the. +The final two lines upload this change. + +Step 2: Add a template_description.json +======================================== + +Within this directory we place a template_description.json which is needed in all templates. +The json file contains the following: + +.. code-block:: json + + { + "Authors": ["Noone"], + "Acknowledgements": "Curated and added to TemplateFlow by Thompson WH", + "BIDSVersion": "1.1.0", + "HowToAcknowledge": "You should not use this template", + "Identifier": "test", + "License": "See LICENSE file", + "Name": "A test template to for testing.", + "RRID": "SCR_002823", + "ReferencesAndLinks": [ + "Link to article if there is one"], + "TemplateFlowVersion": "1.0.0", + "res": { + "01": { + "origin": [-91.0, -126.0, -72.0], + "shape": [182, 218, 182], + "zooms": [1.0, 1.0, 1.0] + } + } + } + +Add all the necessary information into the .json file. +Then open a pull request on github to submit this information. diff --git a/docs/contributing_tutorials/prerequisites_to_contributing.rst b/docs/contributing_tutorials/prerequisites_to_contributing.rst new file mode 100644 index 00000000..287c5490 --- /dev/null +++ b/docs/contributing_tutorials/prerequisites_to_contributing.rst @@ -0,0 +1,78 @@ + +Prerequisites to contributing templates to TemplateFlow +############################################################ + +Who is this tutorial for? +================================= + +Have a template that you would like to see on TemplateFlow? Great! +Contributions are welcome. +This document goes through some of the prerequisites needed to submit a template. +Once you have these prerequisites achieved. + +Are you allowed to share the template? +========================================== + +Templates have a licence which specifies the terms that they can be shared. +TemplateFlow can only include templates that allow for redistribution. +It is okay if the template requires attribution, but you need to make sure to add the attribution information. + +What type of contribution are you making? +============================================ + +There are three different types of contributions you can make to TemplateFlow. + +**A new template space**. +This contribution involves adding a new space that does not currently exist. +Let us say you have made a new pediatric space that you transform your images to; this would be a new template space. +All the different MNI templates are each considered their own template space. +Currently this requires writing permissions to the TemplateFlow repo. +For now, if you do not have access open up an issue in the templateflow repo to say which template spaces should be added. + +**Nifti images within an existing template space**. +This contribution involves adding to a template space that currently exists. +An example of this would be adding a nifti file that is an atlas. +You need to know which template space your atlas is in (Note: there are multiple MNI spaces). + +**Meta information**. +This contribution involves additional information about existing templates. +These will generally be in .json or .tsv files. +There are also transform files which help translate between templates. + +There are tutorials for each of these three different types of contributions. + +Prerequisites for uploading nifti files +===================================================== + +Aside from typical requirements (python3, git, pip), some prerequisites need addressing before you can upload a template (i.e. any image file). + +1. Getting access to the OSF repository. Create an issue stating you would like to access to the OSF repo. You need an account at: `osf.io `_ +2. Download the OSF client: ``pip install osfclient``. +3. Install `DataLad `_. See the `installation page `_ for more details. +4. You also need TemplateFlows DataLad/osf tools. To install these type in a terminal: ``pip install git+https://github.com/TemplateFlow/DataLad-osf``. + +After installing these tools, you are ready to upload a template. + +If you only plan to contribute metadata (e.g. json files and tsv files), then these three steps are not needed. + +Initializing the TemplateFlow OSF directory +================================================== + +Once you have the prerequisites set up, you can initialize the OSF directory onto your computer. + +In a new directory type: + +.. code-block:: bash + + osf init + +This will prompt you for your username and TemplateFlow project number. +This project number is ``ue5gx``. + +You can check that your directory has been correctly set up by typing: + +.. code-block:: bash + + osf ls + +You should see the contents of the OSF project folder appear in the console. diff --git a/docs/contributing_tutorials/uploading_to_existing_template.rst b/docs/contributing_tutorials/uploading_to_existing_template.rst new file mode 100644 index 00000000..c2d34ea0 --- /dev/null +++ b/docs/contributing_tutorials/uploading_to_existing_template.rst @@ -0,0 +1,170 @@ +Uploading files to an existing template space +############################################### + +Who is this tutorial for +============================ + +First, this is intended for those wishing to upload templates to TemplateFlow. +Second, this is for people who want to add to a template directory that already exists. +TemplateFlow consists of multiple templates sorted by the space the template is in. + +If the space for you template does not exist, then you should follow this tutorial: :ref: `adding_a_new_template`. + +This tutorial assumes you have done all the steps in the preceding tutorial: :ref: `prerequisites_to_contributing`. + +Big picture +=============== + +Image files (e.g. nifti files) are hosted on OSF. +All other information (e.g. tsv, json files) will be hosted on github as regular files. + +When you use TemplateFlow, the images are only downloaded when they are needed. +This can be done with DataLad, DataLad tracks a dataset and only downloads files when they are needed. +You can read more about DataLad `here `_. + +Thus, when uploading to an existing template, there are two different steps: + +1. You need to place them on the OSF server. +2. you need to tell DataLad about the new items so the Github repo can track the new files. + +Step 1: Placing the template on the OSF server +================================================ + +Go to the TemplateFlow directory you previously initialized. +Place the file you want to upload in directory and give them appropriate . + +Let us say you want to upload: +`tpl-test/tpl-test_atlas-test1_dseg.nii.gz`, `tpl-test/tpl-test_atlas-test2_dseg.nii.gz`. + +.. code-block:: bash + + export OSF_PASSWORD='' + osf upload tpl-test_atlas-test1_dseg.nii.gz tpl-test_atlas-test1_dseg.nii.gz + +If you are overwriting an existing image, use the -f flag. + +If you want to upload multiple images (let us say we have multiple atlases), use a for bash loop: + +.. code-block:: bash + + ls tpl-test/*_atlas-test*_dseg.nii.gz | parallel -j8 'osf upload {} {}' + +And this will iteratively upload all instances of the atlas to OSF. + +Step 2: Telling Datalad where the files are +================================================ + +Github is using Datalad to track the files. +You need to tell Datalad where the new files are. + +To do this, run the following python script, changing the tqo required variables: + +.. code-block:: python + + from pathlib import Path + from datalad_osf.utils import url_from_key, json_from_url, + get_osf_recursive, osf_to_csv + + ## VARIABLES THAT NEED TO BE CHANGED + + # THe template you are adding too + subset = 'tpl-test' + # A string that identifies your new nifit files + # Leave as blank if you want all files added + strinfile = 'atlas-test' + + ## OPTIONAL VARIABLES + + # Name of output file + output_filename = 'new_files.csv' + # templateflow project name + key = 'ue5gx' + + ## Rest of script + + toplevel = json_from_url(url_from_key(key)) + for folder in toplevel['data']: + if folder['attributes']['name'] == subset: + url = folder['links']['move'] + break + + data = json_from_url(url) + hits = ['name,link'] + for item in data['data']: + name = item['attributes']['name'] + ext = ''.join(Path(name).suffixes) + if strinfile in name and '.nii' in ext: + print(item) + link = item['links']['download'] + path = item['attributes']['materialized'] + hits.append(','.join((name, link))) + Path(output_filename).write_text('\n'.join(hits)) + +Either run this script interactively or save the script as a python file (e.g. 'get_datalad_urls.py') +then run the file with `python get_datalad_urls.py`. + +Note, the above script assumes there are no subdirectories within the template folder. +See end of tutorial for an example script when there are subdirectories within the template folder. + +This script will produce a file called new_files.csv. + +Finally, the contents of new_files.csv need to be uploaded via DataLad. + +To do this first, move the local image file into a tmp folder. + +.. code-block:: bash + + mv tpl-test/*_atlas-test*.nii.gz ~/tmp/ + +Then you add the new urls to DataLad. Add a message + +.. code-block:: bash + + datalad addurls new_files.csv '{link}' '{name}' --message 'My test atlases' + datalad publish + +Example script when subdirectories are presents +================================================ + +.. code-block:: python + + from pathlib import Path + from datalad_osf.utils import url_from_key, json_from_url, get_osf_recursive, osf_to_csv + + ## VARIABLES THAT NEED TO BE CHANGED + + # THe template you are adding too + subset = 'tpl-test' + # A string that identifies your new files + strinfile = 'atlas-test' + + ## OPTIONAL VARIABLES + + # Name of output file + output_filename = 'new_files.csv' + # templateflow project name + key = 'ue5gx' + + ## REST OF SCRIPT + + toplevel = json_from_url(url_from_key(key)) + for folder in toplevel['data']: + if folder['attributes']['name'] == subset: + url = folder['links']['move'] + break + + data = json_from_url(url) + hits = ['name,link'] + for item in data['data']: + if item['attributes']['kind'] == 'folder': + subdata = json_from_url(item['links']['move']) + for subitem in subdata['data']: + if subitem['attributes']['kind'] == 'file': + name = subitem['attributes']['name'] + ext = ''.join(Path(name).suffixes) + if strinfile in name and '.nii' in ext: + print(name) + link = subitem['links']['download'] + path = subitem['attributes']['materialized'] + hits.append(','.join((name, link))) + Path(output_filename).write_text('\n'.join(hits)) \ No newline at end of file diff --git a/docs/templateflow_naming.rst b/docs/templateflow_naming.rst new file mode 100644 index 00000000..90d1395b --- /dev/null +++ b/docs/templateflow_naming.rst @@ -0,0 +1,57 @@ + +TemplateFlow Naming +=========================== + +Who is this tutorial for? +================================= + +Anyone trying to understand the naming structure in TemplateFlow. + +Naming information +=========================== + +TemplateFlow generally follows the `BIDS naming structure `_, but has a few deviations (e.g. the tpl key). +Here we outline the most common names that are found in TemplateFlow. + +Common key names using in TemplateFlow: + +===== ===== +Key Description +===== ===== +tpl The space of the template. +res Resolution index. See ``template_description.json`` within each template for more information about what the index specifies. +atlas Name of an atlas. +desc Additional information about the file to differentiate it from other files. +===== ===== + +Common suffixes used in TemplateFlow: + +======= ============ +Suffix Description +======= ============ +dseg discrete segmentation +pseg probability segmentation +mask binary mask +xfm transform file +T2w T2 weighted image +T1w T1 weighted image +======= ============ + +Common file-formats used in TemplateFlow: + +========== ======= +Extension Description +========== ======= +.nii.gz Image +.tsv Tabular information +.json Meta-information +.h5 Transform file +========== ======= + +Thus a template with the following name: + +``tpl-test_res-01_atlas-myatlas_desc-200nodes_dseg.nii.gz`` + +This would be a nifti image containing discrete segmentation of "myatlas" which contains 200nodes. +And it is in the template space "test". +The resolution information will be found in the dataset_description.json file under '01'.