MAINT: Use PyBIDS 0.9.2+ #369

effigies · 2019-07-09T13:51:23Z

This is a pre-condition for:

Depends on:

MAINT: Use PyBIDS 0.9.x templateflow/python-client#15

effigies · 2019-07-09T17:25:29Z

I think this will require a 0.10.0 release to avoid version conflicts with things expecting pybids 0.7.x, templateflow 0.3.x, and niworkflows 0.9.x.

effigies · 2019-07-09T17:29:58Z

It's not showing up yet, but I'm moving us to the PEP 440 Compatible Release syntax:

templateflow ~= 0.4.0  # equivalent to...
templateflow >= 0.4.0, == 0.4.*

niworkflows/interfaces/bids.py

niworkflows/reports/core.py

effigies · 2019-07-09T19:29:43Z

@oesteban The pure Python bits (I didn't build the container) test clean locally. Most of the changes are in the reports bit, so you should definitely have a look at that.

effigies · 2019-07-09T20:43:47Z

Ah. Missed utils. Looks like there are some knock-on problems in templateflow, but I don't really know how to diagnose this:

_____________ [doctest] niworkflows.utils.misc.get_template_specs ______________
013 
014     Parse template specifications
015 
016     >>> get_template_specs('MNI152NLin2009cAsym', {'suffix': 'T1w'})[1]
UNEXPECTED EXCEPTION: RuntimeError('Could not find template "MNI152NLin2009cAsym" with specs={\'suffix\': \'T1w\', \'desc\': None, \'atlas\': None, \'resolution\': 1}. Please revise your template argument.')
Traceback (most recent call last):

  File "/usr/local/miniconda/lib/python3.7/doctest.py", line 1329, in __run
    compileflags, 1), test.globs)

  File "<doctest niworkflows.utils.misc.get_template_specs[0]>", line 1, in <module>

  File "/src/niworkflows/niworkflows/utils/misc.py", line 54, in get_template_specs
    argument.""".format(in_template, template_spec))

RuntimeError: Could not find template "MNI152NLin2009cAsym" with specs={'suffix': 'T1w', 'desc': None, 'atlas': None, 'resolution': 1}. Please revise your template argument.

/src/niworkflows/niworkflows/utils/misc.py:16: UnexpectedException

effigies · 2019-07-09T21:18:18Z

Ah, looks like it's a bug in pybids. bids-standard/pybids#458

effigies · 2019-07-10T04:05:07Z

Okay, so I think I've fixed the bug in pybids... Now getting this one:

022     >>> get_template_specs('MNIInfant', {'res': '2', 'cohort': '10', 'suffix': 'T1w'})[1]
UNEXPECTED EXCEPTION: RuntimeError('Could not find template "MNIInfant" with specs={\'cohort\': \'10\', \'suffix\': \'T1w\', \'desc\': None, \'atlas\': None, \'resolution\': \'2\'}. Please revise your template argument.',)
Traceback (most recent call last):

  File "/anaconda3/lib/python3.6/doctest.py", line 1330, in __run
    compileflags, 1), test.globs)

  File "<doctest niworkflows.utils.misc.get_template_specs[2]>", line 1, in <module>

  File "/Users/markiewicz/Projects/crn/niworkflows/niworkflows/utils/misc.py", line 54, in get_template_specs
    argument.""".format(in_template, template_spec))

RuntimeError: Could not find template "MNIInfant" with specs={'cohort': '10', 'suffix': 'T1w', 'desc': None, 'atlas': None, 'resolution': '2'}. Please revise your template argument.

/Users/markiewicz/Projects/crn/niworkflows/niworkflows/utils/misc.py:22: UnexpectedException

I think it's because MNIInfant doesn't get installed anywhere, which makes me wonder why it's not failing on Circle.

oesteban · 2019-07-10T04:52:56Z

Probably removing the templateflow folder will refresh the installation with the latest scaffold.

effigies · 2019-07-10T14:01:13Z

Dockerfile

    python -c "from templateflow import api as tfapi; \
               tfapi.get('MNI152Lin|MNI152NLin2009cAsym|OASIS30ANTs', suffix='T1w'); \
               tfapi.get('MNI152Lin|MNI152NLin2009cAsym|OASIS30ANTs', desc='brain', suffix='mask'); \
               tfapi.get('OASIS30ANTs', resolution=1, desc='4', suffix='dseg'); \
               tfapi.get('OASIS30ANTs|NKI', resolution=1, label='brain', suffix='probseg'); \
-               tfapi.get('OASIS30ANTs|NKI', resolution=1, desc='BrainCerebellumRegistration', suffix='mask'); "
+               tfapi.get('OASIS30ANTs|NKI', resolution=1, desc='BrainCerebellumRegistration', suffix='mask'); " && \
+    deactivate && rm -rf /tmp/venv


pybids was getting installed with templateflow, blocking its upgrade with requirements.txt.

effigies · 2019-07-10T15:51:58Z

New issue:

________________________________ test_ROIsPlot _________________________________
[gw1] linux -- Python 3.7.1 /usr/local/miniconda/bin/python

    def test_ROIsPlot():
        """ the BET report capable test """
        import nibabel as nb
        import numpy as np
    
        im = nb.load(str(
            get_template('OASIS30ANTs', resolution=1, desc='4',
>                        suffix='dseg', extension=['.nii', '.nii.gz'])))

/src/niworkflows/niworkflows/tests/test_segmentation.py:42: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

filename = "[PosixPath('/home/niworkflows/.cache/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_desc-4_dseg.nii.gz'), PosixPath('/home/niworkflows/.cache/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_desc-4_dseg.tsv')]"
kwargs = {}

Can't debug locally because I can't get templateflow configured properly. I can try the Docker image, but this is very annoying.

effigies · 2019-07-10T16:05:58Z

Turns out it was templateflow/python-client#16.

oesteban · 2019-07-10T16:13:22Z

Can't debug locally because I can't get templateflow configured properly. I can try the Docker image, but this is very annoying.

I agree. The most urgent thing, I believe, is making sure that updating TemplateFlow updates the HOME directory at installation (templateflow/python-client#11). Within the context of #374 we can address the problems of testing niworkflows. WDYT?

effigies · 2019-07-10T16:38:00Z

Sounds reasonable.

effigies · 2019-07-10T19:12:33Z

_____________ [doctest] niworkflows.utils.misc.get_template_specs ______________
013 
014     Parse template specifications
015 
016     >>> get_template_specs('MNI152NLin2009cAsym', {'suffix': 'T1w'})[1]
017     {'resolution': 1}
018 
019     >>> get_template_specs('MNI152NLin2009cAsym', {'res': '2', 'suffix': 'T1w'})[1]
020     {'resolution': '2'}
021 
022     >>> get_template_specs('MNIInfant', {'res': '2', 'cohort': '10', 'suffix': 'T1w'})[1]
UNEXPECTED EXCEPTION: RuntimeError('Could not find template "MNIInfant" with specs={\'cohort\': \'10\', \'suffix\': \'T1w\', \'desc\': None, \'atlas\': None, \'resolution\': \'2\'}. Please revise your template argument.')
Traceback (most recent call last):

  File "/usr/local/miniconda/lib/python3.7/doctest.py", line 1329, in __run
    compileflags, 1), test.globs)

  File "<doctest niworkflows.utils.misc.get_template_specs[2]>", line 1, in <module>

  File "/src/niworkflows/niworkflows/utils/misc.py", line 54, in get_template_specs
    argument.""".format(in_template, template_spec))

RuntimeError: Could not find template "MNIInfant" with specs={'cohort': '10', 'suffix': 'T1w', 'desc': None, 'atlas': None, 'resolution': '2'}. Please revise your template argument.

/src/niworkflows/niworkflows/utils/misc.py:22: UnexpectedException

Now we're getting the MNIInfant thing on Circle, which is at least vaguely relieving because it's not fetched anywhere. It's unclear why it ever worked.

effigies · 2019-07-10T19:22:14Z

@oesteban Now I can't replicate the failure locally. Is it just stochastic? I can try restarting, but it's a long wait to find out.

oesteban · 2019-07-10T19:31:58Z

It is very weird because a few lines above it says it is downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-01_T1w.nii.gz which should be already cached in the image.

There needs to be an obscure way that propagates some older templateflow folder.

effigies · 2019-07-10T20:12:39Z

Hmm. It repeated, though the Docker image was identical. Do you see any ways forward?

oesteban · 2019-07-10T20:15:30Z

Maybe rerun the pytest build with ssh and check what the heck TemplateFlow is using for home and whether stuff is really precached

…

On Wed, Jul 10, 2019, 13:12 Chris Markiewicz ***@***.***> wrote: Hmm. It repeated, though the Docker image was identical. Do you see any ways forward? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#369>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAESDRUZUHIL47HVWUQ5JU3P6Y7DPANCNFSM4H7FXZVA> .

effigies · 2019-07-10T20:21:13Z

Probably just easiest to ask: How would you identify the templateflow home and the status of the files?

effigies · 2019-07-10T20:30:12Z

oot@607307af2c0a:/tmp# python -c 'import templateflow.api as tfapi; print(tfapi.TF_LAYOUT.root)'
/home/niworkflows/.cache/templateflow
root@607307af2c0a:/tmp# ls /home/niworkflows/.cache/templateflow/
tpl-MNI152Lin  tpl-MNI152NLin2009cAsym  tpl-NKI  tpl-OASIS30ANTs  tpl-PNC  tpl-fsLR  tpl-fsaverage
root@607307af2c0a:/tmp# ls -AFl /home/niworkflows/.cache/templateflow/
total 28
drwxr-xr-x 3 root root 4096 Jul 10 18:39 tpl-MNI152Lin/
drwxr-xr-x 2 root root 4096 Jul 10 18:39 tpl-MNI152NLin2009cAsym/
drwxr-xr-x 3 root root 4096 Jul 10 18:39 tpl-NKI/
drwxr-xr-x 2 root root 4096 Jul 10 18:39 tpl-OASIS30ANTs/
drwxr-xr-x 2 root root 4096 Jul 10 18:39 tpl-PNC/
drwxr-xr-x 2 root root 4096 Jul 10 18:39 tpl-fsLR/
drwxr-xr-x 2 root root 4096 Jul 10 18:39 tpl-fsaverage/
root@607307af2c0a:/tmp# mv /home/niworkflows/.cache/templateflow{,.old} 
root@607307af2c0a:/tmp# python -c 'import templateflow.api as tfapi; tfapi.get("MNIInfant")'
root@607307af2c0a:/tmp# ls /home/niworkflows/.cache/templateflow
tpl-MNI152Lin  tpl-MNI152NLin2009cAsym  tpl-NKI  tpl-OASIS30ANTs  tpl-PNC  tpl-fsLR  tpl-fsaverage

Maybe MNIInfant isn't always available via the S3 method?

oesteban · 2019-07-10T20:35:57Z

Sorry for the slow response. I believe I know what is happening. Please give me a second.

oesteban · 2019-07-10T20:43:27Z

Okay, yes.

When TemplateFlow is not using DataLad, the first time the api is called the following happens:

A file like https://github.com/templateflow/python-client/blob/master/templateflow/conf/templateflow-skel.zip is unzipped to the templateflow home. This file contains the skeleton of the templateflow repo, with empty files instead of niftis or giftis. As the files are there, pybids can then query.
Selected files by a query get pulled down from S3 if the selected file size is zero bytes. Empty files are replaced by the freshly downloaded.
Subsequent queries to the same files will end up finding the newly fetched version.

What is happening?

Updating the skeleton file is part of the packaging process. The problem is that, at this moment, the circleci build does not push back the new skeleton to the repo.

Since you are installing from a commit, a very old skeleton is found in the repo (honestly, if it does not get updated it should go away).

Fastest solution for you: cut an rc1 pre-release - it takes 5 minutes.

effigies · 2019-07-10T20:46:23Z

Sure, let's do that then. Do you want to or should I?

oesteban · 2019-07-10T20:47:41Z

Done https://github.com/templateflow/python-client/releases/tag/0.4.1rc1

effigies · 2019-07-10T20:57:32Z

Thanks.

oesteban · 2019-07-10T20:58:37Z

Thank you for diving into these frozen waters

Dockerfile

niworkflows/reports/core.py

codecov · 2019-07-12T11:25:06Z

Codecov Report

❗ No coverage uploaded for pull request base (master@44da844). Click here to learn what that means.
The diff coverage is 75%.

@@            Coverage Diff            @@
##             master     #369   +/-   ##
=========================================
  Coverage          ?   39.01%           
=========================================
  Files             ?       43           
  Lines             ?     4788           
  Branches          ?        0           
=========================================
  Hits              ?     1868           
  Misses            ?     2920           
  Partials          ?        0

Flag	Coverage Δ
#unittests	`39.01% <75%> (?)`

Impacted Files	Coverage Δ
niworkflows/utils/bids.py	`88.75% <ø> (ø)`
niworkflows/interfaces/bids.py	`88.52% <100%> (ø)`
niworkflows/reports/core.py	`80.98% <66.66%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 44da844...df6f3fc. Read the comment docs.

codecov · 2019-07-12T11:25:06Z

Codecov Report

❗ No coverage uploaded for pull request base (master@44da844). Click here to learn what that means.
The diff coverage is 75%.

@@            Coverage Diff            @@
##             master     #369   +/-   ##
=========================================
  Coverage          ?   39.01%           
=========================================
  Files             ?       43           
  Lines             ?     4788           
  Branches          ?        0           
=========================================
  Hits              ?     1868           
  Misses            ?     2920           
  Partials          ?        0

Flag	Coverage Δ
#reportlettests	`100% <ø> (?)`
#unittests	`39.01% <75%> (?)`

Impacted Files	Coverage Δ
niworkflows/utils/bids.py	`88.75% <ø> (ø)`
niworkflows/interfaces/bids.py	`88.52% <100%> (ø)`
niworkflows/reports/core.py	`80.98% <66.66%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 44da844...df6f3fc. Read the comment docs.

effigies force-pushed the dep/pybids_0.9 branch from 036bbc6 to b034c09 Compare July 9, 2019 14:16

effigies mentioned this pull request Jul 9, 2019

MAINT: Use PyBIDS 0.9.x templateflow/python-client#15

Merged

oesteban added this to the 0.9.7 milestone Jul 9, 2019

effigies force-pushed the dep/pybids_0.9 branch from a302172 to 48d1e5e Compare July 9, 2019 17:29

effigies commented Jul 9, 2019

View reviewed changes

niworkflows/interfaces/bids.py Show resolved Hide resolved

effigies commented Jul 9, 2019

View reviewed changes

niworkflows/reports/core.py Show resolved Hide resolved

effigies commented Jul 9, 2019

View reviewed changes

niworkflows/reports/core.py Show resolved Hide resolved

effigies commented Jul 10, 2019

View reviewed changes

effigies force-pushed the dep/pybids_0.9 branch from 1c61d60 to df36006 Compare July 10, 2019 14:08

effigies added 10 commits July 10, 2019 14:28

MAINT: Use PyBIDS 0.9.x

4f187e7

MAINT: Upgrade pip in Docker

28b2201

FIX: Pybids 0.9.x compatibility

6c10c98

FIX: scope="raw" fails

79fd68a

FIX: missing_value is unknown configuration option

d66283c

FIX: entities is a model, filenames return full paths

655b71c

FIX: Specify regex_search when using it

25c5b2f

FIX: Extensions is deprecated for extension

350863b

STY: flake8

648ba72

MAINT: Update pins

b72581a

FIX: extensions is deprecated for extension

2ab9594

effigies force-pushed the dep/pybids_0.9 branch from fb423c3 to 2ab9594 Compare July 10, 2019 18:30

oesteban mentioned this pull request Jul 10, 2019

Push new skeletons back to the repo with every release templateflow/python-client#17

Closed

PIN: templateflow 0.4.1rc1

f14002c

effigies and others added 2 commits July 11, 2019 09:22

ENH: Add meaningful error message to failed path build

cc49ba3

Merge branch 'master' into dep/pybids_0.9

73526db

oesteban approved these changes Jul 11, 2019

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

niworkflows/reports/core.py Show resolved Hide resolved

effigies added 4 commits July 11, 2019 18:30

MAINT: Update pins in setup.cfg

7e22432

DOCKER: Update templateflow get to use lists, note regexes

8a612b2

CI: Allow pip 10 failures due to PEP 508 URLs

7a2e89a

DEP: Require pybids ~= 0.9.2

ef6fcd2

effigies changed the title ~~MAINT: Use PyBIDS 0.9.x~~ MAINT: Use PyBIDS 0.9.2+ Jul 12, 2019

CI: Hard-code FreeSurfer license for testing

df6f3fc

effigies merged commit 08138b8 into nipreps:master Jul 12, 2019

effigies deleted the dep/pybids_0.9 branch July 12, 2019 12:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MAINT: Use PyBIDS 0.9.2+ #369

MAINT: Use PyBIDS 0.9.2+ #369

effigies commented Jul 9, 2019 •

edited

Loading

effigies commented Jul 9, 2019

effigies commented Jul 9, 2019

effigies commented Jul 9, 2019

effigies commented Jul 9, 2019

effigies commented Jul 9, 2019

effigies commented Jul 10, 2019

oesteban commented Jul 10, 2019

effigies Jul 10, 2019

effigies commented Jul 10, 2019

effigies commented Jul 10, 2019

oesteban commented Jul 10, 2019

effigies commented Jul 10, 2019

effigies commented Jul 10, 2019

effigies commented Jul 10, 2019

oesteban commented Jul 10, 2019

effigies commented Jul 10, 2019

oesteban commented Jul 10, 2019 via email

effigies commented Jul 10, 2019

effigies commented Jul 10, 2019

oesteban commented Jul 10, 2019

oesteban commented Jul 10, 2019

effigies commented Jul 10, 2019

oesteban commented Jul 10, 2019

effigies commented Jul 10, 2019

oesteban commented Jul 10, 2019

codecov bot commented Jul 12, 2019

codecov bot commented Jul 12, 2019 •

edited

Loading

MAINT: Use PyBIDS 0.9.2+ #369

MAINT: Use PyBIDS 0.9.2+ #369

Conversation

effigies commented Jul 9, 2019 • edited Loading

effigies commented Jul 9, 2019

effigies commented Jul 9, 2019

effigies commented Jul 9, 2019

effigies commented Jul 9, 2019

effigies commented Jul 9, 2019

effigies commented Jul 10, 2019

oesteban commented Jul 10, 2019

effigies Jul 10, 2019

Choose a reason for hiding this comment

effigies commented Jul 10, 2019

effigies commented Jul 10, 2019

oesteban commented Jul 10, 2019

effigies commented Jul 10, 2019

effigies commented Jul 10, 2019

effigies commented Jul 10, 2019

oesteban commented Jul 10, 2019

effigies commented Jul 10, 2019

oesteban commented Jul 10, 2019 via email

effigies commented Jul 10, 2019

effigies commented Jul 10, 2019

oesteban commented Jul 10, 2019

oesteban commented Jul 10, 2019

effigies commented Jul 10, 2019

oesteban commented Jul 10, 2019

effigies commented Jul 10, 2019

oesteban commented Jul 10, 2019

codecov bot commented Jul 12, 2019

Codecov Report

codecov bot commented Jul 12, 2019 • edited Loading

Codecov Report

effigies commented Jul 9, 2019 •

edited

Loading

codecov bot commented Jul 12, 2019 •

edited

Loading