Maintenance, bug fixes and enhancement of the Bidsappmanager

[sebastientourbier]

This PR is dedicated to the upcoming beta-RC2 release. It includes in particular the following changes:

New feature

• Computes of ROI volumetry stored in <output_dir>/sub-<label>(/ses<label>)/anat folder, recognized by their _stats.tsv file name suffix. (Computes parcellation ROI volumes #13)

Upgrade

• Uses fsleyes instead of fslview (now deprecated). (Migration from fslview to fsleyes #21)

Improved replicability

• Sets the MATRIX_RNG_SEED environment variable (used by MRtrix) and seed for the numpy random number generator (numpy.random.seed())

Bug fixes

• Fixes the output inspector window of the cmpbidsappmanager (GUI) that fails to find existing outputs, after adoption of /bids_dir and /output_dir in the bidsapp docker image. (Cannot launch the output inspector window of the GUI #34)
• Fixes the way to get the list of networkx edge attributes in inspect_outputs() of ConnectomeStage for the output inspector window of the cmpbidsappmanager (GUI)
• Added missing package dependencies (fury and vtk) that fixes dipy_CSD execution error when trying to import module actor from dipy.viz to save the results in a png (dipy_CSD executes with error #36)
• Fixes a number of unresolved references identified by pycharm code inspection tool

Code refactoring

• Interfaces for fMRI processing were moved to cmtklib/functionalMRI.py.
• Interface for fMRI connectome creation (rsfmri_conmat) moved to cmtklib/connectome.py

[sebastientourbier]

Installation of BIDSAppmanager fully (creation of new conda environment) and successfully tested on Ubuntu 16.04 (Commit 5a2e00c).

[sebastientourbier]

Tasked before merged:

• Update ds-sample_outputs.txt (used by circleci)
• Update version and release date in info.py
• Update release date in .zenodo.json
• Update doc with list of changes

[Kenji-Hirose]

Hello Sébastien,

I tried the newest method(1~5) to introduce the Connectome Mapper BIDS App Manager noted on the post below:
#29 (comment)
(In order to completely remove the cloned repository of connectomemapper3, I D&Ded it from home to desktop repository)
but had the same error noted on the next post by amitjc:
#29 (comment)
Error is [ ImportError: No module named apptools.io.api ].

I'd appreciate it if you look see the logs(screenshots) attached and give adivice.

Here is version information.
OS: Ubuntu18.04LTS
Connectomemapper3: v3.0.0-beta-RC2

And results of conda list:
WARNING conda.models.version:get_matcher(540): Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 2., but conda is ignoring the . and treating it as 2
packages in environment at /home/kenji/miniconda2/envs/py27cmp-gui:
Name Version Build Channel
appdirs 1.4.4 pypi_0 pypi
asn1crypto 1.4.0 py_0 anaconda
atomicwrites 1.4.0 py_0 anaconda
attrs 19.3.0 py_0 anaconda
backports 1.0 py_2 anaconda
backports.functools_lru_cache 1.6.1 py_0 anaconda
backports_abc 0.5 py_1 anaconda
bids-validator 1.5.4 pypi_0 pypi
blas 1.0 mkl anaconda
boto 2.49.0 pypi_0 pypi
brotlipy 0.7.0 py27h516909a_1000 conda-forge
bz2file 0.98 py27_1 anaconda
ca-certificates 2020.6.24 0 anaconda
cairo 1.14.12 h8948797_3 anaconda
certifi 2019.11.28 py27_0 anaconda
cffi 1.14.0 py27he30daa8_1 anaconda
chardet 3.0.4 py27_1003 anaconda
citeproc-py 0.5.1 pypi_0 pypi
clang 10.0.0 default_hde54327_2 conda-forge
clang-tools 10.0.0 default_hde54327_2 conda-forge
clangdev 10.0.0 default_hde54327_2 conda-forge
clangxx 10.0.0 default_hde54327_2 conda-forge
click 7.1.2 py_0 anaconda
cmpbidsappmanager v3.0.0-beta-RC2 pypi_0 pypi
configparser 4.0.2 py27_0 anaconda
contextlib2 0.6.0.post1 py_0 anaconda
cryptography 2.8 py27h1ba5d50_0 anaconda
cycler 0.10.0 py27_0 anaconda
dbus 1.13.12 h746ee38_0 anaconda
decorator 4.4.2 py_0 anaconda
deprecated 1.2.10 pypi_0 pypi
deprecation 2.0 py_0 conda-forge
dipy 0.15.0 pypi_0 pypi
docopt 0.6.2 pypi_0 pypi
duecredit 0.7.0 pypi_0 pypi
entrypoints 0.3 pypi_0 pypi
enum34 1.1.6 py27_1 anaconda
expat 2.2.9 he6710b0_2 anaconda
fasteners 0.15 pypi_0 pypi
fontconfig 2.13.0 h9420a91_0 anaconda
freetype 2.10.2 h5ab3b9f_0 anaconda
fribidi 1.0.10 h7b6447c_0 anaconda
fsleyes 0.27.3 py_1 conda-forge
fsleyes-props 1.6.6 py_0 conda-forge
fsleyes-widgets 0.7.3 py_1 conda-forge
fslpy 1.13.2 pyh24bf2e0_0 conda-forge
funcsigs 1.0.2 py27_0 anaconda
functools32 3.2.3.2 py27_1 anaconda
future 0.18.2 py27_0 anaconda
futures 3.3.0 py27_0 anaconda
gettext 0.19.8.1 h9b4dc7a_1 anaconda
git 2.23.0 pl526hacde149_0 anaconda
git-annex 7.20190219 h4a95a7c_12 conda-forge
gitdb2 2.0.6 pypi_0 pypi
gitpython 2.1.15 pypi_0 pypi
glib 2.56.2 hd408876_0 anaconda
gmp 6.1.2 hb3b607b_0 anaconda
gnutls 3.6.5 h71b1129_1002 anaconda
graphite2 1.3.14 h23475e2_0 anaconda
graphviz 2.40.1 h21bd128_2 anaconda
gst-plugins-base 1.14.0 hbbd80ab_1 anaconda
gstreamer 1.14.0 hb453b48_1 anaconda
h5py 2.9.0 py27h7918eee_0 anaconda
harfbuzz 1.8.8 hffaf4a1_0 anaconda
hdf5 1.10.4 hb1b8bf9_0 anaconda
html5lib 1.1 py_0 anaconda
humanize 1.0.0 pypi_0 pypi
icu 58.2 he6710b0_3 anaconda
idna 2.10 py_0 anaconda
importlib_metadata 1.3.0 py27_0 anaconda
intel-openmp 2020.1 217 anaconda
ipaddress 1.0.23 py_0 anaconda
iso8601 0.1.12 pypi_0 pypi
isodate 0.6.0 py_1 anaconda
jinja2 2.11.2 py_0 anaconda
jpeg 9b habf39ab_1 anaconda
jsmin 2.2.2 pypi_0 pypi
keepalive 0.5 py_1 conda-forge
keyring 18.0.1 pypi_0 pypi
keyrings-alt 3.2.0 pypi_0 pypi
kiwisolver 1.1.0 py27he6710b0_0 anaconda
krb5 1.18.2 h173b8e3_0 anaconda
libclang 10.0.0 default_hde54327_2 conda-forge
libcurl 7.71.1 h20c2e04_1 anaconda
libedit 3.1.20191231 h14c3975_1 anaconda
libffi 3.3 he6710b0_2 anaconda
libgcc-ng 9.1.0 hdf63c60_0 anaconda
libgfortran-ng 7.3.0 hdf63c60_0 anaconda
libglu 9.0.0 hf484d3e_1 anaconda
libiconv 1.15 h63c8f33_5 anaconda
libllvm10 10.0.0 h4a3c616_1 anaconda
libpng 1.6.37 hbc83047_0 anaconda
libspatialindex 1.9.3 he6710b0_0 anaconda
libssh2 1.9.0 h1ba5d50_1 anaconda
libstdcxx-ng 9.1.0 hdf63c60_0 anaconda
libtiff 4.1.0 h2733197_1 anaconda
libuuid 1.0.3 h1bed415_2 anaconda
libxcb 1.14 h7b6447c_0 anaconda
libxml2 2.9.10 he19cac6_1 anaconda
libxslt 1.1.34 hc22bd24_0 anaconda
linecache2 1.0.0 py_1 anaconda
llvm-tools 10.0.0 h4a3c616_1 anaconda
llvmdev 10.0.0 h4a3c616_1 anaconda
lxml 4.4.2 py27hefd8a0e_0 anaconda
lz4-c 1.9.2 he6710b0_1 anaconda
markupsafe 1.1.1 py27h7b6447c_0 anaconda
matplotlib 2.2.3 py27hb69df0a_0 anaconda
mkl 2019.4 243 anaconda
mkl-service 2.3.0 py27he904b0f_0 anaconda
mkl_fft 1.0.15 py27ha843d7b_0 anaconda
mkl_random 1.1.0 py27hd6b4f25_0 anaconda
mne 0.17.1 pypi_0 pypi
mock 3.0.5 pypi_0 pypi
monotonic 1.5 pypi_0 pypi
more-itertools 5.0.0 py27_0 anaconda
msgpack 1.0.0 pypi_0 pypi
ncurses 6.2 he6710b0_1 anaconda
nettle 3.4.1 hbb512f6_0 anaconda
networkx 2.2 py27_1 anaconda
neurdflib 5.0.1 py_0 conda-forge
nibabel 2.5.1 py_0 conda-forge
nipype 1.1.8 py27_0 conda-forge
num2words 0.5.10 pypi_0 pypi
numpy 1.15.4 py27h7e9f1db_0 anaconda
numpy-base 1.15.4 py27hde5b4d6_0 anaconda
obspy 1.1.1 pypi_0 pypi
olefile 0.46 py27_0 anaconda
openssl 1.1.1g h7b6447c_0 anaconda
packaging 20.4 py_0 anaconda
pandas 0.24.2 pypi_0 pypi
pango 1.42.4 h049681c_0 anaconda
pathlib2 2.3.5 py27_0 anaconda
patool 1.12 pypi_0 pypi
patsy 0.5.1 pypi_0 pypi
pcre 8.44 he6710b0_0 anaconda
perl 5.26.2 h14c3975_0 anaconda
pillow 6.2.1 py27h34e0f95_0 anaconda
pip 19.0.3 py27_0 anaconda
pixman 0.40.0 h7b6447c_0 anaconda
pluggy 0.13.1 py27_0 anaconda
popt 1.16 h299ea2f_2002 conda-forge
progressbar2 3.37.1 py27_0 anaconda
prov 1.5.3 py_1 conda-forge
psutil 5.6.7 py27h7b6447c_0 anaconda
py 1.9.0 py_0 anaconda
pybids 0.9.1 pypi_0 pypi
pycparser 2.20 py_2 anaconda
pydicom 2.0.0 pyh9f0ad1d_0 conda-forge
pydot 1.4.1 py27_0 anaconda
pydotplus 2.0.2 py_3 anaconda
pyface 6.1.2 py27_0 anaconda
pygithub 1.45 pypi_0 pypi
pygments 2.5.2 py_0 anaconda
pyjwt 1.7.1 pypi_0 pypi
pyopengl 3.1.1a1 py27_0 anaconda
pyopenssl 19.1.0 py_1 anaconda
pyparsing 2.4.7 py_0 anaconda
pyqt 5.9.2 py27h22d08a2_1 anaconda
pyside2 5.9.0a1 py27h4dc837a_0 conda-forge
pysocks 1.7.1 py27_0 anaconda
pytest 4.6.2 py27_0 anaconda
pytest-runner 5.2 py_0 anaconda
python 2.7.15 h9bab390_6 anaconda
python-dateutil 2.8.1 py_0 anaconda
python-utils 2.3.0 py27_0 anaconda
pytz 2020.1 py_0 anaconda
qt 5.9.7 h5867ecd_1 anaconda
rdflib 4.2.2 py27_1000 conda-forge
readline 7.0 h7b6447c_5 anaconda
requests 2.24.0 py_0 anaconda
rsync 3.1.3 hed695b0_1002 conda-forge
rtree 0.8.3 py27_0 anaconda
scandir 1.10.0 py27h7b6447c_0 anaconda
scipy 1.2.1 py27h7c811a0_0 anaconda
secretstorage 2.3.1 pypi_0 pypi
setuptools 44.0.0 py27_0 anaconda
simplejson 3.17.0 py27h7b6447c_0 anaconda
singledispatch 3.4.0.3 py27_0 anaconda
sip 4.19.13 py27he6710b0_0 anaconda
six 1.15.0 py_0 anaconda
smmap 3.0.4 pypi_0 pypi
smmap2 3.0.1 pypi_0 pypi
sparqlwrapper 1.8.2 py27_1000 conda-forge
sphinx_rtd_theme 0.4.3 py_0 anaconda
sqlalchemy 1.3.19 pypi_0 pypi
sqlite 3.33.0 h62c20be_0 anaconda
subprocess32 3.5.4 py27h7b6447c_0 anaconda
tk 8.6.10 hbc83047_0 anaconda
tornado 5.1.1 py27h7b6447c_0 anaconda
tqdm 4.48.2 pypi_0 pypi
traceback2 1.4.0 py27_0 anaconda
traits 5.1.2 py27h7b6447c_0 anaconda
traitsui 6.0.0 py27_1 anaconda
trimesh 2.38.42 py_0 conda-forge
unittest2 1.1.0 py27_0 anaconda
urllib3 1.25.10 py_0 anaconda
wcwidth 0.2.5 py_0 anaconda
webencodings 0.5.1 py27_1 anaconda
wheel 0.33.6 py27_0 anaconda
whoosh 2.7.4 pypi_0 pypi
wrapt 1.12.1 pypi_0 pypi
wxnatpy 0.3.2 py_0 conda-forge
wxpython 4.0.4 py27hc99224d_0 anaconda
xnat 0.3.24 pyh9f0ad1d_0 conda-forge
xvfbwrapper 0.2.9 py27_1001 conda-forge
xz 5.2.5 h7b6447c_0 anaconda
zipp 0.6.0 py_0 anaconda
zlib 1.2.11 h7b6447c_3 anaconda
zstd 1.4.4 h0b5b093_3 anaconda

I'm deeply grateful for your constant support.
(and I apologize for my poor English.

Best regards,
Kenji.

[sebastientourbier]

Hi @Kenji-Hirose,

I have a good news, I made an end of the beta- series and released a few days ago the first release candidate of CMP3 (version v3.0.0-RC1) where I finally managed to migrate to Python3 and did a lot of cleaning of import <package> (See #40).

By checking the new pipelines/common.py, it seems this import has been removed:

connectomemapper3/cmp/pipelines/common.py

Lines 10 to 22 in e3d1554

	import os
	# import fnmatch
	import threading
	import time
	import nipype.pipeline.engine as pe
	import nipype.interfaces.utility as util
	from nipype.interfaces.base import File, Directory
	from traits.api import *

To install the new latest version (v3.0.0-RC1) you will have to:

1. Update the git repo and create a new branch that tracks to the version tag

   $ git fetch
   $ git checkout tags/v3.0.0-RC1 -b v3.0.0-RC1
   

2. You can remove the old conda env (py27cmp-gui) as this would not be used anymore.

   $ conda env remove py27cmp-gui
   

   Note: If you already have downloaded the docker image for version v3.0.0-beta-RC2, you can also delete it with the help of the docker images command to identify the image_ID followed by the docker rmi image_ID command that specify the ID of the image to be removed.
3. Install the new conda environment (py37cmp-gui)

   $ conda env create -f environment.yml
   

[Kenji-Hirose]

Hello Sébastien,

Thank you for your very quick response!

I tried the procedure you mentioned(first 2 screenshots),
but I got the message saying "there is no such command like 'cmpbidsappmanager'"(the last screenshot).

The branch would be correct, I guess(the last screenshot).

Do you have any idea for this??

Many thanks.

Best regards,
Kenji.

[sebastientourbier]

@Kenji-Hirose Forgot to add, you would need to also reinstall CMP3 with the GUI:

(base)$ conda activate py37cmp-gui
(py37cmp-gui)$ python setup_gui.py install 

And then cmpbidsappmanager command should work.

[Kenji-Hirose]

Thanks again for your prompt advice!

Finally I made it(see screenshot)!

Next step I will grasp the BIDS format and make the cmpbidsappmanager app load the data.

Again and again, I greatly appreciate your kindness.

Sincerely,
Kenji.