Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvements and debugs #173

Merged
merged 74 commits into from Mar 7, 2019

Conversation

Projects
None yet
3 participants
@tiborauer
Copy link
Member

tiborauer commented Oct 17, 2018

Improvements

  • CoSMoMVPA:
    • allow specifying mask
    • classification
    • settings for tasks
    • balance partitions for classification
  • generic modules
    • coreg_extended_2
    • norm_write(_dartel)
    • dartel_denorm
  • seperate streams for mask_fromsegment
  • provenance - align with NIDM-W
  • diagnostics visual
    • more intuitive tsdiffana plots
    • consistent colours for segmentations (histogram and overlay)
  • checkreg can now process any number of overlays (i.e. ROIs)

Debugs:

  • connection
    • with generic modules
    • with optional stream
  • epifromnifti with dummies
  • *fromnifti modules uncompress data into aa rather than into the raw directory
  • aamod_freesurfer_register on study level
  • connecting to pipeline + initialstream
  • firstlevel_threshold report when not all are preceded by firstlevel_contrasts

tiborauer added some commits Aug 13, 2018

@tiborauer

This comment has been minimized.

Copy link
Member Author

tiborauer commented Jan 31, 2019

I have resolved the conflict. What "BIDS dataset problems" do you mean? Is it about the demo we are using (ds114) or the one you mention in #186 ?

@jpeelle

This comment has been minimized.

Copy link
Member

jpeelle commented Feb 1, 2019

When running the ds114 demo script my computer crashes (= turns off, which I suspect might be a segfault). I have not seen this behavior at all doing any other activity so I don't think it's something like a bad hard drive. However, I haven't been back at my desk long enough to investigate further. It could be totally unrelated (but suspicious that it happened 5 times in a row when running that script, but no others). (I realize segfaults while running these analyses should be rate/nonexistent but that's all I can think of.) That was also on Matlab 2016b, although that's never been an issue in the past.

@jpeelle

This comment has been minimized.

Copy link
Member

jpeelle commented Feb 1, 2019

Given how unusual the problem is, and that this is my first time running through the aa ds114 example script, .I suspect that whatever problem is occurring is not because of the PR. Thus I'm happy to merge it. But since I haven't officially tested it as I should I wanted to document that before moving ahead (I'm assuming no one else's computer is shutting down when running example scripts).

@tiborauer

This comment has been minimized.

Copy link
Member Author

tiborauer commented Feb 1, 2019

The ds114 has DWI data, and the pipeline contains aamod_diffusion_dtinlfit, which is memory intensive and may crash with segmentation fault, so I guess it is that particular module which causes the issue. Can you confirm this?
Have you tried aamod_diffusion_dtinlfit before or any other dataset? What happens if you replace aamod_diffusion_dtinlfit with aamod_diffusion_dtifit?

@tiborauer

This comment has been minimized.

Copy link
Member Author

tiborauer commented Feb 7, 2019

Hi, @jpeelle, Any news on this #173?

@jpeelle

This comment has been minimized.

Copy link
Member

jpeelle commented Feb 7, 2019

@tiborauer

This comment has been minimized.

Copy link
Member Author

tiborauer commented Feb 7, 2019

It is quite a bit PR, and I also do not want to plant seed of some potential issues. But it is your call, and we can also debug things later. Merging it may open up space for wider testing. :)

Also, if you merge it here, you may also need to update your fork(s) from where you submit PR(s).

@jpeelle

This comment has been minimized.

Copy link
Member

jpeelle commented Feb 7, 2019

@jooh what do you think?

@jooh

This comment has been minimized.

Copy link
Collaborator

jooh commented Feb 8, 2019

Sorry, I've been busy on the python side of things. I can run the tests at cbu if you haven't already.

One small point - it might be worth doing a new example script for your MVPA modules so those get some test coverage.

@jooh

This comment has been minimized.

Copy link
Collaborator

jooh commented Feb 12, 2019

Actually, @tiborauer, to speed up testing of your various PRs, would it make sense to consolidate them all into a single PR that I can test once?

@tiborauer

This comment has been minimized.

Copy link
Member Author

tiborauer commented Feb 12, 2019

Can do. Shall I merge the into this one?

@jooh

This comment has been minimized.

Copy link
Collaborator

jooh commented Feb 12, 2019

tiborauer added some commits Feb 12, 2019

Merge pull request #82 from tiborauer/standalone
DEBUG: standalone according to the recent updates
@tiborauer

This comment has been minimized.

Copy link
Member Author

tiborauer commented Feb 12, 2019

Done!

@jooh

This comment has been minimized.

Copy link
Collaborator

jooh commented Feb 12, 2019

Ok, tests running... I will run tests both under CSH and Bash this time, so we will probably need to fix #191 to get this one through. But we shall see!

@jooh

This comment has been minimized.

Copy link
Collaborator

jooh commented Feb 13, 2019

Yes, indeed. We crash on aa_demo on CSH shells. So I suggest we get this fixed ASAP by including the fix in this PR.

***LINUX ERROR FROM SHELL ulimit: Command not found.
        
***WHILE RUNNING COMMAND
setenv TERM dumb;ulimit -m
        
        
**** automatic analysis failed - see reason and line numbers below
***WITH ENVIRONMENT VARIABLES
REMOTEHOST=login27.mrc-cbu.cam.ac.uk
NNTPSERVER=nntp-serv.cam.ac.uk
MANPATH=/cluster-software/gold/2.2.0.5/man:/cluster-software/torque-2.3/man:/usr/share/man
HOSTNAME=login12
FS_OVERRIDE=0
HOST=login12
TERM=screen-256color
SHELL=/bin/tcsh
SSH_CLIENT=172.31.120.37 44716 22
PERL5LIB=/imaging/local/software/freesurfer/latest/x86_64/mni/lib/perl5/5.8.5:/imaging/local/software/freesurfer/latest/x86_64/mni/lib/perl5/5.8.5:/imaging/local/lib/perl5
OLDPWD=/hpc-software/matlab/r2018a
QTDIR=/usr/lib64/qt-3.3
QTINC=/usr/lib64/qt-3.3/include
OS=Linux
SSH_TTY=/dev/pts/2
LOCAL_DIR=/imaging/local/software/freesurfer/latest/x86_64/local
sourcefile=/imaging/local/software/freesurfer/latest/x86_64/FreeSurferEnv.csh
GROUP=ftp
USER=train01linux
LD_LIBRARY_PATH=/imaging/local/software/mrtrix/v0.2.10/lib:/imaging/local/software/mrtrix/v0.2.10/lib:/imaging/local/linux/lib
LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lz=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.bz=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.rar=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
AUTOMOUNT_MAP=
FSFAST_HOME=/imaging/local/software/freesurfer/latest/x86_64/fsfast
MNI_PERL5LIB=/imaging/local/software/freesurfer/latest/x86_64/mni/lib/perl5/5.8.5
FMRI_ANALYSIS_DIR=/imaging/local/software/freesurfer/latest/x86_64/fsfast
HOSTTYPE=x86_64-linux
FSF_OUTPUT_FORMAT=nii
GOLD_HOME=/cluster-software/gold/2.2.0.5
MINC_BIN_DIR=/imaging/local/software/freesurfer/latest/x86_64/mni/bin
OSG_LD_LIBRARY_PATH=/hpc-software/matlab/r2018a/sys/openscenegraph/lib/glnxa64
SUBJECTS_DIR=/imaging/train01linux/subjects
TOOLBOX=/hpc-software/matlab/r2018a/toolbox
MAIL=/var/spool/mail/train01linux
PATH=/imaging/local/software/mrtrix/v0.2.10/bin:/imaging/local/software/freesurfer/latest/x86_64/bin:/imaging/local/software/freesurfer/latest/x86_64/fsfast/bin:/imaging/local/software/freesurfer/latest/x86_64/tktools:/imaging/local/software/fsl/v5.0.6/x86_64/fsl/bin:/imaging/local/software/freesurfer/latest/x86_64/mni/bin:/imaging/local/software/fsl/v5.0.6/x86_64/fsl/bin:/hpc-software/bin:/imaging/local/software:/opt/TurboVNC/bin:/opt/TurboVNC/bin:/imaging/local/software/anaconda/latest/x86_64/bin:/opt/gold/bin:/opt/gold/sbin:/imaging/local/software/mrtrix/v0.2.10/bin:/imaging/local/software/freesurfer/latest/x86_64/bin:/imaging/local/software/freesurfer/latest/x86_64/fsfast/bin:/imaging/local/software/freesurfer/latest/x86_64/tktools:/imaging/local/software/fsl/v5.0.6/x86_64/fsl/bin:/imaging/local/software/freesurfer/latest/x86_64/mni/bin:/imaging/local/software/fsl/v5.0.6/x86_64/fsl/bin:/hpc-software/bin:/imaging/local/software:/cluster-software/gold/2.2.0.5/sbin:/cluster-software/maui/bin:/cluster-software/torque-2.3/sbin:/cluster-software/torque-2.3/bin:/opt/TurboVNC/bin:/opt/TurboVNC/bin:/usr/lib64/qt-3.3/bin:/imaging/local/software/anaconda/latest/x86_64/bin:/opt/gold/bin:/opt/gold/sbin:/usr/local/bin:/bin:/usr/bin:/opt/bin:/usr/local/wrapper/bin:/usr/local/bin:/usr/bin/X11:/usr/local/cbu/scripts/:/imaging/local/linux/bin:/imaging/local/linux/bin/elekta:/imaging/local/linux/bin/tractography/bin:/imaging/local/scripts:/home/train01linux/bin:/imaging/local/software/dcmtk/v3.6.0/dcmtk/bin:/usr/local/wrapper/bin:/usr/local/bin:/usr/bin/X11:/usr/local/cbu/scripts/:/imaging/local/linux/bin:/imaging/local/linux/bin/elekta:/imaging/local/linux/bin/tractography/bin:/imaging/local/scripts:/home/train01linux/bin
PWD=/imaging/train01linux/aa/aa_demo/auditory/aamod_convert_structural_00001/S1/structurals
FUNCTIONALS_DIR=/imaging/train01linux/sessions
_LMFILES_=/cluster-software/modulefiles/torque/2.3.0:/cluster-software/modulefiles/maui:/cluster-software/modulefiles/gold/2.2.0.5
EDITOR=vi
MOABHOMEDIR=/cluster-software/maui-home
MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles:/cluster-software/modulefiles
MAUIHOMEDIR=/cluster-software/maui-home
LOADEDMODULES=torque/2.3.0:maui:gold/2.2.0.5
KDEDIRS=/usr
XFILESEARCHPATH=/hpc-software/matlab/r2018a/sys/java/jre/glnxa64/jre/lib/locale/for help, see the <a href="https://github.com/rhodricusack/automaticanalysis/wiki">aa wiki</a>
FAIL - /home/train01linux/matlab/automaticanalysis/examples/aa_user_demo.m
@jooh

This comment has been minimized.

Copy link
Collaborator

jooh commented Feb 14, 2019

On the Bash side, cbu/aa_user_diffusion appears to be broken.

running test aa_user_diffusion.m...

Please wait a moment, adding <a href = "matlab: cd /home/jc01/matlab/aa5_devel"></a> to the path
Welcome to aa version 5.4.0 (222744c4128647cad4c2c4c2d5330a8fbb3d1e74) Feb 2019
 If you publish work that has used aa, please cite our manuscript:
 <a href = "http://dx.doi.org/10.3389/fninf.2014.00090">Cusack R, Vicente-Grabovetsky A, Mitchell DJ, Wild CJ, Auer T, Linke AC, Peelle JE (2015)</a>
 <a href = "http://dx.doi.org/10.3389/fninf.2014.00090">Automatic analysis (aa): Efficient neuroimaging workflows and parallel processing using Matlab and XML</a>
 <a href = "http://dx.doi.org/10.3389/fninf.2014.00090">Frontiers in Neuroinformatics 8:90</a>

Please visit <a href = "http://automaticanalysis.org">The aa website</a> for more information!

Here you can find example <a href = "matlab: cd /home/jc01/matlab/aa5_devel/aa_parametersets">tasklists</a> and <a href = "matlab: cd /home/jc01/matlab/aa5_devel/examples">scripts</a>.
Ready.


**** automatic analysis failed - see reason and line numbers below
ERROR: Stream normalised_grey of module aamod_diffusion_dartel_denormDKI not found!
for help, see the <a href="https://github.com/rhodricusack/automaticanalysis/wiki">aa wiki</a>

This appears to be because the stream names have changed, which the example script does not reflect.

@tiborauer

This comment has been minimized.

Copy link
Member Author

tiborauer commented Feb 20, 2019

I have looked at the script and, indeed, its streamname has no prefix according to my commit.
I guess we have not started to consistently remove prefixes...

The quick fix is aap = aas_renamestream(aap, 'aamod_diffusion_dartel_denormDKI_00001', 'grey', 'normalised_white', 'input');. However, the output will be also named "normalised_white" (no change in streamname within the module).

jooh and others added some commits Feb 21, 2019

@jooh jooh merged commit 1a68e1d into automaticanalysis:master Mar 7, 2019

@jooh

This comment has been minimized.

Copy link
Collaborator

jooh commented Mar 7, 2019

And that's it, all the tests pass. Merging. Thanks Tibor!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.