Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] macstoolbox during stats step in parfor not working #1225

Closed
4 of 9 tasks
marcobarilari opened this issue Apr 8, 2024 · 16 comments · Fixed by #1228
Closed
4 of 9 tasks

[BUG] macstoolbox during stats step in parfor not working #1225

marcobarilari opened this issue Apr 8, 2024 · 16 comments · Fixed by #1228
Labels
bug 🐛 Something isn't working

Comments

@marcobarilari
Copy link
Collaborator

Is there an existing issue for this?

  • I have searched the existing issues

Operating system

  • Linux
  • Mac
  • Windows

Operating system version

  • Linux Ubuntu 22.04 "jammy jellyfish"

SPM 12 version

  • 7771
  • 7487
  • 7219

Platform

  • MATLAB
  • Octave

Platform version

  • MATLAB 2018a

bidspm version

v3.1.0

bidspm branch / commit number

branch =

'main'

commit =

'94a83c9e'

Expected Behavior

No response

BIDS stats model

No response

Current Behavior & Error message

% copy the error you got below
% before the three backticks (```)


cp: cannot create regular file '/home/marcobarilari/data/tools/spm12/toolbox/MACS/batch_MA_LFE_group_auto.m': File exists


Anything else?

the erros message is repeated for all the jobs

@marcobarilari marcobarilari added the bug 🐛 Something isn't working label Apr 8, 2024
Copy link

github-actions bot commented Apr 8, 2024

Thank you for your issue. Give us a little time to review it.

PS. You might want to check the FAQ if you haven't done so already.

This is an automated reply, generated by FAQtory

@Remi-Gau
Copy link
Contributor

Remi-Gau commented Apr 8, 2024

can you copy the script?

just curious to know at what step this is happening

@marcobarilari
Copy link
Collaborator Author

Well, I guess we are witnessing again he magic of bug disappearing while having someone else attention remotely. I cannot reproduce the error and it is running now.

Though it might be the case that first run in parallel was not able to copy something, it was able to it w/o parallel option, now running it in parallel does not complain anymore. I had a bad book keeping the first time and now I do not have the full error message.

Are you still interested in the script?

@Remi-Gau
Copy link
Contributor

Remi-Gau commented Apr 8, 2024

not any more!!! 😉

but if the shy bug build confidence and reappears, send it with the script.

@marcobarilari
Copy link
Collaborator Author

all right, not the same but within the same domain.

running some stats step in parallel.

this is the error:

ERROR LOG SAVED:
        /mnt/HD_jupiter/marcobarilari/datalad/analysis_high-res_BLAM_biMotion_localizers/outputs/derivatives/error_logs/error_2024-04-08T17-21.log
Use it when opening an issue:
        https://github.com/cpp-lln-lab/bidspm/issues/new/choose.

                line 17 in /mnt/HD_jupiter/marcobarilari/datalad/analysis_high-re
Error using bidspm>installMacstoolbox (line 272)
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MA_cvLME_single.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MA_init_header.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MA_inspect_GoF.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MA_load_betas.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MA_load_data.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MA_load_mask.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MA_model_space.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MC_LBF_group.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MD_Dir_exc_prob.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MD_Dir_mean.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MD_Dir_mode.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MD_dirrnd.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MD_gamrnd.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MD_multrnd.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MD_mvn_logdet.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/ME_BMS_FFX.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/ME_BMS_RFX_GS.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/ME_GLM.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/ME_GLM_DIC.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/ME_GLM_GoF.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/ME_GLM_LL.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/ME_GLM_NG_AnC.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/ME_GLM_NG_LME.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/ME_GLM_SNR.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/ME_GLM_con.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/ME_GLM_pinv.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/ME_GLM_pinv_con.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/ME_MF_LFE.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/ME_MS_PPs.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MF_int2str0.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MF_visualize.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MS_BMA_group.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MS_BMS_group.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MS_BMS_group_fams.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MS_PPs_uniform.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MS_SMM_BMS.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/MS_create_mask.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/README.md could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/batch_MA_LFE_group_auto.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/batch_MA_LFE_group_man.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/batch_MA_classic_ICs_auto.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/batch_MA_classic_ICs_man.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/batch_MA_cvLME_auto.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/batch_MA_cvLME_man.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/batch_MA_inspect_GoF.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/batch_MA_model_space.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/batch_MC_LBF_group_auto.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/batch_MC_LBF_group_man.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/batch_MF_visualize.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/batch_MS_BMA_group_auto.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/batch_MS_BMA_group_man.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/batch_MS_BMS_fams_auto.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/batch_MS_BMS_fams_man.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/batch_MS_BMS_group_auto.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/batch_MS_BMS_group_man.m could not be removed.
/home/marcobarilari/data/tools/spm12/toolbox/MACS/batch_MS_PPs_group_auto.m could not be removed.


Error in bidspm>initBidspm (line 175)
    installMacstoolbox();

Error in bidspm>executeAction (line 49)
      initBidspm();

Error in bidspm (line 26)
    returnCode = executeAction(action, varargin{:});

Error in getOptionsFromCliArgument (line 17)
  bidspm('action', 'init');

Error in cliStats (line 31)
  opt = getOptionsFromCliArgument(args);

Error in bidspm>executeAction (line 86)
      cliStats(varargin{2:end});

Error in bidspm (line 26)
    returnCode = executeAction(action, varargin{:});

the error is printed n = jobs

@marcobarilari
Copy link
Collaborator Author

entire script is this one

% (C) Copyright 2024 Marco Barilari
clear;
clc;

%% Initialize bidspm

this_dir = fileparts(mfilename('fullpath'));

% monster path
% this_dir = '/mnt/HD_jupiter/marcobarilari/datalad/analysis_high-res_BLAM_biMotion_localizers/code/src';

% paul hd path
% this_dir = '/Volumes/PAUL/datalad/analysis_high-res_BLAM_biMotion_localizers/code/src';

% version 3.1.0dev
addpath(fullfile(this_dir, '..', 'lib', 'bidspm'));

bidspm();

%% Settings

yoda_dir = fullfile(this_dir, '..', '..');
bids_dir = fullfile(yoda_dir, 'inputs', 'raw');
% bids_dir = '/Volumes/PAUL/datalad/analysis_high-res_BLAM_biMotion_localizers/inputs/raw';
output_dir = fullfile(yoda_dir, 'outputs', 'derivatives');
preproc_dir = fullfile(output_dir, 'bidspm-preproc');

% the subject we want to analyse;
subject_label = {'SC08', 'SC12', 'SC13', 'SC14'}; % SC08 SC12 SC13 SC14
sessio_label = '01'; 

% subject_label = {'SC04'}; % SC04
% sessio_label = '02'; 

task = 'visualLocalizer';

opt.bidsFilterFile.t1w.suffix = 'UNIT1';
opt.bidsFilterFile.t1w.ses = sessio_label;

% opt.bidsFilterFile.bold.part = {'mag'};
opt.bidsFilterFile.bold.ses = sessio_label;

dummy_scans = 4;

verbosity = 3;

dry_run = false;

% ignore = {'qa'};


%% Stats - subject level

% this bids app call will run:
%
% - GLM specification + estimation,
% - compute contrasts and
% - show results
%
% that are otherwise handled by the bidsFFX.m and bidsResults.m workflows
%
% type bidspm('action', 'help')
% or see this page: https://bidspm.readthedocs.io/en/stable/bids_app_api.html
% for more information on what parameters are obligatory or optional
%
% The results to show are specified in the model

model_file = fullfile(this_dir, 'models', 'model-visualLocalizer_space-individual_smdl.json');

fwhm = 6;

% parfor i = 1:length(subject_label)

%     bidspm(bids_dir, output_dir, 'subject', ...
%         'action', 'stats', ...
%         'participant_label', subject_label(i), ...
%         'preproc_dir', preproc_dir, ...
%         'model_file', model_file, ...
%         'dry_run', dry_run, ...
%         'verbosity', verbosity, ...         
%         'fwhm', fwhm, ...
%         'concatenate', false);

% end

fwhm = 4;

% parfor i = 1:length(subject_label)

    bidspm(bids_dir, output_dir, 'subject', ...
        'action', 'stats', ...
        'participant_label', subject_label, ...
        'preproc_dir', preproc_dir, ...
        'model_file', model_file, ...
        'dry_run', dry_run, ...
        'verbosity', verbosity, ...
        'fwhm', fwhm, ...
        'concatenate', false);

% end

fwhm = 0;

parfor i = 1:length(subject_label)

    bidspm(bids_dir, output_dir, 'subject', ...
        'action', 'stats', ...
        'participant_label', subject_label(i), ...
        'preproc_dir', preproc_dir, ...
        'model_file', model_file, ...
        'dry_run', dry_run, ...
        'verbosity', verbosity, ...
        'fwhm', fwhm, ...
        'concatenate', false);

end

%% Results

% prepare to print the results
results = defaultResultsStructure();

opt.subjects = cell(1,1);

results.nodeName = 'subject_level';
results.name = {'static', 'motion', 'motion_gt_static', 'static_gt_motion'};

results.k = 0;
results.threshSpm = true();
results.png = false();
results.csv = true();
results.binary = true();
opt.results.nidm = false();
results.montage.do = true();
results.montage.slices = -50:10:50;
results.montage.orientation = 'axial';
results.montage.background = struct('sub', opt.subjects{1}, ...
    'suffix', 'UNIT1', ...
    'space', 'individual', ...
    'desc', 'skullstripped', ...
    'modality', 'anat');

% 001 unc
results.MC = 'none';
results.p = 0.001;

% 05 FWE
results(2) = results;

results(2).MC = 'FWE';
results(2).p = 0.05;

% run results
opt.results = results;

fwhm = 6;

parfor i = 1:length(subject_label)
    
    bidspm(bids_dir, output_dir, 'subject', ...
        'action', 'results', ...
        'participant_label', subject_label(i), ...
        'preproc_dir', preproc_dir, ...
        'model_file', model_file, ...
        'dry_run', dry_run, ...
        'verbosity', verbosity, ...
        'fwhm', fwhm, ...
        'options', opt);
    
end

fwhm = 4;

parfor i = 1:length(subject_label)
    
    bidspm(bids_dir, output_dir, 'subject', ...
        'action', 'results', ...
        'participant_label', subject_label(i), ...
        'preproc_dir', preproc_dir, ...
        'model_file', model_file, ...
        'dry_run', dry_run, ...
        'verbosity', verbosity, ...
        'fwhm', fwhm, ...
        'options', opt);
    
end

fwhm = 0;

parfor i = 1:length(subject_label)
    
    bidspm(bids_dir, output_dir, 'subject', ...
        'action', 'results', ...
        'participant_label', subject_label(i), ...
        'preproc_dir', preproc_dir, ...
        'model_file', model_file, ...
        'dry_run', dry_run, ...
        'verbosity', verbosity, ...
        'fwhm', fwhm, ...
        'options', opt);
    
end

%%%%%%%%%%%%%%%%%%%%%% from another project

% results = defaultResultsStructure();
%
% opt.subjects = cell(1,1);
%
% results.nodeName = 'subject_level';
%
% results.name = {'visual_full_motion', 'auditory_full_motion', ...
%                 'visual_sf_high', 'visual_sf_low',...
%                 'auditory_sf_high', 'auditory_sf_low', ...
%                 'visual_full_motion_Gt_auditory_full_motion', ...
%                 'auditory_full_motion_Gt_visual_full_motion', ...
%                 'visual_sf_high_Gt_auditory_sf_high', ...
%                 'auditory_sf_high_Gt_visual_sf_high', ...
%                 'visual_sf_low_Gt_auditory_sf_low', ...
%                 'auditory_sf_low_Gt_visual_sf_low', ...
%                 'auditory_sf_high_Gt_auditory_sf_low', ...
%                 'auditory_sf_low_Gt_auditory_sf_high', ...
%                 'visual_sf_high_Gt_visual_sf_low', ...
%                 'visual_sf_low_Gt_visual_sf_high', ...
%                 'auditory_sf_Gt_visual_sf', ...
%                 'visual_sf_Gt_auditory_sf'};
%
% results.MC = 'none';
% results.p = 0.001;
% results.k = 0;
% results.threshSpm = true();
% results.png = false();
% results.csv = false();
% results.binary = true();
% results.nidm = false();
% results.montage.do = true();
% results.montage.slices = -50:10:50;
% results.montage.orientation = 'axial';
% results.montage.background = struct('sub', opt.subjects{1}, ...
%                                     'suffix', 'T1w', ...
%                                     'space', space, ...
%                                     'desc', 'skullstripped', ...
%                                     'modality', 'anat');
%
% results(2) = results;
%
% results(2).MC = 'FWE';
% results(2).p = 0.05;
%
% opt.results = results;
%
% bidspm(bids_dir, output_dir, 'subject', ...
%     'action', 'results', ...
%     'participant_label', subject_label, ...
%     'preproc_dir', preproc_dir, ...
%     'model_file', model_file, ...
%     'dry_run', dry_run, ...
%     'verbosity', 3, ...
%     'space', space, ...
%     'fwhm', 6, ...
%     'options', opt);
%
% bidspm(bids_dir, output_dir, 'subject', ...
%     'action', 'results', ...
%     'participant_label', subject_label, ...
%     'preproc_dir', preproc_dir, ...
%     'model_file', model_file, ...
%     'dry_run', dry_run, ...
%     'verbosity', 3, ...
%     'space', space, ...
%     'fwhm', 4, ...
%     'options', opt);
%
% bidspm(bids_dir, output_dir, 'subject', ...
%     'action', 'results', ...
%     'participant_label', subject_label, ...
%     'preproc_dir', preproc_dir, ...
%     'model_file', model_file, ...
%     'dry_run', dry_run, ...
%     'verbosity', 3, ...
%     'space', space, ...
%     'fwhm', 2, ...
%     'options', opt);
%
% bidspm(bids_dir, output_dir, 'subject', ...
%     'action', 'results', ...
%     'participant_label', subject_label, ...
%     'preproc_dir', preproc_dir, ...
%     'model_file', model_file, ...
%     'dry_run', dry_run, ...
%     'verbosity', 3, ...
%     'space', space, ...
%     'fwhm', 0, ...
%     'options', opt);

@marcobarilari
Copy link
Collaborator Author

running without parfor works

@Remi-Gau
Copy link
Contributor

Remi-Gau commented Apr 8, 2024

OK so I suspect that each worker in the parfor is trying to reinstall the MACs toolbox when initialising bidspm but it cannot because other workers are using it?

@Remi-Gau
Copy link
Contributor

Remi-Gau commented Apr 8, 2024

I can try to make check the MACs toolbox version (or git commit shasum) before recopying it every time, this way it would only update it if it finds that the previous version is too old...

@marcobarilari
Copy link
Collaborator Author

marcobarilari commented Apr 8, 2024 via email

@Remi-Gau
Copy link
Contributor

Remi-Gau commented Apr 9, 2024

trying a fix in #1228

@Remi-Gau
Copy link
Contributor

Remi-Gau commented Apr 9, 2024

@marcobarilari
this should be fixed: maybe trying updating bidspm and re-run things in your parfor-loop

@marcobarilari
Copy link
Collaborator Author

Hi there, yes this indeed fixed the problem. However, a similar one happens when CPP-ROI copies the atlases.

Here though is quite strange, it only happens if I run the script from terminal matlab call, while if I run the code (copy pasted) in the same terminal matlab session it work perfectly.

in the file attached what is printed in the terminal/matlab command window

20240419_error_command_window_parallelpool_atalases_installation.txt

@marcobarilari
Copy link
Collaborator Author

I see that this is a specifc problem that I might be the only one having so no need to rush or find a solution if you think it might requires too much debugging.

A workaround could be to let CPP ROI checking if the atlases are already there and skip copying in case.

@Remi-Gau
Copy link
Contributor

A workaround could be to let CPP ROI checking if the atlases are already there and skip copying in case.

yup: can you open an issue in CPP_ROI and copy paste content from those 2 last comments there ?

@marcobarilari
Copy link
Collaborator Author

done, thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug 🐛 Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants