Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AMICA not working on NSG server #24

Open
moo-n opened this issue Oct 26, 2020 · 6 comments
Open

AMICA not working on NSG server #24

moo-n opened this issue Oct 26, 2020 · 6 comments

Comments

@moo-n
Copy link

moo-n commented Oct 26, 2020

Hello, I've been trying to run AMICA on NSG server but it doesn't work. Attached the code I used to run it and the outputs I got.
the dataset I'm using is epoched.
I tried both running it using one node and using multiple nodes by setting numprocs=4 and both failed.
the input file:

%% test_script_amica_nsg.m
% add eeglab to path
eeglab; close;
 
% load dataset 
filepath = [ pwd filesep ];
filename = '23456.set';
EEG = pop_loadset(filename, filepath);
 
% define parameters
numprocs = 1;       % # of nodes (1-4: default 1)
max_threads = 24;   % # of threads (1-24: default = 24)
num_models = 1;     % # of models of mixture ICA 
max_iter = 3000;    % max number of learning steps
 
% run amica on NSG
outdir = [ pwd filesep 'amicaouttmp' filesep ];
runamica15_nsg(EEG, 'num_models',num_models, 'outdir',outdir, 'max_iter',max_iter);

output files STDOUT

                            < M A T L A B (R) >
                  Copyright 1984-2019 The MathWorks, Inc.
              R2019b Update 3 (9.7.0.1261785) 64-bit (glnxa64)
                             November 27, 2019

 
To get started, type doc.
For product information, visit www.mathworks.com.
 
[�Warning: Adding folders named 'resources' to the path is not supported:
/projects/ps-nsg/home/nsguser/applications_centos7/eeglab_new/functions/resources]� 
[�> In path (line 109)
  In addpath (line 86)
  In eeglab>addpathifnotinlist (line 2053)
  In eeglab>myaddpath (line 2082)
  In eeglab (line 287)
  In runAmica (line 3)]� 
eeglab: options file is ~/eeg_options.m
EEGLAB: adding "ANTeepimport" v1.13 (see >> help eegplugin_eepimport)
EEGLAB: adding "BCI2000import" v0.36 (see >> help eegplugin_BCI2000import)
EEGLAB: adding "BDFimport" v1.10 (see >> help eegplugin_bdfimport)
EEGLAB: adding "Biosig" to the path; subfolders (if any) might be missing from the path
EEGLAB: adding "Fieldtrip-lite" to the path; subfolders (if any) might be missing from the path
EEGLAB: adding "Fileio" to the path; subfolders (if any) might be missing from the path
EEGLAB: adding "MEF_import" v1.10 (see >> help eegplugin_mefimport)
EEGLAB: adding "NEUROPRAXimport" v1.00 (see >> help eegplugin_NEUROPRAXimport)
EEGLAB: adding "WearableSensing" v1.09 (see >> help eegplugin_pop_WearableSensing)
EEGLAB: adding "amica" v1.5.1 (see >> help eegplugin_amica)
EEGLAB: adding "bci2000legacy" v1.0 (see >> help eegplugin_bci2000legacy)
EEGLAB: adding "bdfimport" v1.1 (see >> help eegplugin_bdfimport)
EEGLAB: adding "biopac" v1.00 (see >> help eegplugin_biopac)
EEGLAB: adding "bva-io-" v1.5.13 (see >> help eegplugin_bva_io)
EEGLAB: adding "bva-io" v1.5.13 (see >> help eegplugin_bva_io)
EEGLAB: adding "cogniscan" v1.1 (see >> help eegplugin_cogniscan)
EEGLAB: adding "corrmap" v2.02.1 (see >> help eegplugin_corrmap)
EEGLAB: adding "ctfimport" v1.04 (see >> help eegplugin_ctfimport)
EEGLAB: adding "dipfit" v2.3 (see >> help eegplugin_dipfit)
EEGLAB: adding "egilegacy" v1.0 (see >> help eegplugin_egilegacy)
EEGLAB: adding "erpssimport" v1.01 (see >> help eegplugin_erpssimport)
EEGLAB: adding "firfilt" v1.6.2 (see >> help eegplugin_firfilt)
EEGLAB: adding "gtecimport" v3.15.01 (see >> help eegplugin_gtecimport)
EEGLAB: adding "import_NihonKodenM00files" v1.11 (see >> help eegplugin_importNihonKodenM00)
EEGLAB: adding "instepascimport" v1.00 (see >> help eegplugin_ascinstep)
EEGLAB: adding "loadcurry" v2.0 (see >> help eegplugin_loadcurry)
EEGLAB: adding "loadhdf5v" v1.1 (see >> help eegplugin_loadhdf5)
EEGLAB: error while adding plugin "eegplugin_mffimport"
   Error using error
Function is not defined for 'cell' inputs.
EEGLAB: adding "mffmatlabio" v2.01 (see >> help eegplugin_mffmatlabio)
EEGLAB: adding "musemonitor" v3.1 (see >> help eegplugin_musemonitor)
EEGLAB: adding "neuroimaging4d" v1.00 (see >> help eegplugin_4dneuroimaging)
EEGLAB: adding "neuroneimport" v1.0.3.4 (see >> help eegplugin_neurone)
EEGLAB: adding "neuroscanio" v1.1 (see >> help eegplugin_neuroscanio)
EEGLAB: adding "procominfinity" v1.00 (see >> help eegplugin_procom)
EEGLAB: adding "snapmaster" v1.0 (see >> help eegplugin_snapmaster)
EEGLAB: adding "xdf-EEGLAB" v1.14 (see >> help eegplugin_xdfimport)
[�Warning: Adding folders named 'resources' to the path is not supported:
/projects/ps-nsg/home/nsguser/applications_centos7/eeglab_new/functions/resources]� 
[�> In path (line 109)
  In addpath (line 86)
  In eeglab>addpathifnotinlist (line 2053)
  In eeglab>myaddpath (line 2082)
  In eeglab (line 287)
  In eeglab (line 1084)
  In runAmica (line 3)]� 
pop_loadset(): loading file /projects/ps-nsg/home/nsguser/ngbw/workspace/NGBW-JOB-EEGLAB_TG-985C5F6F32174E75A1931EDA1AD2E76B/12456/12456.set ...
Running AMICA on single node ...
INTERNAL ERROR: invalid error code ffffffff (Ring Index out of range) in rdma_cm_get_hostnames:1267
Warning: RDMA CM Initialization failed. Continuing without RDMA CM support. Please set MV2_USE_RDMA_CM=0 to disable RDMA CM.
           1 processor name = comet-18-44.sdsc.edu
           1 host_num =   1457401042
 This is MPI process           1 of           1 ; I am process           1 of
           1 on node: comet-18-44.sdsc.edu
           1  : node root process           1 of           1
Processing arguments ...
forrtl: severe (66): output statement overflows record, unit -5, file Internal Formatted Write
Image              PC                Routine            Line        Source             
amica15c           0000000000E88B8B  Unknown               Unknown  Unknown
amica15c           0000000000E86B97  Unknown               Unknown  Unknown
amica15c           000000000044D799  Unknown               Unknown  Unknown
amica15c           000000000040CD6C  Unknown               Unknown  Unknown
amica15c           000000000040A6B6  Unknown               Unknown  Unknown
libc.so.6          00002B5285D823D5  Unknown               Unknown  Unknown
amica15c           000000000040A529  Unknown               Unknown  Unknown
Oops! AMICA did not run succesfully...
`

STDERR
`

{�Unrecognized function or variable 'filename'.

Error in runamica15_nsg (line 922)
if exist(filename,'file')

Error in runAmica (line 18)
runamica15_nsg(EEG, 'num_models',num_models, 'outdir',outdir,
'max_iter',max_iter);
}� 

`

@japalmer29
Copy link
Collaborator

japalmer29 commented Oct 26, 2020 via email

@moo-n
Copy link
Author

moo-n commented Oct 26, 2020

Are you sure you're using pop_loadset correctly? I think you need keywords, like EEG=pop_loadset('filename',filename,'filepath',filepath);
.

I just tried the command and it loads the set just fine.

@japalmer29
Copy link
Collaborator

japalmer29 commented Oct 26, 2020 via email

@moo-n
Copy link
Author

moo-n commented Oct 26, 2020

This time it was one file .set.
I will use two files and let you know.

@moo-n
Copy link
Author

moo-n commented Oct 28, 2020

An update:
this is the code I used:

%% test_script_amica_nsg.m
% add eeglab to path
eeglab; close;
 
% load dataset 
filepath = [ pwd filesep ];
filename = '23456.set';
EEG = pop_loadset('filename',filename,'filepath',filepath);
 
% define parameters
numprocs = 4;       % # of nodes (1-4: default 1)
max_threads = 24;   % # of threads (1-24: default = 24)
num_models = 1;     % # of models of mixture ICA 
max_iter = 3000;    % max number of learning steps
 
% run amica on NSG
outdir = [ pwd filesep 'amicaouttmp' filesep ];
runamica15_nsg(EEG, 'numprocs', numprocs,'num_models',num_models, 'outdir',outdir, 'max_iter',max_iter);

the output from the cluster machine:
stdout.txt


                            < M A T L A B (R) >
                  Copyright 1984-2019 The MathWorks, Inc.
              R2019b Update 3 (9.7.0.1261785) 64-bit (glnxa64)
                             November 27, 2019

 
To get started, type doc.
For product information, visit www.mathworks.com.
 
[�Warning: Adding folders named 'resources' to the path is not supported:
/projects/ps-nsg/home/nsguser/applications_centos7/eeglab_new/functions/resources]� 
[�> In path (line 109)
  In addpath (line 86)
  In eeglab>addpathifnotinlist (line 2053)
  In eeglab>myaddpath (line 2082)
  In eeglab (line 287)
  In runAmicanodes (line 3)]� 
eeglab: options file is ~/eeg_options.m
EEGLAB: adding "ANTeepimport" v1.13 (see >> help eegplugin_eepimport)
EEGLAB: adding "BCI2000import" v0.36 (see >> help eegplugin_BCI2000import)
EEGLAB: adding "BDFimport" v1.10 (see >> help eegplugin_bdfimport)
EEGLAB: adding "Biosig" to the path; subfolders (if any) might be missing from the path
EEGLAB: adding "Fieldtrip-lite" to the path; subfolders (if any) might be missing from the path
EEGLAB: adding "Fileio" to the path; subfolders (if any) might be missing from the path
EEGLAB: adding "MEF_import" v1.10 (see >> help eegplugin_mefimport)
EEGLAB: adding "NEUROPRAXimport" v1.00 (see >> help eegplugin_NEUROPRAXimport)
EEGLAB: adding "WearableSensing" v1.09 (see >> help eegplugin_pop_WearableSensing)
EEGLAB: adding "amica" v1.5.1 (see >> help eegplugin_amica)
EEGLAB: adding "bci2000legacy" v1.0 (see >> help eegplugin_bci2000legacy)
EEGLAB: adding "bdfimport" v1.1 (see >> help eegplugin_bdfimport)
EEGLAB: adding "biopac" v1.00 (see >> help eegplugin_biopac)
EEGLAB: adding "bva-io-" v1.5.13 (see >> help eegplugin_bva_io)
EEGLAB: adding "bva-io" v1.5.13 (see >> help eegplugin_bva_io)
EEGLAB: adding "cogniscan" v1.1 (see >> help eegplugin_cogniscan)
EEGLAB: adding "corrmap" v2.02.1 (see >> help eegplugin_corrmap)
EEGLAB: adding "ctfimport" v1.04 (see >> help eegplugin_ctfimport)
EEGLAB: adding "dipfit" v2.3 (see >> help eegplugin_dipfit)
EEGLAB: adding "egilegacy" v1.0 (see >> help eegplugin_egilegacy)
EEGLAB: adding "erpssimport" v1.01 (see >> help eegplugin_erpssimport)
EEGLAB: adding "firfilt" v1.6.2 (see >> help eegplugin_firfilt)
EEGLAB: adding "gtecimport" v3.15.01 (see >> help eegplugin_gtecimport)
EEGLAB: adding "import_NihonKodenM00files" v1.11 (see >> help eegplugin_importNihonKodenM00)
EEGLAB: adding "instepascimport" v1.00 (see >> help eegplugin_ascinstep)
EEGLAB: adding "loadcurry" v2.0 (see >> help eegplugin_loadcurry)
EEGLAB: adding "loadhdf5v" v1.1 (see >> help eegplugin_loadhdf5)
EEGLAB: error while adding plugin "eegplugin_mffimport"
   Error using error
Function is not defined for 'cell' inputs.
EEGLAB: adding "mffmatlabio" v2.01 (see >> help eegplugin_mffmatlabio)
EEGLAB: adding "musemonitor" v3.1 (see >> help eegplugin_musemonitor)
EEGLAB: adding "neuroimaging4d" v1.00 (see >> help eegplugin_4dneuroimaging)
EEGLAB: adding "neuroneimport" v1.0.3.4 (see >> help eegplugin_neurone)
EEGLAB: adding "neuroscanio" v1.1 (see >> help eegplugin_neuroscanio)
EEGLAB: adding "procominfinity" v1.00 (see >> help eegplugin_procom)
EEGLAB: adding "snapmaster" v1.0 (see >> help eegplugin_snapmaster)
EEGLAB: adding "xdf-EEGLAB" v1.14 (see >> help eegplugin_xdfimport)
[�Warning: Adding folders named 'resources' to the path is not supported:
/projects/ps-nsg/home/nsguser/applications_centos7/eeglab_new/functions/resources]� 
[�> In path (line 109)
  In addpath (line 86)
  In eeglab>addpathifnotinlist (line 2053)
  In eeglab>myaddpath (line 2082)
  In eeglab (line 287)
  In eeglab (line 1084)
  In runAmicanodes (line 3)]� 
pop_loadset(): loading file /projects/ps-nsg/home/nsguser/ngbw/workspace/NGBW-JOB-EEGLAB_TG-DCA1920B6CF94C7F9E354186B73EBEDF/23456/23456.set ...
Reading float file '/projects/ps-nsg/home/nsguser/ngbw/workspace/NGBW-JOB-EEGLAB_TG-DCA1920B6CF94C7F9E354186B73EBEDF/23456/23456.fdt'...
Running AMICA across multiple nodes ...
IBRUN WARNING: Your job geometry appears to suggest that you want 1 threads per MPI rank, but your OMP_NUM_THREADS is 24.  This will cause thread binding to be disabled.
           1 processor name = comet-21-47.sdsc.edu
           1 host_num =    498679133
           2 processor name = comet-21-48.sdsc.edu
           2 host_num =    695192638
 This is MPI process           1 of           4 ; I am process           1 of
           1 on node: comet-21-47.sdsc.edu
           1  : node root process           1 of           4
           4 processor name = comet-21-52.sdsc.edu
           4 host_num =   1313062967
 This is MPI process           4 of           4 ; I am process           1 of
           1 on node: comet-21-52.sdsc.edu
Processing arguments ...
 This is MPI process           2 of           4 ; I am process           1 of
           1 on node: comet-21-48.sdsc.edu
           2  : node root process           2 of           4
           4  : node root process           4 of           4
           3 processor name = comet-21-50.sdsc.edu
           3 host_num =    920035957
 This is MPI process           3 of           4 ; I am process           1 of
           1 on node: comet-21-50.sdsc.edu
           3  : node root process           3 of           4
 num_files =            1
 FILES: 
 /projects/ps-nsg/home/nsguser/ngbw/workspace/NGBW-JOB-EEGLAB_TG-DCA1920B6CF94C7
 F9E354186B73EBEDF/23456//23456.fdt
 num_dir_files =            1
 initial matrix block_size =          128
 do_opt_block =            0
 blk_min =          256
 blk_step =          256
 blk_max =         1024
 number of models =            1
 max_thrds =           24
 use_min_dll =            1
 min dll =   1.000000000000000E-009
 use_grad_norm =            1
 min grad norm =   1.000000000000000E-007
 number of density mixture components =            3
 pdf type =            0
 max_iter =         3000
 num_samples =            1
 data_dim =          127
 field_dim =      4915200
 do_history =            0
 histstep =           10
 share_comps =            0
 share_start =          100
 comp_thresh =   0.990000000000000     
 share_int =          100
 initial lrate =   5.000000000000000E-002
 minimum lrate =   1.000000000000000E-008
 minimum data covariance eigenvalue =   1.000000000000000E-012
 lrate factor =   0.500000000000000     
 initial rholrate =   5.000000000000000E-002
 rho0 =    1.50000000000000     
 min rho =    1.00000000000000     
 max rho =    2.00000000000000     
 rho lrate factor =   0.500000000000000     
 kurt_start =            3
 num kurt =            5
 kurt interval =            1
 do_newton =            1
 newt_start =           50
 newt_ramp =           10
 initial newton lrate =    1.00000000000000     
 do_reject =            0
 num reject =            3
 reject sigma =    3.00000000000000     
 reject start =            2
 reject interval =            3
 write step =           20
 write_nd =            0
 write_LLt =            1
 dec window =            1
 max_decs =            3
 fix_init =            0
 update_A =            1
 update_c =            1
 update_gm =            1
 update_alpha =            1
 update_mu =            1
 update_beta =            1
 invsigmax =    100.000000000000     
 invsigmin =   0.000000000000000E+000
 do_rho =            1
 load_rej =            0
 load_c =            0
 load_gm =            0
 load_alpha =            0
 load_mu =            0
 load_beta =            0
 load_rho =            0
 load_comp_list =            0
 do_mean =            1
 do_sphere =            1
 pcakeep =          127
 pcadb =    30.0000000000000     
 byte_size =            4
 doscaling =            1
 scalestep =            1
mkdir: cannot create directory ‘/projects/ps-nsg/home/nsguser/ngbw/workspace/NGBW-JOB-EEGLAB_TG-DCA1920B6CF94C7F9E354186B73EBEDF/23456/amicaouttmp/’: File exists
 output directory = 
 /projects/ps-nsg/home/nsguser/ngbw/workspace/NGBW-JOB-EEGLAB_TG-DCA1920B6CF94C7
 F9E354186B73EBEDF/23456/amicaouttmp/
           2 : setting num_thrds to           24  ...
           3 : setting num_thrds to           24  ...
           1 : setting num_thrds to           24  ...
           4 : setting num_thrds to           24  ...
           4 : using          24 threads.
           1 : using          24 threads.
           2 : using          24 threads.
           3 : using          24 threads.
           1 : node_thrds =           24          24          24          24
 bytes in real =            1
           1 : REAL nbyte =            1
 getting segment list ...
 blocks in sample =      4915200
 total blocks =      4915200
 node blocks =      1228800     1228800     1228800     1228800
 node            1  start: file            1  sample            1  index 
           1
 node            1  stop : file            1  sample            1  index 
     1228800
 node            2  start: file            1  sample            1  index 
     1228801
 node            2  stop : file            1  sample            1  index 
     2457600
 node            3  start: file            1  sample            1  index 
     2457601
 node            3  stop : file            1  sample            1  index 
     3686400
 node            4  start: file            1  sample            1  index 
     3686401
 node            4  stop : file            1  sample            1  index 
     4915200
           1 : data =    11.9744024276733        4.24514341354370     
           3 : data =   -7.78323650360107      -0.299957871437073     
           2 : data =    1.66699647903442       -26.0488491058350     
           4 : data =   -2.59526801109314       -6.62773466110229     
 getting the mean ...
  mean =   -1.36149215149254      -0.697963101435924     
 -0.908440884302629     
 subtracting the mean ...
 getting the covariance matrix ...
 cnt =      4915200
 doing eig nx =          127  lwork =       161290
 doing eig nx =          127  lwork =       161290
 doing eig nx =          127  lwork =       161290
 doing eig nx =          127  lwork =       161290
 minimum eigenvalues =   2.439452986291257E-002   2.24693689752528     
   2.29249564029623     
 minimum eigenvalues =   2.439452986291257E-002   2.24693689752528     
   2.29249564029623     
 maximum eigenvalues =    13003.3767440951        1393.52042042707     
   1052.95147913640     
 num eigs kept =            0
 maximum eigenvalues =    13003.3767440951        1393.52042042707     
   1052.95147913640     
 num eigs kept =          127
 getting the sphering matrix ...
 minimum eigenvalues =   2.439452986291257E-002   2.24693689752528     
   2.29249564029623     
 maximum eigenvalues =    13003.3767440951        1393.52042042707     
   1052.95147913640     
 num eigs kept =          127
 sphering the data ...
 minimum eigenvalues =   2.439452986291257E-002   2.24693689752528     
   2.29249564029623     
 maximum eigenvalues =    13003.3767440951        1393.52042042707     
   1052.95147913640     
 num eigs kept =            0
 numeigs =          127

@japalmer29
Copy link
Collaborator

japalmer29 commented Oct 28, 2020 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants