accuracy and auc is not same as what you reported.

[fatemehghanadi]

Hello. i run your code (with LOSO=False in config.py for 10CV) but the results are not same as what you reported in ttest directory. would you please tell me if i should edit s.th? the results are below for each fold:

(base) D:\ASDcodes\fMRI-site-adaptation-master\fMRI-site-adaptation-master>python run_model.py --cfg configs/run_default.yaml
DATASET:
ATLAS: cc200
BASE_DIR: ABIDE_pcp/cpac/filt_noglobal/
DOWNLOAD: False
PHENO_FILE: ABIDE_pcp/Phenotypic_V1_0b_preprocessed1.csv
PHENO_ONLY: False
PIPELINE: cpac
QC: False
ROOT: ./data
METHOD:
ALGORITHM: Ridge
CONNECTIVITY: TPE
ENSEMBLE: False
KHSIC: True
LOSO: False
MODEL: MIDA
SEED: 1234
OUTPUT:
OUT_FILE: TPE
OUT_PATH: ./data/abide_tpe_mida_out/
ROOT: ./data
SAVE_FEATURE: True
best parameters from 5CV grid search:
{'acc': 0.7253234431602553, 'mu': 1.0, 'h': 150, 'alpha': 0.5}

Fold number: 0
Linear Accuracy: 0.6990291262135923
Linear AUC: 0.7633962264150943

best parameters from 5CV grid search:
{'acc': 0.7210683686964521, 'mu': 0.75, 'h': 50, 'alpha': 0.5}

Fold number: 1
Linear Accuracy: 0.7281553398058253
Linear AUC: 0.789433962264151

best parameters from 5CV grid search:
{'acc': 0.7189293312632972, 'mu': 0.75, 'h': 50, 'alpha': 0.25}

Fold number: 2
Linear Accuracy: 0.7087378640776699
Linear AUC: 0.7962264150943396

best parameters from 5CV grid search:
{'acc': 0.7274797308952907, 'mu': 1.0, 'h': 150, 'alpha': 0.25}

Fold number: 3
Linear Accuracy: 0.6601941747572816
Linear AUC: 0.7030188679245283

best parameters from 5CV grid search:
{'acc': 0.7167442930251278, 'mu': 1.0, 'h': 50, 'alpha': 0.25}

Fold number: 4
Linear Accuracy: 0.7572815533980582
Linear AUC: 0.869811320754717

best parameters from 5CV grid search:
{'acc': 0.7272209763670864, 'mu': 1.0, 'h': 50, 'alpha': 0.25}

Fold number: 5
Linear Accuracy: 0.6634615384615384
Linear AUC: 0.6862745098039215

best parameters from 5CV grid search:
{'acc': 0.7261572077511356, 'mu': 0.5, 'h': 50, 'alpha': 0.25}

Fold number: 6
Linear Accuracy: 0.7115384615384616
Linear AUC: 0.7639659637439882

best parameters from 5CV grid search:
{'acc': 0.7122074636306135, 'mu': 0.5, 'h': 50, 'alpha': 0.25}

Fold number: 7
Linear Accuracy: 0.7596153846153846
Linear AUC: 0.8312985571587125

best parameters from 5CV grid search:
{'acc': 0.7347708584900234, 'mu': 1.0, 'h': 50, 'alpha': 0.25}

Fold number: 8
Linear Accuracy: 0.6923076923076923
Linear AUC: 0.7299297077321495

best parameters from 5CV grid search:
{'acc': 0.7186188258294519, 'mu': 0.75, 'h': 50, 'alpha': 0.25}

Fold number: 9
Linear Accuracy: 0.7788461538461539
Linear AUC: 0.8398076211616723

accuracy average 0.7159167289021658
standard deviation accuracy 0.03807932567772887
auc average 0.7773163152053273
standard deviation auc 0.056837675351087726
KHSIC sample value: 1.24 Threshold: 0.60 p value: 0.0000000000

[kundaMwiza]

Which Python version are you using? I used Python 3.6/3.7 ~5 years ago and I get the results below on Ubuntu 18.04.

I don't know what you mean by s.th.

(venv_3.6) root@56d8edac67b8:/fmri# python3.6 run_model.py --cfg configs/run_default.yaml 
/fmri/venv_3.6/lib/python3.6/site-packages/sklearn/externals/joblib/__init__.py:15: DeprecationWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=DeprecationWarning)
DATASET:
  ATLAS: cc200
  BASE_DIR: ABIDE_pcp/cpac/filt_noglobal/
  DOWNLOAD: False
  PHENO_FILE: ABIDE_pcp/Phenotypic_V1_0b_preprocessed1.csv
  PHENO_ONLY: False
  PIPELINE: cpac
  QC: False
  ROOT: ./data
METHOD:
  ALGORITHM: Ridge
  CONNECTIVITY: TPE
  ENSEMBLE: False
  KHSIC: True
  LOSO: False
  MODEL: MIDA
  SEED: 1234
OUTPUT:
  OUT_FILE: TPE
  OUT_PATH: ./data/abide_tpe_mida_out/
  ROOT: ./data
  SAVE_FEATURE: True
best parameters from 5CV grid search: 
 {'acc': 0.7221030042918455, 'mu': 0.5, 'h': 150, 'alpha': 0.5}
----------------------------------------------------------------------------------------------------
Fold number: 0
Linear Accuracy: 0.7378640776699029
Linear AUC: 0.7762264150943397
----------------------------------------------------------------------------------------------------
best parameters from 5CV grid search: 
 {'acc': 0.7145922746781116, 'mu': 0.75, 'h': 50, 'alpha': 0.25}
----------------------------------------------------------------------------------------------------
Fold number: 1
Linear Accuracy: 0.7475728155339806
Linear AUC: 0.830943396226415
----------------------------------------------------------------------------------------------------
best parameters from 5CV grid search: 
 {'acc': 0.7199570815450643, 'mu': 0.75, 'h': 50, 'alpha': 0.25}
----------------------------------------------------------------------------------------------------
Fold number: 2
Linear Accuracy: 0.7281553398058253
Linear AUC: 0.7573584905660378
----------------------------------------------------------------------------------------------------
best parameters from 5CV grid search: 
 {'acc': 0.7242489270386266, 'mu': 0.5, 'h': 50, 'alpha': 0.25}
----------------------------------------------------------------------------------------------------
Fold number: 3
Linear Accuracy: 0.7572815533980582
Linear AUC: 0.7867924528301887
----------------------------------------------------------------------------------------------------
best parameters from 5CV grid search: 
 {'acc': 0.721030042918455, 'mu': 0.75, 'h': 50, 'alpha': 0.25}
----------------------------------------------------------------------------------------------------
Fold number: 4
Linear Accuracy: 0.6893203883495146
Linear AUC: 0.7724528301886792
----------------------------------------------------------------------------------------------------
best parameters from 5CV grid search: 
 {'acc': 0.7207303974221267, 'mu': 1.0, 'h': 50, 'alpha': 0.25}
----------------------------------------------------------------------------------------------------
Fold number: 5
Linear Accuracy: 0.7692307692307693
Linear AUC: 0.8050314465408805
----------------------------------------------------------------------------------------------------
best parameters from 5CV grid search: 
 {'acc': 0.723952738990333, 'mu': 0.5, 'h': 50, 'alpha': 0.25}
----------------------------------------------------------------------------------------------------
Fold number: 6
Linear Accuracy: 0.7115384615384616
Linear AUC: 0.80614132445431
----------------------------------------------------------------------------------------------------
best parameters from 5CV grid search: 
 {'acc': 0.7250268528464017, 'mu': 0.5, 'h': 150, 'alpha': 0.25}
----------------------------------------------------------------------------------------------------
Fold number: 7
Linear Accuracy: 0.7211538461538461
Linear AUC: 0.8091009988901221
----------------------------------------------------------------------------------------------------
best parameters from 5CV grid search: 
 {'acc': 0.7357679914070892, 'mu': 1.0, 'h': 50, 'alpha': 0.25}
----------------------------------------------------------------------------------------------------
Fold number: 8
Linear Accuracy: 0.6826923076923077
Linear AUC: 0.7243803181650018
----------------------------------------------------------------------------------------------------
best parameters from 5CV grid search: 
 {'acc': 0.7185821697099892, 'mu': 0.5, 'h': 50, 'alpha': 0.25}
----------------------------------------------------------------------------------------------------
Fold number: 9
Linear Accuracy: 0.7019230769230769
Linear AUC: 0.7447280799112098
----------------------------------------------------------------------------------------------------
accuracy average 0.7246732636295742
standard deviation accuracy 0.02733841719355403
auc average 0.7813155752867185
standard deviation auc 0.031148619498405913
KHSIC sample value: 1.24 Threshold: 0.60 p value: 0.0000000000
(venv_3.6) root@56d8edac67b8:/fmri# 

Edit:

Python 3.6/3.7 requirements

numpy==1.16.4
pandas==0.25.0
scipy==1.3.0
scikit-learn==0.21.2
nilearn==0.5.2
networkx==2.3
joblib==1.1.0
yacs