Reproducing paper results

[adnanhd]

Hi everybody!

We are trying to reproduce S-EDF-39/U-Time results in table 2 in your paper.

We followed the exact instructions as ones on the repository's readme page for that purpose. (except --N_first 39 argument instead of 6, stated on the page)

However, we are very far from your results. The tables below are what we get after evaluating the predictions.

/	Pred 0	Pred 1	Pred 2	Pred 3	Pred 4
True 0	1	135	103	1158	0
True 1	0	107	92	2	0
True 2	0	25	1193	4	0
True 3	0	0	174	27	0
True 4	0	142	201	3	0

Raw Metrics:

/	F1	Precision	Recall/Sens.
Class 0	0.00	1.00	0.00
Class 1	0.35	0.26	0.53
Class 2	0.80	0.68	0.98
Class 3	0.04	0.02	0.13
Class 4	0.00	0.00	0.00
mean	0.24	0.39	0.33

Do you have any opinion on why we couldn't replicate the results above?
Thx

[perslev]

Hi adnanhd

Just to be sure, are these the instructions that you followed?

Would you mind sending or pasting your hyperparameter file(s) and log files? Then I will have a look.

Also if you could let me know what version of tensorflow you are using.

Cheers
Mathias

[adnanhd]

Hi Mathias,
Thanks for your quick and kind reply. Here are the instructions I have followed to get the results above

# clone the repo and create an environment
git clone https://github.com/perslev/U-Time
python3 -m venv utime
source utime/bin/activate

pip install mpunet==0.2.12
# installing mpunet requirements -- This part is to suppress warnings from mpunet
# I put them in a file and used -r option to install them with pip actually
pip install h5py>=2.10.0 matplotlib>=3.3.0 nibabel>=3.1.0 pandas>=1.0.0 \
                psutil>=5.7.0 ruamel.yaml>=0.16.0 scikit-learn>=0.23.0 scipy>=1.4.0 \
               tensorflow==2.3.2 tensorflow-addons==0.12.0
pip install U-Time/

# Preparing a project directory
ut init --name demo --model usleep_demo
cd demo

# fetching datasets
ut fetch --dataset sedf_sc --out_dir data/sedf_sc --N_first 39
# I did ain't skip this part. But I am not focusing 
# on the model's scores on this dataset.
ut fetch --dataset dcsm --out_dir data/dcsm --N_first 6

# Split dataset SEDF-SC
ut cv_split --data_dir data/sedf_sc/ \
            --subject_dir_pattern 'SC*' \
            --CV 20 \
            --validation_fraction 0.10 \
            --subject_matching_regex 'SC4(\d{2}).*' \
            --seed 123
            
# Split dataset DCSM
ut cv_split --data_dir data/dcsm/ \
            --subject_dir_pattern 'tp*' \
            --CV 20 \
            --validation_fraction 0.10 \
            --seed 123

# core of the pipeline -- prepare, train, predict and eventually evaluate
ut preprocess --out_path data/processed_data.h5 --dataset_splits train_data val_data
ut train --num_GPUs=1 --preprocessed --seed 123
ut predict --num_GPUs=1 \
           --data_split test_data \
           --strip_func strip_to_match \
           --one_shot \
           --save_true \
           --majority \
           --out_dir predictions

ut cm --true 'predictions/test_data/sedf_sc/*TRUE.npy' \
      --pred 'predictions/test_data/sedf_sc/majority/*PRED.npy' \
      --ignore 5 \
      --round 2 \
      --wake_trim_min 30

Here are the files for which you asked me in the previous comment,

└─ demo
   ├─ hyperparameters
   │    ├─ hparams.yaml
   │    ├─ pre_proc_hparams.yaml
   │    └─ dataset_configurations
   │          └─ ...
   ├─ logs
   │   ├─ log.txt
   │   └─ warnings.txt
   ├─ ...

params-and-logs.zip

Lastly, the version of TensorFlow I have installed and used is 2.3.2.
I have used a docker image from tensorflow/tensorflow:2.1.0-gpu.
Those commands I shared with you above are run in the docker container, the tag of which I shared just above.

Best
Harun

[perslev]

Thanks for sharing this.

So there are two important differences in what you are doing in the above, and what you should do to recreate Table two in the U-Time paper:

1. In the above you create a U-Sleep typed model (from this follow-up paper) in the line ut init --name demo --model usleep_demo. Instead, you should follow this guide, most importantly use ut init --name demo --model utime to initialise this correct model.
2. The U-Time model should be trained only on the S-EDF-SC dataset if you want to recreate the same table. In the above, you also train on a very small sample from a different dataset. Again, following the U-Time specific guide that I posted a link to above would be the easiest.

After following the U-Time guide please let me know if you are still experiencing any issues.

Cheers
Mathias

[adnanhd]

Hi Mathias,

Sorry for my late response. I have spent the last three days replicating U-Time network results as in your paper. Since I got some errors in the pipeline above, I decided to work on a closer version of the code to the paper than the latest version, which is under the tag utime-paper-version.

However, I have got some errors in the following step

ut train --num_GPUs=1 --channels 'EEG Fpz-Cz'

The error in short is like the following

Traceback (most recent call last):
  File "/usr/local/bin/ut", line 11, in <module>
    load_entry_point('utime', 'console_scripts', 'ut')()
  File "/utime/utime/bin/ut.py", line 55, in entry_func
    mod.entry_func(parsed.args)
  File "/utime/utime/bin/train.py", line 209, in entry_func
    raise e
  File "/utime/utime/bin/train.py", line 206, in entry_func
    run(args=args, gpu_mon=gpu_mon)
  File "/utime/utime/bin/train.py", line 142, in run
    d.load(1 if args.just_one else None)
  File "/utime/utime/dataset/sleep_study_dataset.py", line 163, in load
    "traceback.".format(e.study_id)) from e
utime.errors.CouldNotLoadError: Could not load sleep study SC4411E0. Please refer to the above traceback.

I am also sending the log files generated by the execution of the code snippet above.
log.txt
warnings.txt

Could you help me fix the error above?
Thanks
Harun

[adnanhd]

If you like I can also share the error I got in the first version? But if the latter is quicker, let us go through the paper version.

[perslev]

Hi adnanhd

Sorry for my late reply, I am on vacation these days. The above error occurs when a file is not able to be loaded from disk. I will need to see the full traceback to know more about why the loading failed.

If you don't mind, please also send me the errors you experienced on the latest version.