Error when extracting data

[farhantandia]

hey, i just have an assignment from my class about deeplearning, i choose your topic for my assignment. I have done the way in the readme file, but there is an error when extracting the EEG data.
I use google colab for simulation your code.

!python2 drive/deepsleepnet-master/prepare_physionet.py --data_dir drive/deepsleepnet-master/data --output_dir drive/deepsleepnet-master/data/eeg_fpz_cz --select_ch 'EEG Fpz-Cz' !python2 drive/deepsleepnet-master/prepare_physionet.py --data_dir drive/deepsleepnet-master/data --output_dir drive/deepsleepnet-master/data/eeg_pz_oz --select_ch 'EEG Pz-Oz'

the error is
Extracting EDF parameters from drive/deepsleepnet-master/data/SC4001E0-PSG.edf...
EDF file detected
Setting channel info structure...
Creating raw.info structure...
Reading 0 ... 7949999 = 0.000 ... 79499.990 secs...
Traceback (most recent call last):
File "drive/deepsleepnet-master/prepare_physionet.py", line 217, in
main()
File "drive/deepsleepnet-master/prepare_physionet.py", line 95, in main
raw_ch_df = raw.to_data_frame(scale_time=100)[select_ch]
TypeError: to_data_frame() got an unexpected keyword argument 'scale_time'
Extracting EDF parameters from drive/deepsleepnet-master/data/SC4001E0-PSG.edf...
EDF file detected
Setting channel info structure...
Creating raw.info structure...
Reading 0 ... 7949999 = 0.000 ... 79499.990 secs...
Traceback (most recent call last):
File "drive/deepsleepnet-master/prepare_physionet.py", line 217, in
main()
File "drive/deepsleepnet-master/prepare_physionet.py", line 95, in main
raw_ch_df = raw.to_data_frame(scale_time=100)[select_ch]
TypeError: to_data_frame() got an unexpected keyword argument 'scale_time

when i removed the scale time, the code can working but another error like this happen.
..........................
Include onset:50550.0, duration:480.0, label:4 (Sleep stage R)
Include onset:51030.0, duration:33030.0, label:0 (Sleep stage W)
Remove onset:84060.0, duration:2340.0, label:5 (Sleep stage ?)
before remove unwanted: (675000,)
after remove unwanted: (675000,)
before intersect label: (675000,)
after intersect label: (675000,)
before remove extra labels: (675000,), (2802,)
after remove extra labels: (0,), (0,)
Traceback (most recent call last):
File "drive/deepsleepnet-master/prepare_physionet.py", line 217, in
main()
File "drive/deepsleepnet-master/prepare_physionet.py", line 183, in main
x = np.asarray(np.split(raw_ch, n_epochs)).astype(np.float32)
File "/usr/local/lib/python2.7/dist-packages/numpy/lib/shape_base.py", line 557, in split
if N % sections:
ZeroDivisionError: float modulo

can you explain why?
sorry im new in AI or deeplearning topics :)
thank you.

[akaraspt]

Which Python version are you using?

[farhantandia]

python in /usr/lib/python2.7/lib-dynload (2.7.14)

[akaraspt]

Can you re-download the data again using the script provided in the data directory?
Also, change your current directory to be under the deepsleepnet-master directory before running the prepare_physionet.py.

[farhantandia]

ok ill download it again, but what's the difference?
and my current directory was under the deepsleepnet-master but in google we using drive before deepsleep folder, is it right?

i got this when downloading the data, there are some 0% of data download , is it right?
..............................
TTP request sent, awaiting response... 200 OK
Length: 49614848 (47M) [application/octet-stream]
Saving to: ‘SC4101E0-PSG.edf’

SC4101E0-PSG.edf 0%[ ] 320.00K 295KB/s SC4101E0-PSG.edf 100%[===================>] 47.32M 226KB/s in 2m 59s

2018-05-15 02:49:39 (271 KB/s) - ‘SC4101E0-PSG.edf’ saved [49614848/49614848]
.......................................

[akaraspt]

The reason for re-downloading is to make sure your downloaded files are not corrupted.

I am trying to find the root cause because I cannot reproduce the errors that you got. Can you run "pip list" and show me your installed packages?

[farhantandia]

many of data size is only 4-7kb?
Btw How much file of data that actually in use for this project? In my data folder there are 125 files..

Package Version

---

absl-py 0.2.0
astor 0.6.2
backports-abc 0.5
backports.functools-lru-cache 1.5
backports.shutil-get-terminal-size 1.0.0
backports.weakref 1.0.post1
beautifulsoup4 4.6.0
bleach 1.5.0
cachetools 2.0.1
certifi 2018.4.16
chardet 3.0.4
configparser 3.5.0
crcmod 1.7
cycler 0.10.0
decorator 4.3.0
entrypoints 0.2.3
enum34 1.1.6
funcsigs 1.0.2
functools32 3.2.3.post2
future 0.16.0
futures 3.2.0
gast 0.2.0
google-api-core 1.1.2
google-api-python-client 1.6.7
google-auth 1.4.1
google-auth-httplib2 0.0.3
google-auth-oauthlib 0.2.0
google-cloud-bigquery 1.1.0
google-cloud-core 0.28.1
google-cloud-language 1.0.2
google-cloud-storage 1.8.0
google-cloud-translate 1.3.1
google-colab 0.0.1a1
google-resumable-media 0.3.1
googleapis-common-protos 1.5.3
grpcio 1.12.0
h5py 2.7.1
html5lib 0.9999999
httplib2 0.11.3
idna 2.6
ipykernel 4.6.1
ipython 5.5.0
ipython-genutils 0.2.0
Jinja2 2.10
jsonschema 2.6.0
jupyter-client 5.2.3
jupyter-core 4.4.0
Keras 2.1.6
Markdown 2.6.11
MarkupSafe 1.0
matplotlib 2.1.2
mistune 0.8.3
mne 0.16.1
mock 2.0.0
mpmath 1.0.0
nbconvert 5.3.1
nbformat 4.4.0
networkx 2.1
nltk 3.2.5
notebook 5.2.2
numpy 1.14.3
oauth2client 4.1.2
oauthlib 2.0.7
olefile 0.45.1
opencv-python 3.4.0.12
pandas 0.22.0
pandas-gbq 0.4.1
pandocfilters 1.4.2
pathlib2 2.3.2
patsy 0.5.0
pbr 4.0.3
pexpect 4.5.0
pickleshare 0.7.4
Pillow 4.0.0
pip 10.0.1
plotly 1.12.12
portpicker 1.2.0
prompt-toolkit 1.0.15
protobuf 3.5.2.post1
psutil 5.4.5
ptyprocess 0.5.2
pyasn1 0.4.2
pyasn1-modules 0.2.1
pycurl 7.43.0
Pygments 2.1.3
pyparsing 2.2.0
python-apt 1.4.0b3
python-dateutil 2.5.3
pytz 2018.4
PyWavelets 0.5.2
PyYAML 3.12
pyzmq 16.0.4
requests 2.18.4
requests-oauthlib 0.8.0
rsa 3.4.2
scandir 1.7
scikit-image 0.13.1
scikit-learn 0.19.1
scipy 0.19.1
seaborn 0.7.1
setuptools 39.1.0
simplegeneric 0.8.1
singledispatch 3.4.0.3
six 1.11.0
statsmodels 0.8.0
subprocess32 3.5.0
sympy 1.1.1
tensorboard 1.7.0
tensorflow 0.12.1
termcolor 1.1.0
terminado 0.8.1
testpath 0.3.1
tornado 4.5.3
traitlets 4.3.2
uritemplate 3.0.0
urllib3 1.22
wcwidth 0.1.7
webencodings 0.5.1
Werkzeug 0.14.1
wheel 0.31.0
xgboost 0.7.post4

[bhargav191098]

I'm getting the same error!
Extracting EDF parameters from data/SC4001E0-PSG.edf...
EDF file detected
Setting channel info structure...
Creating raw.info structure...
Reading 0 ... 7949999 = 0.000 ... 79499.990 secs...
Traceback (most recent call last):
File "prepare_physionet.py", line 218, in
main()
File "prepare_physionet.py", line 96, in main
raw_ch_df = raw.to_data_frame(scale_time=100.0)[select_ch]
TypeError: to_data_frame() got an unexpected keyword argument 'scale_time'
Extracting EDF parameters from data/SC4001E0-PSG.edf...
EDF file detected
Setting channel info structure...
Creating raw.info structure...
Reading 0 ... 7949999 = 0.000 ... 79499.990 secs...
Traceback (most recent call last):
File "prepare_physionet.py", line 218, in
main()
File "prepare_physionet.py", line 96, in main
raw_ch_df = raw.to_data_frame(scale_time=100.0)[select_ch]
TypeError: to_data_frame() got an unexpected keyword argument 'scale_time'

[farhantandia]

hey, i just found the error, just change scale_time to scaling_time,

[farhantandia]

hey there is another error when start pre training
!python2 train.py --data_dir data/eeg_fpz_cz --output_dir output --n_folds 20 --fold_idx 0 --pretrain_epochs 100 --finetune_epochs 200 --resume False

Network (layers=19)
inputs (x_train_inputs:0): (100, 3000, 1, 1)
targets (x_train_targets:0): (100,)
l1_conv (deepfeaturenet/l1_conv/relu:0): (100, 500, 1, 64)
l2_pool (deepfeaturenet/l2_pool:0): (100, 63, 1, 64)
l3_dropout (deepfeaturenet/l3_dropout/mul:0): (100, 63, 1, 64)
l4_conv (deepfeaturenet/l4_conv/relu:0): (100, 63, 1, 128)
l5_conv (deepfeaturenet/l5_conv/relu:0): (100, 63, 1, 128)
l6_conv (deepfeaturenet/l6_conv/relu:0): (100, 63, 1, 128)
l7_pool (deepfeaturenet/l7_pool:0): (100, 16, 1, 128)
l8_flat (deepfeaturenet/l8_flat:0): (100, 2048)
l9_conv (deepfeaturenet/l9_conv/relu:0): (100, 60, 1, 64)
l10_pool (deepfeaturenet/l10_pool:0): (100, 15, 1, 64)
l11_dropout (deepfeaturenet/l11_dropout/mul:0): (100, 15, 1, 64)
l12_conv (deepfeaturenet/l12_conv/relu:0): (100, 15, 1, 128)
l13_conv (deepfeaturenet/l13_conv/relu:0): (100, 15, 1, 128)
l14_conv (deepfeaturenet/l14_conv/relu:0): (100, 15, 1, 128)
l15_pool (deepfeaturenet/l15_pool:0): (100, 8, 1, 128)
l16_flat (deepfeaturenet/l16_flat:0): (100, 1024)
l17_concat (deepfeaturenet/l17_concat:0): (100, 3072)
l18_dropout (deepfeaturenet/l18_dropout/mul:0): (100, 3072)
l19_softmax_linear (deepfeaturenet/l19_softmax_linear/Add:0): (100, 5)

[2018-05-19 23:17:53.482137] Start pre-training ...

========== [Fold-0] ==========

Load training set:
Loading data/eeg_fpz_cz/SC4011E0.npz ...
Loading data/eeg_fpz_cz/SC4012E0.npz ...
Loading data/eeg_fpz_cz/SC4021E0.npz ...
Loading data/eeg_fpz_cz/SC4022E0.npz ...
Loading data/eeg_fpz_cz/SC4031E0.npz ...
Loading data/eeg_fpz_cz/SC4032E0.npz ...
Loading data/eeg_fpz_cz/SC4041E0.npz ...
Loading data/eeg_fpz_cz/SC4042E0.npz ...
Loading data/eeg_fpz_cz/SC4051E0.npz ...
Traceback (most recent call last):
File "train.py", line 95, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 43, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "train.py", line 86, in main
n_epochs=FLAGS.pretrain_epochs
File "train.py", line 48, in pretrain
resume=FLAGS.resume
File "/content/drive/deepsleepnet-master/deepsleep/trainer.py", line 309, in train
x_train, y_train, x_valid, y_valid = data_loader.load_train_data()
File "/content/drive/deepsleepnet-master/deepsleep/data_loader.py", line 109, in load_train_data
data_train, label_train = self._load_npz_list_files(npz_files=train_files)
File "/content/drive/deepsleepnet-master/deepsleep/data_loader.py", line 33, in _load_npz_list_files
tmp_data, tmp_labels, sampling_rate = self._load_npz_file(npz_f)
File "/content/drive/deepsleepnet-master/deepsleep/data_loader.py", line 21, in _load_npz_file
data = f["x"]
File "/usr/local/lib/python2.7/dist-packages/numpy/lib/npyio.py", line 235, in getitem
pickle_kwargs=self.pickle_kwargs)
File "/usr/local/lib/python2.7/dist-packages/numpy/lib/format.py", line 683, in read_array
data = _read_bytes(fp, read_size, "array data")
File "/usr/local/lib/python2.7/dist-packages/numpy/lib/format.py", line 830, in _read_bytes
raise ValueError(msg % (error_template, size, len(data)))
ValueError: EOF: reading array data, expected 199680 bytes got 196480

can you help?

[akaraspt]

@farhantandia @bhargav191098

My guess is that there have been some updates to the pandas and/or numpy packages, which affects the code that I used several months ago.

Can you have a try with the following version:

numpy (1.11.1)
pandas (0.18.1)

and let me know?

Thanks