NameError:

[Muralidare]

when i run 2_process_patches.py, the error is NameError: name 'config' is not defined

[jasonwei20]

@JosephDiPalma

[JosephDiPalma]

@Muralidare I just tested 2_process_patches.py and did not have the error. How are you running it? Are you using the run_all.sh script or python 2_process_patches.py?

[Muralidare]

when i run 2_process_patches.py, i got error like this ValueError: max() arg is an empty sequence

[Muralidare]

what are all the folders i should create? and which folder should have input images? please guide me to solve this issue

[Muralidare]

when i run 1_split.py, its shows

############### CONFIGURATION ###############
all_wsi: all_wsi
val_wsi_per_class: 20
test_wsi_per_class: 30
keep_orig_copy: True
num_workers: 8
patch_size: 224
wsi_train: wsi_train
wsi_val: wsi_val
wsi_test: wsi_test
labels_train: labels_train.csv
labels_val: labels_val.csv
labels_test: labels_test.csv
train_folder: train_folder
patches_eval_train: patches_eval_train
patches_eval_val: patches_eval_val
patches_eval_test: patches_eval_test
num_train_per_class: 80000
type_histopath: True
purple_threshold: 100
purple_scale_size: 15
slide_overlap: 3
gen_val_patches_overlap_factor: 1.5
image_ext: jpg
by_folder: True
color_jitter_brightness: 0.5
color_jitter_contrast: 0.5
color_jitter_saturation: 0.5
color_jitter_hue: 0.2
num_epochs: 20
num_layers: 18
learning_rate: 0.001
batch_size: 16
weight_decay: 0.0001
learning_rate_decay: 0.85
resume_checkpoint: False
save_interval: 1
checkpoints_folder: checkpoints
checkpoint_file: xyz.pt
pretrain: False
log_folder: logs
auto_select: True
preds_train: preds_train
preds_val: preds_val
preds_test: preds_test
inference_train: inference_train
inference_val: inference_val
inference_test: inference_test
vis_train: vis_train
vis_val: vis_val
vis_test: vis_testdevice: cpu
classes: []
num_classes: 0
train_patches: train_folder/train
val_patches: train_folder/val
path_mean: [-33905246208.0, 4.577901953102745e-41, -33905246208.0]
path_std: [nan, 6.766019543915173e-21, nan]
resume_checkpoint_path: checkpoints/xyz.pt
log_csv: logs/log_12252019_14726.csv
eval_model: checkpoints/xyz.pt
threshold_search: (0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9)
colors: ('red', 'white', 'blue', 'green', 'purple', 'orange', 'black', 'pink', 'yellow')

#####################################################

+++++ Running 1_split.py +++++
+++++ Finished running 1_split.py +++++

[Muralidare]

did i miss something?

[JosephDiPalma]

@Muralidare

what are all the folders i should create? and which folder should have input images? please guide me to solve this issue

The code directory should contain a folder called all_wsi containing the WSIs split by class. If you have classes X, Y, and Z, then all_wsi should have subfolders named X, Y, and Z containing the WSIs in either JPG or PNG format (other formats may work but were not tested). I believe your current problem is that no WSIs are being found since in the above configuration printout shows classes: [] and num_classes: 0.

when i run 2_process_patches.py, i got error like this ValueError: max() arg is an empty sequence

This error occurs since the WSIs are not split into the classes correctly. Once you fix the directory issue, this error should not occur.

[Muralidare]

i created class1 and class2 folders in all_wsi folder with three .jpg files in both class folders. and it creates three folders in code directory called wsi_train, wsi_test, wsi_val. in that folders there are 2 folders class1 and class2 but all the folders are empty and i got error like :

File "/home/asus/Desktop/deepslide-master(2)/deepslide-master/code/utils_split.py", line 95, in split
) > val_wsi_per_class + test_wsi_per_class, "Not enough slides in each class."

AssertionError: Not enough slides in each class.

[JosephDiPalma]

If you only have 3 WSIs per class, you will need to change the --val_wsi_per_class and --test_wsi_per_class flags when running the file. Since your have 3 files per class you could run the following command to split them python 1_split.py --val_wsi_per_class 1 --test_wsi_per_class 1. Then wsi_train, wsi_test, and wsi_val will each contain 1 file per class.

[Muralidare]

i run 3_train.py and then

"++++ Running 3_train.py +++++
3 classes: ['class1', 'class2', 'class3']
num train images 3626480
num val images 160
CUDA is_available: False
train_folder: train_folder
num_epochs: 20
num_layers: 18
learning_rate: 0.001
batch_size: 16
weight_decay: 0.0001
learning_rate_decay: 0.85
resume_checkpoint: False
resume_checkpoint_path (only if resume_checkpoint is true): checkpoints/xyz.pt
save_interval: 1
output in checkpoints_folder: checkpoints
pretrain: False
log_csv: logs/log_12282019_182220.csv"

but still its running, what should i do?

[Muralidare]

Finally i got this error

Traceback (most recent call last):
File "3_train.py", line 28, in
weight_decay=config.args.weight_decay)
File "/media/asus/new/deepslide-master(2)/deepslide-master/code/utils_model.py", line 498, in train_resnet
num_classes=num_classes)
File "/media/asus/new/deepslide-master(2)/deepslide-master/code/utils_model.py", line 257, in train_helper
for idx, (inputs, labels) in enumerate(dataloaders["train"]):
File "/home/asus/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 819, in next
return self._process_data(data)
File "/home/asus/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 846, in _process_data
data.reraise()
File "/home/asus/.local/lib/python3.6/site-packages/torch/_utils.py", line 385, in reraise
raise self.exc_type(msg)
OSError: Caught OSError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/asus/.local/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "/home/asus/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/asus/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/asus/.local/lib/python3.6/site-packages/torchvision/datasets/folder.py", line 138, in getitem
sample = self.loader(path)
File "/home/asus/.local/lib/python3.6/site-packages/torchvision/datasets/folder.py", line 174, in default_loader
return pil_loader(path)
File "/home/asus/.local/lib/python3.6/site-packages/torchvision/datasets/folder.py", line 156, in pil_loader
img = Image.open(f)
File "/home/asus/.local/lib/python3.6/site-packages/PIL/Image.py", line 2775, in open
prefix = fp.read(16)
OSError: [Errno 5] Input/output error

[JosephDiPalma]

i run 3_train.py and then

"++++ Running 3_train.py +++++
3 classes: ['class1', 'class2', 'class3']
num train images 3626480
num val images 160
CUDA is_available: False
train_folder: train_folder
num_epochs: 20
num_layers: 18
learning_rate: 0.001
batch_size: 16
weight_decay: 0.0001
learning_rate_decay: 0.85
resume_checkpoint: False
resume_checkpoint_path (only if resume_checkpoint is true): checkpoints/xyz.pt
save_interval: 1
output in checkpoints_folder: checkpoints
pretrain: False
log_csv: logs/log_12282019_182220.csv"

but still its running, what should i do?

The code is probably running but since you are running it on a CPU it is very slow. I would add in either some print statements or use a package like tqdm to display a progress bar.

Finally i got this error

Traceback (most recent call last):
File "3_train.py", line 28, in
weight_decay=config.args.weight_decay)
File "/media/asus/new/deepslide-master(2)/deepslide-master/code/utils_model.py", line 498, in train_resnet
num_classes=num_classes)
File "/media/asus/new/deepslide-master(2)/deepslide-master/code/utils_model.py", line 257, in train_helper
for idx, (inputs, labels) in enumerate(dataloaders["train"]):
File "/home/asus/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 819, in next
return self._process_data(data)
File "/home/asus/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 846, in _process_data
data.reraise()
File "/home/asus/.local/lib/python3.6/site-packages/torch/_utils.py", line 385, in reraise
raise self.exc_type(msg)
OSError: Caught OSError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/asus/.local/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "/home/asus/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/asus/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/asus/.local/lib/python3.6/site-packages/torchvision/datasets/folder.py", line 138, in getitem
sample = self.loader(path)
File "/home/asus/.local/lib/python3.6/site-packages/torchvision/datasets/folder.py", line 174, in default_loader
return pil_loader(path)
File "/home/asus/.local/lib/python3.6/site-packages/torchvision/datasets/folder.py", line 156, in pil_loader
img = Image.open(f)
File "/home/asus/.local/lib/python3.6/site-packages/PIL/Image.py", line 2775, in open
prefix = fp.read(16)
OSError: [Errno 5] Input/output error

I haven't seen this error before. I would try to change the --num_workers flag when running 3_train.py. If that doesn't work, I would try to read in an image using PIL to make sure it is installed correctly.

[Muralidare]

+++++ Running 3_train.py +++++
3 classes: ['X', 'Y', 'Z']
num train images 113968
num val images 2176
CUDA is_available: True
train_folder: train_folder
num_epochs: 20
num_layers: 18
learning_rate: 0.001
batch_size: 16
weight_decay: 0.0001
learning_rate_decay: 0.85
resume_checkpoint: False
resume_checkpoint_path (only if resume_checkpoint is true): checkpoints/xyz.pt
save_interval: 1
output in checkpoints_folder: checkpoints
pretrain: False
log_csv: logs/log_12302019_14333.csv

Traceback (most recent call last):
File "3_train.py", line 28, in
weight_decay=config.args.weight_decay)
File "/media/jetson/4952FC384E420093/deepslide-master (2) (1)/deepslide-master/code/utils_model.py", line 498, in train_resnet
num_classes=num_classes)
File "/media/jetson/4952FC384E420093/deepslide-master (2) (1)/deepslide-master/code/utils_model.py", line 257, in train_helper
for idx, (inputs, labels) in enumerate(dataloaders["train"]):
File "/home/jetson/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 819, in next
return self._process_data(data)
File "/home/jetson/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 846, in _process_data
data.reraise()
File "/home/jetson/.local/lib/python3.6/site-packages/torch/_utils.py", line 385, in reraise
raise self.exc_type(msg)
OSError: Caught OSError in DataLoader worker process 6.
Original Traceback (most recent call last):
File "/home/jetson/.local/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "/home/jetson/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/jetson/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/usr/local/lib/python3.6/dist-packages/torchvision-0.3.0-py3.6-linux-aarch64.egg/torchvision/datasets/folder.py", line 138, in getitem
sample = self.loader(path)
File "/usr/local/lib/python3.6/dist-packages/torchvision-0.3.0-py3.6-linux-aarch64.egg/torchvision/datasets/folder.py", line 174, in default_loader
return pil_loader(path)
File "/usr/local/lib/python3.6/dist-packages/torchvision-0.3.0-py3.6-linux-aarch64.egg/torchvision/datasets/folder.py", line 156, in pil_loader
img = Image.open(f)
File "/usr/local/lib/python3.6/dist-packages/PIL/Image.py", line 2818, in open
raise IOError("cannot identify image file %r" % (filename if filename else fp))
OSError: cannot identify image file <_io.BufferedReader name='train_folder/train/X/10_98_188.jpg'>

[Muralidare]

i am using Jetson TX2 for this process. i gave 15 to 20 WSI as input in X,Y,Z folders in all_wsi. but it creates many wsi in train_folder/train. is there any limitation to adjust input wsi?

[JosephDiPalma]

i am using Jetson TX2 for this process. i gave 15 to 20 WSI as input in X,Y,Z folders in all_wsi. but it creates many wsi in train_folder/train. is there any limitation to adjust input wsi?

By this comment, do you mean how to generate fewer training patches? If you want to generate fewer training patches, you should modify the --overlap_factor flag according to the guidelines listed in config.py.

[Muralidare]

finally i got this error.

+++++ Running 7_final_test.py +++++

----- Finding best thresholds -----
WARNING: One of the image directories is empty. Skipping this directory.
WARNING: One of the image directories is empty. Skipping this directory.
WARNING: One of the image directories is empty. Skipping this directory.
WARNING: One of the image directories is empty. Skipping this directory.
WARNING: One of the image directories is empty. Skipping this directory.
WARNING: One of the image directories is empty. Skipping this directory.
Traceback (most recent call last):
File "7_final_test.py", line 11, in
classes=config.classes)
File "/media/jetson/4952FC384E420093/latest/deepslide-master (2)/deepslide-master/code/utils_evaluation.py", line 226, in find_best_acc_and_thresh
classes=classes)
File "/media/jetson/4952FC384E420093/latest/deepslide-master (2)/deepslide-master/code/utils_evaluation.py", line 175, in get_scores
for _class in class_to_gt_count
File "/media/jetson/4952FC384E420093/latest/deepslide-master (2)/deepslide-master/code/utils_evaluation.py", line 175, in
for _class in class_to_gt_count
ZeroDivisionError: float division by zero

[Muralidare]

" WARNING: One of the image directories is empty. Skipping this directory."

Which directory?

[JosephDiPalma]

" WARNING: One of the image directories is empty. Skipping this directory."

Which directory?

You can figure out which directory is empty by altering the file utils_evaluation.py. From lines 154-170, there is a try and except block which produces this warning. In the except KeyError section, you can add another print() statement to print the file variable from the loop in line 152.

[JosephDiPalma]

@Muralidare Is this issue still open?

[c25kenneth]

@jasonwei20 I'm still facing the " WARNING: One of the image directories is empty. Skipping this directory." error. I've printed out the files as suggested and it returns all of the images that are a part of the validation set. For example, WSI_IMG_0001.png. How do we fix this error? I know that my CSV files and folders aren't empty, so I'm not sure how to fix this error. Thanks!