Data Preprocessing error

[MohamedOmar2020]

Hi Guys, thank you for making this wonderful resource available.
I organized my slides into wsi_train, wsi_val, wsi_test using 1_split.py which ran fine. However I keep getting this error when I run code/2_process_patches.py:

wsi_train/neg: 35359.581701MB, 201 images, overlap_factor=1.00
wsi_train/pos: 92751.49675MB, 451 images, overlap_factor=1.00

getting small crops from 201 images in wsi_train/neg with inverse overlap factor 1.00 outputting in train_folder/train/neg
Traceback (most recent call last):
  File "/home/mao4005/.conda/envs/deepslide/lib/python3.6/site-packages/PIL/ImageFile.py", line 101, in __init__
    self._open()
  File "/home/mao4005/.conda/envs/deepslide/lib/python3.6/site-packages/PIL/TiffImagePlugin.py", line 979, in _open
    self._seek(0)
  File "/home/mao4005/.conda/envs/deepslide/lib/python3.6/site-packages/PIL/TiffImagePlugin.py", line 1046, in _seek
    self._setup()
  File "/home/mao4005/.conda/envs/deepslide/lib/python3.6/site-packages/PIL/TiffImagePlugin.py", line 1170, in _setup
    self._compression = COMPRESSION_INFO[self.tag_v2.get(COMPRESSION, 1)]
KeyError: 33003

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "deepslide/2_process_patches.py", line 20, in <module>
    type_histopath=config.args.type_histopath)
  File "/athena/marchionnilab/scratch/lab_data/Mohamed/pca_outcome/deepslide/utils_processing.py", line 155, in gen_train_patches
    type_histopath=type_histopath)
  File "/athena/marchionnilab/scratch/lab_data/Mohamed/pca_outcome/deepslide/utils_processing.py", line 364, in produce_patches
    uri=image_loc if by_folder else input_folder.joinpath(image_loc))
  File "/home/mao4005/.conda/envs/deepslide/lib/python3.6/site-packages/imageio/core/functions.py", line 265, in imread
    reader = read(uri, format, "i", **kwargs)
  File "/home/mao4005/.conda/envs/deepslide/lib/python3.6/site-packages/imageio/core/functions.py", line 186, in get_reader
    return format.get_reader(request)
  File "/home/mao4005/.conda/envs/deepslide/lib/python3.6/site-packages/imageio/core/format.py", line 170, in get_reader
    return self.Reader(self, request)
  File "/home/mao4005/.conda/envs/deepslide/lib/python3.6/site-packages/imageio/core/format.py", line 221, in __init__
    self._open(**self.request.kwargs.copy())
  File "/home/mao4005/.conda/envs/deepslide/lib/python3.6/site-packages/imageio/plugins/pillow.py", line 125, in _open
    self._im = factory(self._fp, "")
  File "/home/mao4005/.conda/envs/deepslide/lib/python3.6/site-packages/PIL/ImageFile.py", line 110, in __init__
    raise SyntaxError(v)
SyntaxError: 33003

These are the packages in my conda env:
_libgcc_mutex 0.1 main conda-forge
_tflow_select 2.3.0 mkl
attrs 19.3.0 py_0 conda-forge
blas 1.0 mkl conda-forge
bzip2 1.0.8 h7b6447c_0
c-ares 1.15.0 h7b6447c_1001
ca-certificates 2021.9.30 h06a4308_1
cairo 1.16.0 h18b612c_1001 conda-forge
certifi 2021.5.30 py36h06a4308_0
cloudpickle 2.0.0 pyhd3eb1b0_0
cpuonly 2.0 0 pytorch
cudatoolkit 10.1.243 h6bb024c_0
cycler 0.10.0 pypi_0 pypi
cytoolz 0.11.0 py36h7b6447c_0
dask-core 2021.3.0 pyhd3eb1b0_0
dataclasses 0.8 pyh4f3eec9_6
dbus 1.13.12 h746ee38_0
decorator 4.4.2 py_0 conda-forge
defusedxml 0.6.0 py_0 conda-forge
entrypoints 0.3 py36_0
et_xmlfile 1.1.0 py36h06a4308_0
expat 2.2.6 he6710b0_0
ffmpeg 4.2.2 h20bf706_0
fontconfig 2.13.1 he4413a7_1000 conda-forge
freetype 2.9.1 h8a8886c_1
fribidi 1.0.9 h516909a_0 conda-forge
gast 0.3.3 py_0 conda-forge
gdk-pixbuf 2.38.2 h3f25603_4 conda-forge
glib 2.63.1 h5a9c865_0
gmp 6.1.2 h6c8ec71_1
gnutls 3.6.15 he1e5248_0
gobject-introspection 1.56.1 py36hbc4ca2d_2
google-pasta 0.2.0 py_0
graphite2 1.3.13 h23475e2_0
gst-plugins-base 1.14.0 hbbd80ab_1
gstreamer 1.14.0 hb453b48_1
h5py 2.10.0 py36h7918eee_0
harfbuzz 2.4.0 h37c48d4_1 conda-forge
hdf5 1.10.4 hb1b8bf9_0
icu 58.2 h9c2bf20_1
imageio 2.9.0 pyhd3eb1b0_0
importlib-metadata 4.8.1 py37h89c1867_0 conda-forge
intel-openmp 2019.4 243
ipython_genutils 0.2.0 pyhd3eb1b0_1
jinja2 2.11.1 py_0 conda-forge
jpeg 9d h36c2ea0_0 conda-forge
jupyter_client 6.1.0 py_0 conda-forge
jupyter_core 4.8.1 py36h06a4308_0
keras-applications 1.0.8 py_0
keras-preprocessing 1.1.0 py_1
kiwisolver 1.3.1 py36h2531618_0
lame 3.100 h7b6447c_0
ld_impl_linux-64 2.33.1 h53a641e_7 conda-forge
libblas 3.8.0 14_mkl conda-forge
libcroco 0.6.13 h8d621e5_0 conda-forge
libedit 3.1.20181209 hc058e9b_0
libffi 3.2.1 hd88cf55_4
libgcc 7.2.0 h69d50b8_2 conda-forge
libgcc-ng 9.1.0 hdf63c60_0
libgfortran-ng 7.3.0 hdf63c60_0
libiconv 1.15 h516909a_1006 conda-forge
libidn2 2.3.2 h7f8727e_0
liblapack 3.8.0 14_mkl conda-forge
libopus 1.3.1 h7b6447c_0
libpng 1.6.37 hbc83047_0
libprotobuf 3.11.4 hd408876_0
librsvg 2.46.2 h33a7fed_1 conda-forge
libsodium 1.0.16 h1bed415_0
libstdcxx-ng 9.1.0 hdf63c60_0
libtasn1 4.16.0 h27cfd23_0
libtiff 4.1.0 h2733197_0
libunistring 0.9.10 h27cfd23_0
libuuid 2.32.1 h14c3975_1000 conda-forge
libuv 1.40.0 h7b6447c_0
libvpx 1.7.0 h439df22_0
libxcb 1.13 h1bed415_1
libxml2 2.9.9 hea5a465_1
markupsafe 2.0.1 py36h27cfd23_0
matplotlib 3.3.4 pypi_0 pypi
matplotlib-base 3.2.1 py36hef1b27d_0
mkl 2019.4 243
mkl-service 2.3.0 py36he8ac12f_0
mkl_fft 1.3.0 py36h54f3939_0
mkl_random 1.1.0 py36hd6b4f25_0
nbconvert 5.6.1 py37_0 conda-forge
nbformat 5.0.4 py_0 conda-forge
ncurses 6.2 he6710b0_0
nettle 3.7.3 hbbd107a_1
networkx 2.5.1 pyhd3eb1b0_0
notebook 6.0.1 py37_0 conda-forge
numpy 1.19.5 pypi_0 pypi
numpy-base 1.19.2 py36hfa32c7d_0
olefile 0.46 pyhd3eb1b0_0
openh264 2.1.0 hd408876_0
openjpeg 2.3.1 h981e76c_3 conda-forge
openpyxl 3.0.9 pyhd3eb1b0_0
openslide 3.4.1 h8137273_0 conda-forge
openssl 1.1.1l h7f8727e_0
packaging 21.0 pyhd8ed1ab_0 conda-forge
pandoc 2.2.3.2 0
pango 1.42.4 h7062337_3 conda-forge
parso 0.6.2 py_0 conda-forge
pcre 8.43 he6710b0_0
pillow 5.3.0 py36h34e0f95_0
pip 21.2.4 pyhd8ed1ab_0 conda-forge
pixman 0.38.0 h7b6447c_0
prometheus_client 0.7.1 py_0 conda-forge
prompt-toolkit 3.0.4 py_0 conda-forge
prompt_toolkit 3.0.4 0 conda-forge
protobuf 3.11.4 py36he6710b0_0
pycparser 2.20 py_0 conda-forge
pygments 2.6.1 py_0 conda-forge
pyparsing 2.4.6 py_0 conda-forge
python 3.6.10 hcf32534_1
python-dateutil 2.8.1 py_0 conda-forge
python_abi 3.6 2_cp36m conda-forge
pytorch 1.10.0 py3.6_cpu_0 pytorch
pytorch-mutex 1.0 cpu pytorch
pytz 2019.3 py_0 conda-forge
pywavelets 1.1.1 py36h7b6447c_2
pyyaml 5.4.1 py36h27cfd23_1
pyzmq 18.1.1 py36he6710b0_0
qt 5.9.7 h5867ecd_1
readline 8.0 h7b6447c_0
scikit-image 0.17.2 pypi_0 pypi
scipy 1.5.2 py36h0b6359f_0
setuptools 58.0.4 py36h06a4308_0
six 1.16.0 pyhd3eb1b0_0
sqlite 3.31.1 h7b6447c_0
testpath 0.4.4 py_0 conda-forge
tifffile 2020.9.3 pypi_0 pypi
tk 8.6.8 hbc83047_0
toolz 0.11.1 pyhd3eb1b0_0
torchaudio 0.10.0 py36_cpu [cpuonly] pytorch
torchvision 0.11.1 py36_cpu [cpuonly] pytorch
tornado 6.1 py36h27cfd23_0
traitlets 4.3.3 py36h06a4308_0
typing_extensions 3.10.0.2 pyh06a4308_0
wcwidth 0.1.8 py_0 conda-forge
webencodings 0.5.1 py_1 conda-forge
werkzeug 1.0.0 py_0 conda-forge
wheel 0.37.0 pyhd8ed1ab_1 conda-forge
x264 1!157.20191217 h7b6447c_0
xlrd 2.0.1 pyhd3eb1b0_0
xorg-kbproto 1.0.7 h14c3975_1002 conda-forge
xorg-libice 1.0.10 h516909a_0 conda-forge
xorg-libsm 1.2.3 h84519dc_1000 conda-forge
xorg-libx11 1.6.9 h516909a_0 conda-forge
xorg-libxext 1.3.4 h516909a_0 conda-forge
xorg-libxrender 0.9.10 h516909a_1002 conda-forge
xorg-renderproto 0.11.1 h14c3975_1002 conda-forge
xorg-xextproto 7.3.0 h14c3975_1002 conda-forge
xorg-xproto 7.0.31 h14c3975_1007 conda-forge
xz 5.2.4 h14c3975_4
yaml 0.2.5 h7b6447c_0
zeromq 4.3.1 he6710b0_3
zipp 2.2.0 py_0 conda-forge
zlib 1.2.11 h7b6447c_3
zstd 1.3.7 h0b5b093_0

[JosephDiPalma]

@MohamedOmar2020 I think your problem is that one or more of the images is corrupted or unreadable. Try printing the image name that causes the crash and then reading it in a separate Python interpreter session. If the same crash occurs, then the problem is caused by that image.

[Tejussurendran]

@MohamedOmar2020 I am getting the same error that you had, by any chance did you find a resolution to the problem? Also are you trying to use .svs files?

[JosephDiPalma]

@Tejussurendran @MohamedOmar2020 Would either of you be able to provide an image that allows me to reproduce this error?

[Tejussurendran]

@JosephDiPalma I am trying to use a .svs file, and it appears that svs files are not supported by imread(), however I am not sure if you have any workarounds for this.
Also I tried to provide an image however, github said it was a file type they didnt support.

[JosephDiPalma]

@Tejussurendran You can fix this issue in 2 ways:

1. Convert all the svs files to png, jpg, or another supported format.
2. Replace lines 363-364 in code/utils_processing.py as follows:

slide = OpenSlide(filename=str(image_loc if by_folder else input_folder.joinpath(image_loc)))
image = np.array(slide.read_region(location=(0, 0), level=0, size=slide.dimensions).convert("RGB"))

Also, you will need to have the OpenSlide Python package installed and imported at the top of the file code/utils_processing.py with from openslide import OpenSlide.

[Tejussurendran]

@JosephDiPalma Thank you very much for your help!

It is currently running without an issue(hopefully!), and I will let you know the outcome of my attempt.

Thank you once again for making this platform available to use!

[JosephDiPalma]

@Tejussurendran Can you upload the problematic svs file to this dropbox link (https://www.dropbox.com/request/W5tcTkohyOaRocaO8AAr)?

Make sure not to upload any PHI or otherwise unauthorized data.

[Tejussurendran]

@JosephDiPalma it appears that at the process step, the process keeps being killed after a specific svs. There is no error being thrown however. Any ideas as to why?

[Tejussurendran]

@JosephDiPalma I tried doing the 2_processing step over the weekend, and it appeared to keep getting killed, regardless of it was a validation training or testing.

[JosephDiPalma]

@Tejussurendran The process is likely being killed due to insufficient RAM.

Try changing the num_workers parameter in config.py to something smaller and see if that helps.

[Tejussurendran]

@JosephDiPalma I think I have solved that issue, thank you!

However it appears that when trying to generate the validation evaluation patches it is throwing an error.

I have attached a copy of it below

.

[JosephDiPalma]

@Tejussurendran Does it work for the other sets?
If it does, then one of the images in your validation set is probably corrupted.

[Tejussurendran]

@JosephDiPalma It works on the validation set, and I am trying it out on the training set currently. On the testing set it had the same issues as the validation evaluation set.

[Tejussurendran]

@JosephDiPalma i tried setting the num_workers variable to 1 and it the program is still being killed. Is there a bigger underlying issue that maybe causing this to happen? This is occurring with all 3 sets being validation testing and training.

[JosephDiPalma]

@Tejussurendran I believe the underlying issue is that you don't have enough system memory.
How much memory do you have, and what is the size of your images?

[Tejussurendran]

@JosephDiPalma I am sshing on to a workstation with 256 gb of ram. I have also uploaded one of the sample images to your dropbox link from earlier. They are only a couple hundred kilobytes

[JosephDiPalma]

@Tejussurendran That should be more than enough memory.
I'm not sure what the issue is now, so give me some time to test the code using the sample image.

Can you also provide the Python package names, including versions, to us for debugging?

[Tejussurendran]

@JosephDiPalma Thank you very much for your help! With regards to the issue, it is not consistent on where it is killed, sometimes it is 5 images, sometimes 1 etc..

Also for the python package names, I am not sure how to find the version number, however I have attached the packages being used in utils_processing.py

From what I understand it should be the same packages as those provided.

import functools
import itertools
import math
import time
from multiprocessing import (Process, Queue, RawArray)
from pathlib import Path
from shutil import copyfile
from typing import (Callable, Dict, List, Tuple)
from openslide import OpenSlide
import numpy as np
from PIL import Image
from imageio import (imsave, imread)
from skimage.measure import block_reduce

[Tejussurendran]

@JosephDiPalma Also, I am not sure if this will help with solving the issue, but I am also trying to save the images as png.

[ntomita]

@Tejussurendran Could you give us the output of pip list and conda list?

I checked your image in our dropbox. It is an image patch. Is this your input? This library assumes you feed a large slide image file as input, usually a few gigabytes file in the preprocess stage.

[Tejussurendran]

pip list:

cycler 0.11.0
imageio 2.10.1
kiwisolver 1.3.2
matplotlib 3.4.3
networkx 2.6.3
numpy 1.21.3
opencv-python 4.5.4.58
openslide-python 1.1.2
pandas 1.3.4
Pillow 8.4.0
pip 21.3.1
pydicom 2.2.2
pyparsing 3.0.4
python-dateutil 2.8.2
pytz 2021.3
PyWavelets 1.1.1
scikit-image 0.18.3
scipy 1.7.1
setuptools 41.2.0
six 1.16.0
tifffile 2021.10.12
torch 1.10.0
torchvision 0.11.1
typing-extensions 3.10.0.2

conda list:

packages in environment at /opt/anaconda3:

Name Version Build Channel

_ipyw_jlab_nb_ext_conf 0.1.0 py38_0
_libgcc_mutex 0.1 main
alabaster 0.7.12 py_0
anaconda 2020.07 py38_0
anaconda-client 1.7.2 py38_0
anaconda-navigator 1.9.12 py38_0
anaconda-project 0.8.4 py_0
argh 0.26.2 py38_0
asn1crypto 1.3.0 py38_0
astroid 2.4.2 py38_0
astropy 4.0.1.post1 py38h7b6447c_1
atomicwrites 1.4.0 py_0
attrs 19.3.0 py_0
autopep8 1.5.3 py_0
babel 2.8.0 py_0
backcall 0.2.0 py_0
backports 1.0 py_2
backports.functools_lru_cache 1.6.1 py_0
backports.shutil_get_terminal_size 1.0.0 py38_2
backports.tempfile 1.0 py_1
backports.weakref 1.0.post1 py_1
beautifulsoup4 4.9.1 py38_0
bitarray 1.4.0 py38h7b6447c_0
bkcharts 0.2 py38_0
blas 1.0 mkl
bleach 3.1.5 py_0
blosc 1.19.0 hd408876_0
bokeh 2.1.1 py38_0
boto 2.49.0 py38_0
bottleneck 1.3.2 py38heb32a55_1
brotlipy 0.7.0 py38h7b6447c_1000
bzip2 1.0.8 h7b6447c_0
ca-certificates 2020.6.24 0
cairo 1.14.12 h8948797_3
certifi 2020.6.20 py38_0
cffi 1.14.0 py38he30daa8_1
chardet 3.0.4 py38_1003
click 7.1.2 py_0
cloudpickle 1.5.0 py_0
clyent 1.2.2 py38_1
colorama 0.4.3 py_0
conda 4.8.3 py38_0
conda-build 3.18.11 py38_0
conda-env 2.6.0 1
conda-package-handling 1.6.1 py38h7b6447c_0
conda-verify 3.4.2 py_1
contextlib2 0.6.0.post1 py_0
cryptography 2.9.2 py38h1ba5d50_0
curl 7.71.1 hbc83047_1
cycler 0.10.0 py38_0
cython 0.29.21 py38he6710b0_0
cytoolz 0.10.1 py38h7b6447c_0
dask 2.20.0 py_0
dask-core 2.20.0 py_0
dbus 1.13.16 hb2f20db_0
decorator 4.4.2 py_0
defusedxml 0.6.0 py_0
diff-match-patch 20200713 py_0
distributed 2.20.0 py38_0
docutils 0.16 py38_1
entrypoints 0.3 py38_0
et_xmlfile 1.0.1 py_1001
expat 2.2.9 he6710b0_2
fastcache 1.1.0 py38h7b6447c_0
filelock 3.0.12 py_0
flake8 3.8.3 py_0
flask 1.1.2 py_0
fontconfig 2.13.0 h9420a91_0
freetype 2.10.2 h5ab3b9f_0
fribidi 1.0.9 h7b6447c_0
fsspec 0.7.4 py_0
future 0.18.2 py38_1
get_terminal_size 1.0.0 haa9412d_0
gevent 20.6.2 py38h7b6447c_0
glib 2.65.0 h3eb4bd4_0
glob2 0.7 py_0
gmp 6.1.2 h6c8ec71_1
gmpy2 2.0.8 py38hd5f6e3b_3
graphite2 1.3.14 h23475e2_0
greenlet 0.4.16 py38h7b6447c_0
gst-plugins-base 1.14.0 hbbd80ab_1
gstreamer 1.14.0 hb31296c_0
h5py 2.10.0 py38h7918eee_0
harfbuzz 2.4.0 hca77d97_1
hdf5 1.10.4 hb1b8bf9_0
heapdict 1.0.1 py_0
html5lib 1.1 py_0
icu 58.2 he6710b0_3
idna 2.10 py_0
imageio 2.9.0 py_0
imagesize 1.2.0 py_0
importlib-metadata 1.7.0 py38_0
importlib_metadata 1.7.0 0
intel-openmp 2020.1 217
intervaltree 3.0.2 py_1
ipykernel 5.3.2 py38h5ca1d4c_0
ipython 7.16.1 py38h5ca1d4c_0
ipython_genutils 0.2.0 py38_0
ipywidgets 7.5.1 py_0
isort 4.3.21 py38_0
itsdangerous 1.1.0 py_0
jbig 2.1 hdba287a_0
jdcal 1.4.1 py_0
jedi 0.17.1 py38_0
jeepney 0.4.3 py_0
jinja2 2.11.2 py_0
joblib 0.16.0 py_0
jpeg 9b h024ee3a_2
json5 0.9.5 py_0
jsonschema 3.2.0 py38_0
jupyter 1.0.0 py38_7
jupyter_client 6.1.6 py_0
jupyter_console 6.1.0 py_0
jupyter_core 4.6.3 py38_0
jupyterlab 2.1.5 py_0
jupyterlab_server 1.2.0 py_0
keyring 21.2.1 py38_0
kiwisolver 1.2.0 py38hfd86e86_0
krb5 1.18.2 h173b8e3_0
lazy-object-proxy 1.4.3 py38h7b6447c_0
lcms2 2.11 h396b838_0
ld_impl_linux-64 2.33.1 h53a641e_7
libarchive 3.4.2 h62408e4_0
libcurl 7.71.1 h20c2e04_1
libedit 3.1.20191231 h14c3975_1
libffi 3.3 he6710b0_2
libgcc-ng 9.1.0 hdf63c60_0
libgfortran-ng 7.3.0 hdf63c60_0
liblief 0.10.1 he6710b0_0
libllvm9 9.0.1 h4a3c616_1
libpng 1.6.37 hbc83047_0
libsodium 1.0.18 h7b6447c_0
libspatialindex 1.9.3 he6710b0_0
libssh2 1.9.0 h1ba5d50_1
libstdcxx-ng 9.1.0 hdf63c60_0
libtiff 4.1.0 h2733197_1
libtool 2.4.6 h7b6447c_5
libuuid 1.0.3 h1bed415_2
libxcb 1.14 h7b6447c_0
libxml2 2.9.10 he19cac6_1
libxslt 1.1.34 hc22bd24_0
llvmlite 0.33.0 py38hc6ec683_1
locket 0.2.0 py38_1
lxml 4.5.2 py38hefd8a0e_0
lz4-c 1.9.2 he6710b0_0
lzo 2.10 h7b6447c_2
markupsafe 1.1.1 py38h7b6447c_0
matplotlib 3.2.2 0
matplotlib-base 3.2.2 py38hef1b27d_0
mccabe 0.6.1 py38_1
mistune 0.8.4 py38h7b6447c_1000
mkl 2020.1 217
mkl-service 2.3.0 py38he904b0f_0
mkl_fft 1.1.0 py38h23d657b_0
mkl_random 1.1.1 py38h0573a6f_0
mock 4.0.2 py_0
more-itertools 8.4.0 py_0
mpc 1.1.0 h10f8cd9_1
mpfr 4.0.2 hb69a4c5_1
mpmath 1.1.0 py38_0
msgpack-python 1.0.0 py38hfd86e86_1
multipledispatch 0.6.0 py38_0
navigator-updater 0.2.1 py38_0
nbconvert 5.6.1 py38_0
nbformat 5.0.7 py_0
ncurses 6.2 he6710b0_1
networkx 2.4 py_1
nltk 3.5 py_0
nose 1.3.7 py38_2
notebook 6.0.3 py38_0
numba 0.50.1 py38h0573a6f_1
numexpr 2.7.1 py38h423224d_0
numpy 1.18.5 py38ha1c710e_0
numpy-base 1.18.5 py38hde5b4d6_0
numpydoc 1.1.0 py_0
olefile 0.46 py_0
openpyxl 3.0.4 py_0
openssl 1.1.1g h7b6447c_0
packaging 20.4 py_0
pandas 1.0.5 py38h0573a6f_0
pandoc 2.10 0
pandocfilters 1.4.2 py38_1
pango 1.45.3 hd140c19_0
parso 0.7.0 py_0
partd 1.1.0 py_0
patchelf 0.11 he6710b0_0
path 13.1.0 py38_0
path.py 12.4.0 0
pathlib2 2.3.5 py38_0
pathtools 0.1.2 py_1
patsy 0.5.1 py38_0
pcre 8.44 he6710b0_0
pep8 1.7.1 py38_0
pexpect 4.8.0 py38_0
pickleshare 0.7.5 py38_1000
pillow 7.2.0 py38hb39fc2d_0
pip 20.1.1 py38_1
pixman 0.40.0 h7b6447c_0
pkginfo 1.5.0.1 py38_0
pluggy 0.13.1 py38_0
ply 3.11 py38_0
prometheus_client 0.8.0 py_0
prompt-toolkit 3.0.5 py_0
prompt_toolkit 3.0.5 0
psutil 5.7.0 py38h7b6447c_0
ptyprocess 0.6.0 py38_0
py 1.9.0 py_0
py-lief 0.10.1 py38h403a769_0
pycodestyle 2.6.0 py_0
pycosat 0.6.3 py38h7b6447c_1
pycparser 2.20 py_2
pycurl 7.43.0.5 py38h1ba5d50_0
pydocstyle 5.0.2 py_0
pyflakes 2.2.0 py_0
pygments 2.6.1 py_0
pylint 2.5.3 py38_0
pyodbc 4.0.30 py38he6710b0_0
pyopenssl 19.1.0 py_1
pyparsing 2.4.7 py_0
pyqt 5.9.2 py38h05f1152_4
pyrsistent 0.16.0 py38h7b6447c_0
pysocks 1.7.1 py38_0
pytables 3.6.1 py38h9fd0a39_0
pytest 5.4.3 py38_0
python 3.8.3 hcff3b4d_2
python-dateutil 2.8.1 py_0
python-jsonrpc-server 0.3.4 py_1
python-language-server 0.34.1 py38_0
python-libarchive-c 2.9 py_0
pytz 2020.1 py_0
pywavelets 1.1.1 py38h7b6447c_0
pyxdg 0.26 py_0
pyyaml 5.3.1 py38h7b6447c_1
pyzmq 19.0.1 py38he6710b0_1
qdarkstyle 2.8.1 py_0
qt 5.9.7 h5867ecd_1
qtawesome 0.7.2 py_0
qtconsole 4.7.5 py_0
qtpy 1.9.0 py_0
readline 8.0 h7b6447c_0
regex 2020.6.8 py38h7b6447c_0
requests 2.24.0 py_0
ripgrep 11.0.2 he32d670_0
rope 0.17.0 py_0
rtree 0.9.4 py38_1
ruamel_yaml 0.15.87 py38h7b6447c_1
scikit-image 0.16.2 py38h0573a6f_0
scikit-learn 0.23.1 py38h423224d_0
scipy 1.5.0 py38h0b6359f_0
seaborn 0.10.1 py_0
secretstorage 3.1.2 py38_0
send2trash 1.5.0 py38_0
setuptools 49.2.0 py38_0
simplegeneric 0.8.1 py38_2
singledispatch 3.4.0.3 py38_0
sip 4.19.13 py38he6710b0_0
six 1.15.0 py_0
snappy 1.1.8 he6710b0_0
snowballstemmer 2.0.0 py_0
sortedcollections 1.2.1 py_0
sortedcontainers 2.2.2 py_0
soupsieve 2.0.1 py_0
sphinx 3.1.2 py_0
sphinxcontrib 1.0 py38_1
sphinxcontrib-applehelp 1.0.2 py_0
sphinxcontrib-devhelp 1.0.2 py_0
sphinxcontrib-htmlhelp 1.0.3 py_0
sphinxcontrib-jsmath 1.0.1 py_0
sphinxcontrib-qthelp 1.0.3 py_0
sphinxcontrib-serializinghtml 1.1.4 py_0
sphinxcontrib-websupport 1.2.3 py_0
spyder 4.1.4 py38_0
spyder-kernels 1.9.2 py38_0
sqlalchemy 1.3.18 py38h7b6447c_0
sqlite 3.32.3 h62c20be_0
statsmodels 0.11.1 py38h7b6447c_0
sympy 1.6.1 py38_0
tbb 2020.0 hfd86e86_0
tblib 1.6.0 py_0
terminado 0.8.3 py38_0
testpath 0.4.4 py_0
threadpoolctl 2.1.0 pyh5ca1d4c_0
tk 8.6.10 hbc83047_0
toml 0.10.1 py_0
toolz 0.10.0 py_0
tornado 6.0.4 py38h7b6447c_1
tqdm 4.47.0 py_0
traitlets 4.3.3 py38_0
typing_extensions 3.7.4.2 py_0
ujson 1.35 py38h7b6447c_0
unicodecsv 0.14.1 py38_0
unixodbc 2.3.7 h14c3975_0
urllib3 1.25.9 py_0
watchdog 0.10.3 py38_0
wcwidth 0.2.5 py_0
webencodings 0.5.1 py38_1
werkzeug 1.0.1 py_0
wheel 0.34.2 py38_0
widgetsnbextension 3.5.1 py38_0
wrapt 1.11.2 py38h7b6447c_0
wurlitzer 2.0.1 py38_0
xlrd 1.2.0 py_0
xlsxwriter 1.2.9 py_0
xlwt 1.3.0 py38_0
xmltodict 0.12.0 py_0
xz 5.2.5 h7b6447c_0
yaml 0.2.5 h7b6447c_0
yapf 0.30.0 py_0
zeromq 4.3.2 he6710b0_2
zict 2.0.0 py_0
zipp 3.1.0 py_0
zlib 1.2.11 h7b6447c_3
zope 1.0 py38_1
zope.event 4.4 py38_0
zope.interface 4.7.1 py38h7b6447c_0
zstd 1.4.5 h0b5b093_0

Also, Im sorry! I think I messed up the file I sent. I have attached the correct file in the dropbox now.

Thank you so much for your help!

[JosephDiPalma]

@Tejussurendran Using the provided svs file, the code ran successfully for us.
Could you provide some details on the directory structure of the code and your data?

[Tejussurendran]

@JosephDiPalma I have a folder one large folder containing all my work for this project. In it I have a folder for the svs files, structures with Has_Diabetes and Not_Has_Diabetes. The patches are depending on the type, are stored in independent folders as well.

I also noticed that with num_workers = 8, the job was killed due to ram consumption. So I tried 4 and it appears that the same issue happened again. I was thinking about potentially generating the patches elsewhere and simply moving them in to the appropriate folders. Would that work?

[Tejussurendran]

@JosephDiPalma Hi, I was wondering what the directory difference between the validation set and validation evaluation set were as, I am trying to develop the patches externally and use them in deepslide. What is the difference between the 2 folders?

[Tejussurendran]

@JosephDiPalma I was wondering if there is any way to reduce the memory consumption of the preprocessing step?

[JosephDiPalma]

@Tejussurendran You can reduce the memory consumption of the pre-processing step by reducing the num_workers setting further, or downsampling your slides prior to processing.

[github-actions]

Stale issue message