Backends dataset

[jonasteuwen]

Implement multiple backends in the dataset classes and fix bugs in the backends

[Tkootstra]

I have found at least 3 slides which will not load with the new experimental backends.
To reproduce: select one of these slides from TCGA-BRCA-DX:

'1c55f0d8-b0af-4a1e-84ef-f30aab5d749b/TCGA-OL-A5RY-01Z-00-DX1.AE4E9D74-FC1C-4C1E-AE6D-5DF38899BBA6.svs'

'69254fba-c85c-4bbb-8a4c-b2e14c8e5a2a/TCGA-OL-A5RZ-01Z-00-DX1.6394C05E-1C34-4F4B-8859-F5E961E7EFF9.svs'

'6eca5a11-0405-4619-88dd-04a12780e2bb/TCGA-OL-A5RX-01Z-00-DX1.15A0D4F4-2744-4D44-8883-27FF83D9C824.svs'

Full stacktrace:

Traceback (most recent call last):
File "/home/tkootstra/miniconda3/envs/dlup_test2/lib/python3.9/site-packages/dlup/experimental_backends/init.py", line 97, in _try_pyvips
slide = PyVipsSlide(filename)
File "/home/tkootstra/miniconda3/envs/dlup_test2/lib/python3.9/site-packages/dlup/experimental_backends/_pyvips.py", line 51, in init
self._read_as_openslide(filename)
File "/home/tkootstra/miniconda3/envs/dlup_test2/lib/python3.9/site-packages/dlup/experimental_backends/_pyvips.py", line 111, in _read_as_openslide
mpp_x = float(self._images[0].get("openslide.mpp-x"))
File "/home/tkootstra/miniconda3/envs/dlup_test2/lib/python3.9/site-packages/pyvips/vimage.py", line 1062, in get
raise Error('unable to get {0}'.format(name))
pyvips.error.Error: unable to get openslide.mpp-x
vips_image_get: field "openslide.mpp-x" not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/tkootstra/miniconda3/envs/dlup_test2/lib/python3.9/site-packages/dlup/experimental_backends/init.py", line 58, in autodetect_backend
return _try_pyvips(filename)
File "/home/tkootstra/miniconda3/envs/dlup_test2/lib/python3.9/site-packages/dlup/experimental_backends/init.py", line 102, in _try_pyvips
raise UnsupportedSlideError(f"Cannot read {filename} with pyvips.")
dlup._exceptions.UnsupportedSlideError: (UnsupportedSlideError(...), 'Cannot read /data/datasets/TCGA/data/slides/6eca5a11-0405-4619-88dd-04a12780e2bb/TCGA-OL-A5RX-01Z-00-DX1.15A0D4F4-2744-4D44-8883-27FF83D9C824.svs with pyvips.')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/tkootstra/miniconda3/envs/dlup_test2/lib/python3.9/site-packages/dlup/_image.py", line 119, in from_file_path
wsi = backend(filename=wsi_file_path)
File "/home/tkootstra/miniconda3/envs/dlup_test2/lib/python3.9/site-packages/dlup/experimental_backends/init.py", line 60, in autodetect_backend
raise UnsupportedSlideError(f"Cannot read {filename} with pyvips or openslide.")
dlup._exceptions.UnsupportedSlideError: (UnsupportedSlideError(...), 'Cannot read /data/datasets/TCGA/data/slides/6eca5a11-0405-4619-88dd-04a12780e2bb/TCGA-OL-A5RX-01Z-00-DX1.15A0D4F4-2744-4D44-8883-27FF83D9C824.svs with pyvips or openslide.')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/tmp/pycharm_project_960/debug.py", line 26, in
slide = SlideImage.from_file_path(files[2], backend=ImageBackends.AUTODETECT)
File "/home/tkootstra/miniconda3/envs/dlup_test2/lib/python3.9/site-packages/dlup/_image.py", line 121, in from_file_path
raise UnsupportedSlideError(f"Unsupported file: {wsi_file_path}")
dlup._exceptions.UnsupportedSlideError: (UnsupportedSlideError(...), 'Unsupported file: /data/datasets/TCGA/data/slides/6eca5a11-0405-4619-88dd-04a12780e2bb/TCGA-OL-A5RX-01Z-00-DX1.15A0D4F4-2744-4D44-8883-27FF83D9C824.svs')

Process finished with exit code 1

Env is clean python 3.9, with dlup install via git with commit hash for the PR, as such: pip install git+https://github.com/NKI-AI/dlup.git@ff879a58ce8745ef6e9aa28d52f379c550dfdff4

Environment:

Name Version Build Channel

_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
ca-certificates 2022.4.26 h06a4308_0
certifi 2022.6.15 py39h06a4308_0
cffi 1.15.1 pypi_0 pypi
charset-normalizer 2.1.0 pypi_0 pypi
cycler 0.11.0 pypi_0 pypi
dlup 0.3.0.dev0 pypi_0 pypi
fonttools 4.34.2 pypi_0 pypi
idna 3.3 pypi_0 pypi
imageio 2.19.3 pypi_0 pypi
kiwisolver 1.4.3 pypi_0 pypi
ld_impl_linux-64 2.38 h1181459_1
libffi 3.3 he6710b0_2
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
matplotlib 3.5.2 pypi_0 pypi
ncurses 6.3 h5eee18b_3
networkx 2.8.4 pypi_0 pypi
numpy 1.23.0 pypi_0 pypi
openslide-python 1.2.0 pypi_0 pypi
openssl 1.1.1p h5eee18b_0
packaging 21.3 pypi_0 pypi
pillow 9.2.0 pypi_0 pypi
pip 21.2.4 py39h06a4308_0
pkgconfig 1.5.5 pypi_0 pypi
pycparser 2.21 pypi_0 pypi
pyparsing 3.0.9 pypi_0 pypi
python 3.9.12 h12debd9_1
python-dateutil 2.8.2 pypi_0 pypi
pyvips 2.2.1 pypi_0 pypi
pywavelets 1.3.0 pypi_0 pypi
readline 8.1.2 h7f8727e_1
requests 2.28.1 pypi_0 pypi
scikit-image 0.19.3 pypi_0 pypi
scipy 1.8.1 pypi_0 pypi
setuptools 61.2.0 py39h06a4308_0
shapely 2.0.0.dev0+66.ga277741 pypi_0 pypi
six 1.16.0 pypi_0 pypi
sqlite 3.38.5 hc218d9a_0
tifffile 2022.5.4 pypi_0 pypi
tifftools 1.3.4 pypi_0 pypi
tk 8.6.12 h1ccaba5_0
tqdm 4.64.0 pypi_0 pypi
tzdata 2022a hda174b7_0
urllib3 1.26.9 pypi_0 pypi
wheel 0.37.1 pyhd3eb1b0_0
xz 5.2.5 h7f8727e_1
zlib 1.2.12 h7f8727e_2

[jonasteuwen]

In my modified version of openslide I can read those slides. They don't adhere properly to the assumptions. I will think about ways to circumvent that but can you review this ignoring that problem for now? It would be good to add as an issue though.

[AjeyPaiK]

The new modifications work as expected. We can merge these changes into the main branch.