KeyError run run_wsi.sh

[caicai2526]

Dear,
when I am running run_wsi.sh,I have some problems.The fowlling is running result.

WARNING: Detect checkpoint saved in data-parallel mode. Start converting saved model to single GPU mode.
|2023-05-11|09:04:47.183| [INFO] Processing 1 ...
|2023-05-11|09:04:47.329| [WARNING] Read: Scale > 1.This means that the desired resolution is higher than the WSI baseline (maximum encoded resolution). Interpolation of read regions may occur.
|2023-05-11|09:04:50.070| [INFO] Preparing Input Output Placement: 2.886942900000001
Traceback (most recent call last):
File "D:\Project\Nuclei_Seg_Cls\cerberus\run_infer_wsi.py", line 139, in
infer.process_wsi_list(run_args)
File "D:\Project\Nuclei_Seg_Cls\cerberus\infer\wsi.py", line 971, in process_wsi_list
self.process_single_file(
File "D:\Project\Nuclei_Seg_Cls\cerberus\infer\wsi.py", line 595, in process_single_file
sel_indices = [
File "D:\Project\Nuclei_Seg_Cls\cerberus\infer\wsi.py", line 596, in
index_by_id[id(geo)] for geo in spatial_indexer.query(sel_box)
KeyError: 1582810922224

[simongraham]

Hi!

Can you provide more info pls? i.e what kind of image are you using as input? Also, please provide me with the command you are running the code with.

[caicai2526]

$ ./run_wsi.sh
Number of WSIs in list: 1
|2023-05-15|14:35:06.737| [WARNING] C:\Users\CC.conda\envs\C1\lib\site-packages\umap\distances.py:1063: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
@numba.jit()

|2023-05-15|14:35:06.739| [WARNING] C:\Users\CC.conda\envs\C1\lib\site-packages\umap\distances.py:1071: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
@numba.jit()

|2023-05-15|14:35:06.740| [WARNING] C:\Users\CC.conda\envs\C1\lib\site-packages\umap\distances.py:1086: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
@numba.jit()

|2023-05-15|14:35:10.694| [WARNING] C:\Users\CC.conda\envs\C1\lib\site-packages\umap\umap_.py:660: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
@numba.jit()

WARNING: Detect checkpoint saved in data-parallel mode. Start converting saved model to single GPU mode.
|2023-05-15|14:35:13.786| [INFO] Processing TCGA-3L-AA1B-01Z-00-DX2.17CE3683-F4B1-4978-A281-8F620C4D77B4 ...
|2023-05-15|14:35:16.910| [WARNING] Read: Scale > 1.This means that the desired resolution is higher than the WSI baseline (maximum encoded resolution). Interpolation of read regions may occur.
|2023-05-15|14:35:19.677| [INFO] Preparing Input Output Placement: 2.968948100000002
Traceback (most recent call last):
File "D:\Project\Nuclei_Seg_Cls\cerberus\run_infer_wsi.py", line 146, in
infer.process_wsi_list(run_args)
File "D:\Project\Nuclei_Seg_Cls\cerberus\infer\wsi.py", line 976, in process_wsi_list
self.process_single_file(
File "D:\Project\Nuclei_Seg_Cls\cerberus\infer\wsi.py", line 595, in process_single_file
sel_indices = [index_by_id[id(geo)] for geo in spatial_indexer.query(sel_box)]
File "D:\Project\Nuclei_Seg_Cls\cerberus\infer\wsi.py", line 595, in
sel_indices = [index_by_id[id(geo)] for geo in spatial_indexer.query(sel_box)]
KeyError: 1355689307216

I used TCGA-COAD data as input to inference. The data format is ".svs". The following is runing code with me:

###run_wsi.sh
python run_infer_wsi.py
--gpu="0"
--batch_size=25
--model="D:\Project\Nuclei_Seg_Cls\cerberus\model\resnet34_cerberus"
--input_dir="wsi_test/"
--output_dir="output_test/"
--cache_path="cache"
--save_thumb

###run_infer_wsi.py
"""run_infer_wsi.py

Usage:
run_infer_wsi.py [--gpu=] [--model=] [--nr_inference_workers=]
[--nr_post_proc_workers=] [--batch_size=] [--tile_shape=] [--chunk_shape=]
[--ambiguous_size=] [--wsi_proc_mag=] [--wsi_file_ext=] [--cache_path=]
[--logging_dir=] [--input_dir=] [--msk_dir=] [--output_dir=] [--patch_input_shape=]
[--patch_output_shape=] [--wsi_bulk_idx=] [--wsi_proc_step=] [--save_thumb] [--save_mask]
run_infer_wsi.py (-h | --help)
run_infer_wsi.py --version

Options:
-h --help Show this string.
--version Show version.
--gpu= GPU list. [default: 0]
--model= Path to saved checkpoint.
--nr_inference_workers= Number of workers during inference. [default: 0]
--nr_post_proc_workers= Number of workers during post-processing. [default: 0]
--batch_size= Batch size. [default: 30]
--tile_shape= Shape of tile for processing. [default: 2048]
--chunk_shape= Shape of tile for processing. [default: 15000]
--ambiguous_size= Define ambiguous region along tiling grid to perform re-post processing. [default: 64]
--wsi_proc_mag= Microns per pixel used for WSI processing. [default: 0.5]
--wsi_file_ext= File extension of WSIs to process. [default: .svs]
--cache_path= Path for cache. Should be placed on SSD with at least 100GB. [default: cache/]
--logging_dir= Path for python logging. [default: logging/]
--input_dir= Path to input data directory. Assumes the files are not nested within directory.
--msk_dir= Path to directory containing tissue masks. Should have the same name as corresponding WSIs.
--output_dir= Path to output data directory. Will create automtically if doesn't exist. [default: output/]
--patch_input_shape= Shape of input patch to the network- Assume square shape. [default: 448]
--patch_output_shape= Shape of network output- Assume square shape. [default: 144]
--wsi_bulk_idx= Index for batch processing. Indexing is from 0 to n-1. [default: 1]
--wsi_proc_step= Increments for batch WSI processing. [default: 10]
--save_thumb Whether to save the slide thumbnail
--save_mask Whether to save the slide mask

"""

import os
import yaml
import glob
import numpy as np
from docopt import docopt

from misc.utils import rm_n_mkdir

-------------------------------------------------------------------------------------------------------

if name == "main":
args = docopt(doc, version="CoBi Gland Inference")

# args["--gpu"] = '0'
# args["--batch_size"] = 25
# args["--model"] = "D:\\Project\\Nuclei_Seg_Cls\\cerberus\\model\\resnet34_cerberus" 
# args["--input_dir"] = "D:\\Project\\Nuclei_Seg_Cls\\cerberus\\wsi_test\\"
# args["--output_dir"] = "D:\\Project\\Nuclei_Seg_Cls\\cerberus\\output_test\\"
# args["--cache_path"] = "D:\\Project\\Nuclei_Seg_Cls\\cerberus\\cache\\"

if args["--gpu"]:
   os.environ["CUDA_VISIBLE_DEVICES"] = args["--gpu"]

input_dir =  args["--input_dir"]
output_dir =  args["--output_dir"]
logging_dir =  args["--logging_dir"]
cache_path = args["--cache_path"] + args["--wsi_bulk_idx"]
wsi_file_ext = args["--wsi_file_ext"]

# create output directory
if not os.path.exists(output_dir):
    rm_n_mkdir(output_dir)

# create logging directory
if not os.path.exists(logging_dir):
    rm_n_mkdir(logging_dir)

wsi_file_list = glob.glob(f"{input_dir}/*{wsi_file_ext}")
wsi_file_list.sort()

wsi_list = []
mask_list = []
for wsi_filename in wsi_file_list:
    wsi_basename = os.path.basename(wsi_filename)
    wsi_basename = wsi_basename[:-5]
    # only consider if mask exists! However, if mask directory not supplied, process all! 
    if not args["--msk_dir"]:
        wsi_list.append(wsi_filename)
        mask_list.append(None)
    elif os.path.isfile(args["--msk_dir"] + wsi_basename + ".png"):
        wsi_list.append(wsi_filename)
        mask_list.append(args["--msk_dir"] + wsi_basename + ".png")

proc_list = []
for idx, entry in enumerate(wsi_list):
    proc_list.append(idx)

step = int(args["--wsi_proc_step"])
start_idx = (int(args["--wsi_bulk_idx"]) - 1) * step
end_idx = int(args["--wsi_bulk_idx"]) * step

proc_list = proc_list[start_idx:end_idx]
wsi_list = np.array(wsi_list)[proc_list].tolist()
mask_list = np.array(mask_list)[proc_list].tolist()

print('Number of WSIs in list:', len(wsi_list))

run_root_dir = args["--model"]
checkpoint_path = "%s/weights.tar" % run_root_dir
with open("%s/settings.yml" % (run_root_dir)) as fptr:
    run_paramset = yaml.full_load(fptr)

target_list = ["gland", "lumen", "nuclei", "patch-class"] # all tasks considered by Cerberus

run_args = {
    "nr_inference_workers": int(args["--nr_inference_workers"]),
    "nr_post_proc_workers": int(args["--nr_post_proc_workers"]),
    "batch_size": int(args["--batch_size"]),
    "input_list": wsi_list,
    "mask_list": mask_list,
    "output_dir": output_dir,
    "patch_input_shape": int(args["--patch_input_shape"]),
    "patch_output_shape": int(args["--patch_output_shape"]),
    "save_thumb": args["--save_thumb"],
    "save_mask": args["--save_mask"],
    "mask_dir": args["--msk_dir"],
    "postproc_list": target_list,
}

wsi_run_args = {
    "msk_dir": args["--msk_dir"],
    "tile_shape": int(args["--tile_shape"]),
    "chunk_shape": int(args["--chunk_shape"]),
    "ambiguous_size": int(args["--ambiguous_size"]),
    "cache_path": cache_path,
    "logging_dir": logging_dir,
    "wsi_proc_mag": float(args["--wsi_proc_mag"]),
}
run_args.update(wsi_run_args)

from infer.wsi import InferManager

infer = InferManager(
    checkpoint_path=checkpoint_path,
    decoder_dict=run_paramset["dataset_kwargs"]["req_target_code"],
    model_args=run_paramset["model_kwargs"],
)
infer.process_wsi_list(run_args)

I didn't change the code anywhere, i just changed the path . I runing code in Windows,

Thanks

[caicai2526]

Hi!

I had sovled these problem .
thanks.

[ruqayya]

@caicai2526 could you please tell how you solved the above issue?

[caicai2526]

@caicai2526 could you please tell how you solved the above issue?
You should use the environment required by this project. At present, some libraries have been upgraded. If you install the latest library, the current error will appear.