Low Quality Segmentation Results on the M3D Dataset

[Humanoid-a]

Hi! Thanks for your work!
I am trying to perform segmentation with vista3d on the m3d dataset, however the results are fuzzy.
I used the monai bundle with these commands:

pip install monai==1.4.0;
python -m monai.bundle download "vista3d" --bundle_dir "bundles/";
cd bundles/vista3d

python -m monai.bundle run --config_file configs/inference.json

with the default settings, the segmentation results are simply sparse spots, while changing the ScaleIntensityRanged's a_min and a_max into the following results in a very fuzzy segmentation result as in the following images.

   { "_target_": "ScaleIntensityRanged",
      "keys": "@image_key",
      "a_min": 0, "a_max": 1,
      "b_min": 0.0,   "b_max": 1.0,
      "clip": true },

I have viewed this issue and I cant find a way to fix this.
sorry for disrupting!!

[heyufan1995]

Hi @Humanoid-a
Seems like the M3D did some preprocessing and caused normalization issue. If it is normalized to 0-255, then set a_max=255, a_min=0, b_max=1, b_min=0. If it's normalized to 0-1 using (x-x_min)/(x_max-x_min), which is different than (x- (-963)) / (1054 - (-963)). It could cause this issue. Could you try other scans? usually x_min and x_max should be around -1000 and 1000 and it should not make such difference.

[Humanoid-a]

yes, the m3d dataset volumes are all normalized to 0-1. I have tried with multiple scans, and they all result in similar issues. Are there any ways to fix this? Here is another scan:

this one is with a_max = 1, a_min = 0

this one is with the default settings:

here is the image info

[heyufan1995]

@Humanoid-a
Are you able to locate the scan from the original dataset? M3D are using public data right? Could you find one and download the scan from original datasets and test the segmentation? Then we can find out what's wrong.

[Humanoid-a]

@heyufan1995
Ok, so the M3D dataset was mostly crawled from radiopaedia, and they were given as jpg files, and a demo script that stacked them together was provided. I will show the operations they did below.
However, a concerning factor was that no HU windows or values were provided, while the images were labeled, for example, Axial_lung_window or Axial_bone_window, which seems to be of different HU windows.

For each image they read in, after basic RGBA conversion to single 2d matrix, they did:
img_array = img_array.astype(np.float32) / 255.0
To stack the image, they first select a “most common image shape” and remove all the other images not of this shape (which also seems concerning), and finally stack these 3d images:

images_3d_pure = []
try:
    img_shapes = [img.shape for img in images_3d]
    item_counts = Counter(img_shapes)
    most_common_shape = item_counts.most_common(1)[0][0]
    for img in images_3d:
        if img.shape == most_common_shape:
            images_3d_pure.append(img)
    final_3d_image = np.vstack(images_3d_pure)

    image = final_3d_image[np.newaxis, ...]

After stacking the images and creating a 3d image, they did:

#this image is the stacked 3d image
image = image - image.min()
image = image / np.clip(image.max(), a_min=1e-8, a_max=None)
img_trans = np.squeeze(transform(image), axis=0)

where transform is:
transform = mtf.Compose([
    mtf.CropForeground(),
    mtf.Resize(spatial_size=[32, 256, 256], mode="bilinear")
])

I was not able to retrieve the raw data right now, while the data in the M3D provided seems to be directly the images they crawled. Is this enough information to determine why? Are there anything I could do to fix the problem with this dataset or do I have to switch one?

[heyufan1995]

@Humanoid-a
The dataset saved in jpeg then normalized to 0-1 and reshaped to 256x256 basically lost all the physical properties. No pixel resolution, no HU, no orientation. Vista3d was trained on CT scans standardized to 1.5mm resolution, normalized HU and RAS orientation, which is based on the correct header information. So I don't think vista3d can work well on this dataset. Probably you can try totalsegmentator to see if they can generalize to this non standard scans.