Preprocess Images

[Nana2929]

Basic Dataset Stats

Note that images in trainset and testset overlap.

Preprocessing Time

HEAD_CT: 36.10 seconds
ABD_CT: 288.61 seconds
CHEST_X-Ray: 58.11 seconds
HEAD_MRI: 1276.83 seconds

Preprocessing Configuration 2023.05.14 16-43

preprocessing/__init__.py

_DEFAULT_PIPELINE_STEPS = {
    'HEAD_CT':
    [(remove_text, _MORPHOLOGY_KERNEL, _GAUSS_VALUE, (_CANNY_MIN, _CANNY_MAX)),
     (hu_transform, *_DEFAULT_HU_TRANSFORM_PARAMS['HEAD']),
     (adjust_tilt, 'HEAD')],
    'HEAD_MRI': [
        (remove_text, _MORPHOLOGY_KERNEL, _GAUSS_VALUE, (_CANNY_MIN,
                                                         _CANNY_MAX)),
        # (hu_transform, *_DEFAULT_HU_TRANSFORM_PARAMS['HEAD']),
        (adjust_tilt, 'HEAD'),
        (fcm_norm, _FCM_NORM_VALUE),
    ],
    'ABD_CT': [
        (remove_text, _MORPHOLOGY_KERNEL, _GAUSS_VALUE, (_CANNY_MIN,
                                                         _CANNY_MAX)),
        # (hu_transform, *_DEFAULT_HU_TRANSFORM_PARAMS['ABD']),
        (median_filter, ),
        # (wiener_filter, ),
        # (adjust_tilt, 'ABD')
    ],
    'CHEST_X-Ray': [
        (remove_text, _MORPHOLOGY_KERNEL, _GAUSS_VALUE, (_CANNY_MIN,
                                                         _CANNY_MAX)),
        (norm, ),
        (create_clahe, 1, 16),
        (gauss_blur, ),
        # (adjust_tilt, 'CHEST')
    ]
}

preprocessing/config.py

_DEFAULT_HU_TRANSFORM_PARAMS = {
    'HEAD': (48, 68),
    'ABD': (70, 104)
}
# define the low-quality images
# see preprocess branch: image-inspection/measure_ct_noise.py
# Q4 images
_CT_NOISE_FUNCS = [
    'median_filter',
    'wiener_filter',
]
_TO_APPLY_ABD_FILTERS = [
    'synpic23631.jpg', 'synpic41050.jpg', 'synpic32136.jpg', 'synpic22791.jpg',
    'synpic19605.jpg', 'synpic26697.jpg', 'synpic21902.jpg', 'synpic42157.jpg',
    'synpic16520.jpg', 'synpic40596.jpg', 'synpic48714.jpg', 'synpic46943.jpg',
    'synpic33889.jpg', 'synpic23571.jpg', 'synpic23008.jpg', 'synpic28180.jpg',
    'synpic42951.jpg', 'synpic33844.jpg', 'synpic54823.jpg', 'synpic38630.jpg',
    'synpic26158.jpg', 'synpic22684.jpg', 'synpic22982.jpg', 'synpic22020.jpg',
    'synpic58261.jpg', 'synpic45914.jpg', 'synpic34922.jpg', 'synpic28695.jpg',
    'synpic43433.jpg', 'synpic21028.jpg', 'synpic29219.jpg', 'synpic24967.jpg',
    'synpic24220.jpg'
]
_MORPHOLOGY_KERNEL = np.ones((5, 5), np.uint8)
_GAUSS_VALUE = 5
_CANNY_MIN = 230
_CANNY_MAX = 250
_FCM_NORM_VALUE = 0.8

Note

1. Some Head_MRI imgs do not have clear enough contour for adjust_tilt
   to perform, hence these images are skipped for this operation.
   (> 25 photos)
2. Need to execute lib/utils/create_resized_images.py according to lib/utils.run.sh again after preprocessing the images. Beware of the channel number.

[Nana2929]

Issue with ABD_CT

• Attempt 1: Cancel REMOVE_TEXT in DEFAULT PIPELINE
  -> Still has these zig-zag edges. Turns out Weiner filter is the culprit.
  -> Remove Weiner filter.

• 26158
  

• 43433