Access Violation Error on remove_outlier3d

[Maksim-Yakovlev]

Describe the problem
Hello! I have installed tomopy following the instructions on https://tomopy.readthedocs.io/en/latest/install.html and have successfully used it to reconstruct several test datasets in a conda environment, running the angles, normalize, find_center, minus_log, and recon functions without a problem. However, when trying to dezinger the projection data using remove_outlier3d, I receive the following error:

Running remove_outlier works successfully but expectedly slow on a 16 GB file. Any help getting remove_outlier3d to remove hot pixels would be greatly appreciated!

To Reproduce
An image is read from an hdf5 file, angle space is calculated, and normalization performed. I'm excluding the latter part of the code which does the reconstruction and writes the output, which works without a problem. The error is thrown whether remove_outlier3d is run before or after the normalization step. The data type of proj_data after normalization is float32, and the normalization step works as expected as confirmed by writing the output and checking in FIJI.

import numpy as np
import tomopy
import dxchange as dx

if __name__ == "__main__":
    # ID file
    fname = 'filename.h5'
    
    # Read projections and flat/dark fields
    proj_data = dx.read_hdf5(fname, "exchange/data")
    wf = dx.read_hdf5(fname, "exchange/data_white")
    df = dx.read_hdf5(fname, "exchange/data_dark")
    
    #Determine total rotation angle of scan, scanDeg can also be manually set
    scanDeg = dx.read_hdf5(fname, "process/acquisition/rotation/rotation_step")
    scanDeg = scanDeg * proj_data.shape[0]

    # Set data collection angles as equally spaced in radians
    theta = tomopy.angles(proj_data.shape[0], ang1=0, ang2=scanDeg)

    # Preprocess data using flat/dark fields
    proj_data = tomopy.normalize(proj_data, wf, df)

    # Dezinger *this is where the error is thrown**************************
    proj_data = tomopy.remove_outlier3d(proj_data, dif=0.5, size=3)
    

Traceback (most recent call last):
  File "C:\Users\yakovlev\Work\Code\Tomopy_Recon_Test\MSU_Maoz_4A_16GB_360deg\tomoTestMaoz.py", line 203, in <module>
    proj_data = tomopy.remove_outlier3d(proj_data, dif=0.5, size=3)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\yakovlev\AppData\Local\anaconda3\envs\TomoRecon\Lib\site-packages\tomopy\misc\corr.py", line 458, in remove_outlier3d
    extern.c_median_filt3d_float32(np.ascontiguousarray(arr), out, kernel_half_size, dif, ncore,
  File "C:\Users\yakovlev\AppData\Local\anaconda3\envs\TomoRecon\Lib\site-packages\tomopy\util\extern\misc.py", line 125, in c_median_filt3d_float32
    LIB_TOMOPY_MISC.medianfilter_main_float(
OSError: exception: access violation reading 0x000002BCCEB9F020

Expected behavior
For remove_outlier3d to return a numpy array the same size as the input projection stack, with pixels deviating from the local containing 3-pixel median by 0.5 or more to be set to the median value.

Helpful images
Posted above

Platform Information:

• OS: Windows 11
• Python Version 3.11.5
• TomoPy Version 1.14.1

Additional context
Issue was reproduced on python version 3.7

Thanks!

[carterbox]

16GB seems like a large dataset. What dimensions is the proj_data that you are providing to remove_outliar3d? Can you try to reproduce your issue with some random data? i.e. something like

    import numpy
    import tomopy
    rng = np.random.default_rng()
    proj_data = rng.standard_normal( <large size here> )  
    proj_data = tomopy.remove_outlier3d(proj_data, dif=0.5, size=3)

[carterbox]

@dkazanc Any other ideas?

[dkazanc]

Thanks, will have a look next week.

[Maksim-Yakovlev]

16GB seems like a large dataset. What dimensions is the proj_data that you are providing to remove_outliar3d? Can you try to reproduce your issue with some random data? i.e. something like

    import numpy
    import tomopy
    rng = np.random.default_rng()
    proj_data = rng.standard_normal( <large size here> )  
    proj_data = tomopy.remove_outlier3d(proj_data, dif=0.5, size=3)

I was indeed able to replicate the issue. Running the following code, I get the same error. The dataset is a collection of 3600 projections on a 1200 x 1920 array, so 3600 x 1200 x 1920 of data type float32.

import numpy as np
import tomopy
import dxchange as dx

if __name__ == "__main__":

    rng = np.random.default_rng()
    proj_data = rng.standard_normal([3600,1200,1920]) 
    proj_data = tomopy.remove_outlier3d(proj_data, dif=0.5, size=3)

[dkazanc]

thanks, was able to reproduce on Linux as well using smaller dataset actually. I think I know the issue as the index overflows given long data type. Will correct and do PR.

[carterbox]

Thanks for reporting this issue! The release with the fix should be available sometime tomorrow?