Plot methods expecting file 000000.dcm when it does not exist

[jovsa]

When plotting scans, annotations or contours, plotting functions expect a 000000.dcm file when LIDC-IRC dataset does not have them.

Bug related to plotting methods: scan.visualize(), ann.visualize_in_scan() and ann.visualize_in_3d()

Error trace:

---

FileNotFoundError Traceback (most recent call last)
in ()
1 ann = pl.query(pl.Annotation).first()
----> 2 vol, seg = ann.uniform_cubic_resample(side_length = 100)
3 print(vol.shape, seg.shape)
4 # => (101, 101, 101) (101, 101, 101)
5

/home/plexadmin/.virtualenvs/data-science-bowl-2017/lib/python3.5/site-packages/pylidc/Annotation.py in uniform_cubic_resample(self, side_length, verbose)
750
751 # Load the images. Get the z positions.
--> 752 images = self.scan.load_all_dicom_images(verbose=verbose)
753 img_zs = [float(img.ImagePositionPatient[-1]) for img in images]
754 img_zs = np.unique(img_zs)

/home/plexadmin/.virtualenvs/data-science-bowl-2017/lib/python3.5/site-packages/pylidc/Scan.py in load_all_dicom_images(self, verbose)
308 images = []
309 for dicom_file_name in sorted_fnames:
--> 310 with open(os.path.join(path, dicom_file_name), 'rb') as f:
311 images.append( dicom.read_file(f) )
312 return images

FileNotFoundError: [Errno 2] No such file or directory: '~/lidc_idri/data/LIDC/DOI/LIDC-IDRI-0078/1.3.6.1.4.1.14519.5.2.1.6279.6001.339170810277323131167631068432/1.3.6.1.4.1.14519.5.2.1.6279.6001.303494235102183795724852353824/000000.dcm'

[jovsa]

If you add a dummy 000000.dcm, plotting functions work as expected. However this involves tampering the raw data

[notmatthancock]

For this annotation, will you list the contents of the directory given by

ann.scan.get_path_to_dicom_files()

?

[jovsa]

Using the basic example code, all 97 patients don't have file 000000.dcm

.zip below contains my code used to replicate this issue:
Debugging+pylidc.html.zip

[notmatthancock]

Ok, thanks. Comparing your file listing with mine, we have the same number of DICOM files, but it appears that mine start with 0 while yours start at 1.

Now, I obviously don't have control over how the DICOM files are delivered on the TCIA download site. You can see in the existing code to load the DICOM images, that there is already a hack for the image loader because in some cases, the DICOM image files download with file names that are zero-padded, and in others they don't. I don't know why this is. In any case, it looks like the loader will need another hack to account for this indexing issue.

Will you try replacing the block of code in the above link (in Scan.py on your system) with

# Some sets have the dicom files padded with 
# zeros in front, some don't apparently.
if all([(len(fname)==len(fnames[0])) for fname in fnames]):
    L = len(fnames[0]) - len('.dcm') # Just being explicit here.
    str_format = ("%0"+str(L)+"d.dcm")
    
    # If the zero file doesn't exist, start at 1.
    offset = not os.path.exists(str_format % 0)

    for i,fname in enumerate(sorted_fnames):
        ii = int(fname.split('.')[0]) + offset
        sorted_fnames[i] = str_format % ii

and running files = scan.load_all_dicom_images();?

[jovsa]

@notmatthancock your fix works (with a small modification). Please see my PR above - thanks!

[notmatthancock]

Glad you got it working.