DICOM Files with Large Number of Frames

[chrisbll971]

Hello,

We were wondering if there is any way to create DICOM files with large numbers of frames? We are trying to create DICOM files for digital pathology and noticed if they have a large number of frames (30,000+) then this line panics. This specific code is creating 32,000 JPEG2000 frames.

Do you know if there are any potential workarounds to creating these files? At a rough glance it looked like it might be coming from the offset table which is u32 and capped to roughly 4 billion in length and were curious if potentially it could be u64?

https://doc.rust-lang.org/std/u32/constant.MAX.html

let mut all_fragments = vec![];
let mut total_frame_len = 0;
for (ix, frame) in frames.iter().enumerate() {
    let frame_len = frame.len();
    total_frame_len += frame_len;
    println!("Adding frame {} with len {} and total len {}", ix, frame_len, total_frame_len);
    let fragments = Fragments::new(frame.to_vec(), frame_len.try_into().unwrap());
    all_fragments.push(fragments);
}

file.put(DataElement::new(
   tags::PIXEL_DATA,
   VR::OB,
   PixelSequence::<InMemDicomObject, InMemFragment>(all_fragments.into()),
));

Message:

Adding frame 32913 with len 271 and total len 4791604010
Adding frame 32914 with len 271 and total len 4791604281
Finished converting frames to fragments for level 0 with 32915 frames
thread 'main' panicked at dicom-core-0.6.3/src/value/fragments.rs:106:35:
attempt to add with overflow

[Enet4]

Thank you for reporting. You are right that the Fragments API bears a few limitations right now, but apparently it can also be misleading. Creating Fragments should only be necessary if you have a very large frame that needs to be split into fragments. In your case, if you have a list of frames and intend to fit exactly one frame in each fragment, I would recommend skipping that API altogether.

let mut all_fragments: Vec<_> = frames.iter().enumerate().map(|(ix, frame)| {
    println!("Adding frame {} with len {}", ix, frame.len());
    frame.to_vec()
}).collect();

file.put(DataElement::new(
   tags::PIXEL_DATA,
   VR::OB,
   PixelSequence::<InMemDicomObject, InMemFragment>(all_fragments.into()),
));

You might encounter two other issues:

1. The accumulation of all frame lengths in total_frame_len could also panic. This is generally not necessary unless you wish to provide Encapsulated Pixel Data Value Total Length (7FE0,0003). If you skip it or make it a u64, it should no longer overflow.

2. One other problem is that pushing all frames into a DICOM object will require all frames to be in memory. The future of the project may hold nicer ways to write DICOM data sequentially.

[chrisbll971]

Ok awesome and good to know. In the example above do you happen to know if I will need to write the pixel data with a different approach as well? On my end it is showing that the From<> trait isn't implemented for converting a Vec<Vec> into a PixelSequence.

let mut all_fragments: Vec<_> = frames.iter().enumerate().map(|(ix, frame)| {
    println!("Adding frame {} with len {}", ix, frame.len());
    frame.to_vec()
}).collect();

file.put(DataElement::new(
   tags::PIXEL_DATA,
   VR::OB,
   PixelSequence::<InMemDicomObject, InMemFragment>(all_fragments.into()),
));

the trait bound `PixelFragmentSequence<Vec<u8>>: From<Vec<Vec<u8>>>` is not satisfied

[Enet4]

Hmm right. You can try PixelFragmentSequence::new_fragments instead. It makes it clear that it keeps the basic offset table empty, which is probably why I withdrew the idea of implementing From<_> for this case.

file.put(DataElement::new(
   tags::PIXEL_DATA,
   VR::OB,
   PixelFragmentSequence::new_fragments(all_fragments.into()),
));

[chrisbll971]

This worked. Thanks so much and the library is great.