Pixel Data with undefined length must start with an item tag

[vsoch]

Opening an issue proper here - I commented on an old PR and that's probably not desired :)

• PR in question: [MRG] Raise ValueError for invalid pixel data with undefined length #490
• Issue in deid Pixel Data with undefined length must start with an item tag deid#65

---

hey @pydicom/pydcom-fff ! We are hitting this issue in deid after cleaning data (replacing some values with 0) and trying to save the image. Complete traceback is here: pydicom/deid#65 (comment) and I'll also copy paste the basic operations for quick viewing here:

1. we read in the dicom
2. we identify a coordinate to black out, e.g.,

 coordinates
[[0, 0, 800, 59]]

Then we fill in with black (0), this is a copy of the array called "cleaned" We replace the PixelData with cleaned (and this may be where the issue arises, in how I am saving here)?

dicom.PixelData = cleaned.tostring()

and then the standard save for the dicom triggers the error seen here:

dicom.save_as(dicom_name)

Traceback (most recent call last):
  File "/home/vanessa/anaconda3/lib/python3.6/site-packages/pydicom-1.1.0-py3.6.egg/pydicom/tag.py", line 30, in tag_in_exception
    yield
  File "/home/vanessa/anaconda3/lib/python3.6/site-packages/pydicom-1.1.0-py3.6.egg/pydicom/filewriter.py", line 475, in write_dataset
    write_data_element(fp, dataset.get_item(tag), dataset_encoding)
  File "/home/vanessa/anaconda3/lib/python3.6/site-packages/pydicom-1.1.0-py3.6.egg/pydicom/filewriter.py", line 435, in write_data_element
    raise ValueError('Pixel Data with undefined length must '
ValueError: Pixel Data with undefined length must start with an item tag

Is the error one level up where we replace the data, and if not, how do we deal with this particular circumstances? I am attaching the image in question to debug, and @fimafurman and I can add further detail if information is needed. Thanks for the help!
IMG00001.dcm.tar.gz

[scaramallion]

The dataset is a compressed transfer syntax (1.2.840.10008.1.2.4.70, JPEGLossless). Compressed transfer syntaxes have 'undefined length' (i.e. 0xFFFFFFFF) and are encapsulated as per Part 5, Annex A. Uncompressed transfer syntaxes (implicit/explicit little/big) don't require encapsulation.

Assuming you're going from compressed ds.PixelData -> uncompressed ds.pixel_array -> uncompressed ds.PixelData then you'll probably first have to change the transfer syntax to an appropriate (uncompressed) value before trying to save since the file writer assumes that with a compressed transfer syntax the pixel data has been encapsulated (and hence tries to find the item tag).

If you're maintaining the compression then you first have to encapsulate your data (there are encapsulation functions in the encaps module that will be present in the v1.2 release and are in the current master), then use the resulting output to set ds.PixelData.

[scaramallion]

Would this be a better message?

ValueError: Pixel Data encoded using a compressed transfer syntax must be encapsulated however no item tag was found. Please check the transfer syntax is appropriate and that your pixel data has been encapsulated

[vsoch]

A change is a good idea! I would be more direct - if it's hit for this save, then maybe the user should hear "if saving originally compressed as uncompressed data be sure to change the ds.TransferSyntax (or whatever is the appropriate field).

For this instance, what specific commands / steps do you recommend? It sounds like the upcoming v1.2 release will have functions that could be useful, but we should have a workaround for now.

[vsoch]

hey @scaramallion could you give an example of how to use the functions here?

https://github.com/pydicom/pydicom/blob/master/pydicom/encaps.py

I tried (cleaned is a uint8 numpy array)

In [42]: encapsulate(cleaned)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-42-8a1999fc0d60> in <module>()
----> 1 encapsulate(cleaned)

<ipython-input-40-24e31b7fee0d> in encapsulate(frames, fragments_per_frame, has_bot)
    567         # `itemised_length` is the total length of each itemised frame
    568         itemised_length = 0
--> 569         for item in itemise_frame(frame, fragments_per_frame):
    570             itemised_length += len(item)
    571             output.extend(item)

<ipython-input-40-24e31b7fee0d> in itemise_frame(frame, nr_fragments)
    501     """
    502     for fragment in fragment_frame(frame, nr_fragments):
--> 503         yield itemise_fragment(fragment)
    504 
    505 

<ipython-input-40-24e31b7fee0d> in itemise_fragment(fragment)
    466     item += pack('<I', len(fragment))
    467     # fragment data
--> 468     item += fragment
    469 
    470     return item

TypeError: ufunc 'add' did not contain a loop with signature matching types dtype('S8') dtype('S8') dtype('S8')

And if a function in here works, am I allowed to include the file (unchanged with pydicom LICENSE) in deid until the official 1.2 release (that I can require for install?)

[scaramallion]

For compressed pixel data that will be written decompressed:

from pydicom import dcmread

ds = dcmread('IMG00001.dcm')
# 'JPEG Lossless, Non-Hierarchical, First-Order Prediction (Process 14 [Selection Value 1])'
print(ds.file_meta.TransferSyntaxUID.name)
# True
print(ds.file_meta.TransferSyntaxUID.is_compressed)

# Decompress Pixel Data, assuming we have the required dependencies
# Also sets the transfer syntax UID and the element length
ds.decompress()
# False
print(ds.file_meta.TransferSyntaxUID.is_compressed)

ds.save_as('out.dcm')

For compressed pixel data that has been modified and will be written compressed:

from pydicom import dcmread
from pydicom.encaps import encapsulate

ds = dcmread('IMG00001.dcm')

# Decompress and modify Pixel Data, assuming we have the required dependencies
arr = ds.pixel_array  
arr = some_function_that_modifies_arr(arr)

# Compress the Pixel Data, outside scope of pydicom
#   assumes that the results of the compression is a list of bytes, 
#   each item being the compressed data from a single frame
pixel_data = some_function_that_compresses_arr(arr)  
# Write encapsulated and compressed data back to Pixel Data
ds.PixelData = encapsulate(pixel_data )  
# Update the transfer syntax with whatever syntax matches the compressed data
ds.file_meta.TransferSyntaxUID = WhateverTransferSyntax  

ds.save_as('out.dcm')

[vsoch]

This is exactly the fix that we need! I simply added this check to the save function (and I think) it's working!

dicom = read_file(dicom_file, force=False)
dicom.file_meta.TransferSyntaxUID.is_compressed is True:
    dicom.decompress()

# cleaned is uint8 numpy array that we've been messing with
dicom.PixelData = cleaned.tostring()
dicom.save_as('/home/vanessa/Desktop/cleaned.dcm')

@fimafurman I will have a PR for you to test in a few minutes for deid.

[fimafurman]

Thank you!

On Sat, Sep 15, 2018 at 10:47 AM Vanessa Sochat ***@***.***> wrote: This is exactly the fix that we need! I simply added this check to the save function (and I think) it's working! dicom = read_file(dicom_file, force=False) dicom.file_meta.TransferSyntaxUID.is_compressed is True: dicom.decompress() # cleaned is uint8 numpy array that we've been messing with dicom.PixelData = cleaned.tostring() dicom.save_as('/home/vanessa/Desktop/cleaned.dcm') @@fimafurman <https://github.com/fimafurman> I will have a PR for you to test in a few minutes for deid. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#738 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Ak2KXGyVf-7hP1lMqhRIS0iG6HE-gjRgks5ubRMQgaJpZM4WnnrH> .

-- Fima Furman 617-512-3462 <(617)+512-3462> Director of Machine Learning and AI ffurman@ambrahealth.com [image: AMBRA-logo-Orange_RGB.png] <http://ambrahealth.com/> ambrahealth.com [image: Ambra_KLASLogos_EmailSig.png] KLAS Category Leader for Image Exchange since 2015

[vsoch]

okay ready for testing! pydicom/deid#67

When everything looks good over there, I'll take care of closing this issue too.

[vsoch]

Thanks @scaramallion @fimafurman ! Fixed with pydicom/deid#67 and https://pypi.org/project/deid/0.1.17/