Image re-ordering Philips with dynamic scans #529
Comments
|
Closing this issue as I do not have any access to a sample dataset that exhibits this issue. Either someone needs to validate a solution and provide a pull request, or I need to be provided with a sample dataset that exhibits this issue. As an aside, while dcm2niix should handle these situations, Philips equipment should generate images in an order that is meaningful (e.g. sequential instances are sequential in space or time). I suspect the current jumbled order reflects the order of data returned from a parallel reconstruction system. While we can fix this one tool, such jumbled data is likely to cause grief for others. There are lots of legal ways to make a legal DICOM that is inscrutable, manufacturers should seek parsimonious solutions for this complicated format. Philips should be admonished for this behavior. We need to cure the root problem, not treating the symptoms. |
|
@neurolabusc I have emailed you an example of what is possibly the same problem - an fMRI where the slices are getting mis-sorted by v1.0.20210317. There is no relationship between InstanceNumber and slice position or TemporalPositionIdentifier in the DICOM files, which I gather might be the underlying issue. These classic frames were exported on the scanner using a python interface to the DICOM leacher. |
|
@baxpr your issue seems different, but is indicative of how Philips assigns instance numbers with no rationale. My kludge generates an instance number based on three private Philips tags. A sensible formula would be: However, your does not follow a logical pattern, and indeed uses a different pattern for each value. You can see this by running the development branch of dcm2niix in verbose mode ( The state of Philips DICOM images has been in disarray for several years now. They were the first to market with enhanced DICOM, but this was implemented in a very inefficient way both in terms of disk space and parsing speed. Since then, Canon and Siemens have released much cleaner enhanced DICOMs. While the initial XA10 Siemens had a lot of rough edges, Siemens quickly cleaned things up. In contrast, Philips seems to have not addressed the issues with either their enhanced or classic scans. I notice that your protocol is listed as a WIP. I am not sure I want to retain the kludge I wrote today in the final form. My sense is use should use the manufacturer provided NIfTI export functions until Philips is able to resolve their DICOMs. To be clear, this is a limitation of Philips images, not my software. Their implementation impacts other tools like Horos and dicm2nii. |
|
Hi The instances of data that have been produced which show have been obtained from a research interface into the patient database and so is not widespread but is being relaunched as a research tool for our users (PRIDE 2.0 is our reference to it) and allows software to directly communicate to the scanner interface and obtain images for processing. The standard DICOM export does present them differently. However both methods are unlikely to consider this a big issue and will probably not be resolved in the short term as there is no requirement to present images in any defined order in the DICOM and while its annoying (and I wish it were different) we will see this as a defect in the recieveing software making unwarrented assumptions. WIth increasingly multidimensional data sets, there is also a question as even if a defined order was chosen, the assumptions are different between different softwares (some use Z,T,Type, others Type,Z,T etc. the only solution is to correctly read the dimension information and act on it. - it is an unfortunate consequence that DICOM is principally a communications protocol bewteen two databases and the dicom file is an attempt to freeze this communcation stream to disk but without any negotiation or handshaking. All the information is correct and available to allow the images to be correctly identified - this does not necessarily become memory inefficient aside from storing the dimension vectors somewhere prior to writing image data to disk. Instance number is just that - a unique identifier for the image - indeed it was renamed this in the earlier DICOM standard from "image number" to imply the unlinking of any physical meaning for this value. I'm not sure I would agree with the terms "in disarray" (unless you mean this very literally in that the images are not provided in an array format :-) ) Also - unfortunatley our provided in built nifti export is limited to certain image types and sequences due to the requirements of only producing a single nifti file, so splitting and mutli file types are currently not supported. M |
|
@drmclem here are four objective criteria that suggest disarray to me:
Please do not get my tone wrong, you have been outstanding at providing example data and describing properties of Philips DICOM data. However, it is discouraging that none of the issues are making their way up stream. Looking at the dcm2niix code shows a huge number of kludges have been inserted to handle Philips data. I worry that since these are being done in isolation from the manufacturer, they are inherently fragile. Consider the kludge discussed above for @baxpr - how will this behave for multi-echo data? Does Philips only use the private volumeNumber tag (2005,1063) for fMRI, or will this also work for diffusion data? I strongly encourage Philips to continue and extend the development of their NIfTI export to handle the broad range of data the instruments can acquire.
With regards to the comment |
|
Hi I think part of the problem is that there as aspects of DICOM which are not fully defined, and certainly public tags we are incorporating as we move forward in software releases and any errors in those we will fix as we move forward (4) - we are looking how best to populate the defined timing blocks in the light of multiband sequences and the like. I would also argue that using the instance number as anything other than a unique identifier is error prone. A comprehensive solution based on public tags which works for philips data should also work for the other vendors, but I also your relcutance to move such solutions from Philips specific to general given that "if not broken don't change it". I think we also need to be careful of confusing public tag requirements and private tag requirements - there is much additional information in there to support recall and rescanning of the patient and our internal useage, which i suspect is behind 1 and 2. While these can be stripped out on PACS communication (requesting "Pure" dicom for example) the disk file export cannot state its requirements in advance through handshake, but equally some of what you may consider bloat, it vital to other or our own users. Specifically for 3 - the background is with the move to parallel moving that cost onto the scanner is detrimental to image reconstruction performance during the scan and is not particularly an issue for PACS - and as we have seen, the standard DICOM File export has resolved much of this - (but relying on instance number is inherently fragile) - this new research interface does not (currently, we have reported the issue and we will see what happens). For 5 - the prinicple user is the PACS community, we understand the the needs of the research community often needs internal information which is not currently defined in the DICOM standard and therfore requires private tags - but in general private tags (as they are intended only to be used by the vendor who created them) are difficult to request and inevitably these lag behind the research community needs but we do try be responsive to them but I would argue we have introduced a lot of tooling to support this for Philips users, just not in the DICOM-nii axis. Even for (3) "There are no official DICOM tags to report this information - on Siemens scanners we have reverse engineered the proprietary CSA header 16." For your final point on Nii export from the tools - it is there but we need good reference and fixed standards to work with (currently based on https://nifti.nimh.nih.gov/nifti-1 and support fMRI and structural formats). It is a testement to your software that it has almost become by definition the reference implementaion (without even considering the BIDS component) and I think support multiple evolving standards is always difficult - I think our efforts are better placed in ensuring the DICOM implementation meets the needs. For 3 and 4 - the problem is that a modern data set potentially is (x,y,z,temporal, image type (R,I,M,P), state (label,control for example) with some elements not unique (for example many z same t as for current dyanmics scans, or unique z - t for fmri with slice timing etc. and different users have different needs - if you are going to sort them anyway ... On the specifics for the solution to the above ill put in a separate comment if thats ok. M |
|
Hi @baxpr - any chance you could share the dataset with me so I can have a look at it and compare the dataset which started this convo ? |
|
Your comment regarding Siemens actually supports my point. Note the move from the proprietary CSA header in the V* systems (Trio, Prisma) to public tags in the XA systems (Vida, Sola). Between XA10 and XA11 there was a constructive dialog with users regarding the meta data that would allow replication and comprehensive analysis. |
|
Thanks y'all - really appreciating this discussion which I'm learning a lot from. @drmclem I have sent you the images as well. The spatial and temporal ordering can be reconstructed from ImagePositionPatient and TemporalPositionIdentifier, but I gather the assumptions that dcm2niix usually makes for Philips images are not met. |
|
fwiw I banged out some python code to rewrite instance numbers so that dcm2niix can handle the DICOMs we have: |
|
Hi,
Best regards, Rob |
|
@baxpr and @captainnova do you want to try out the latest commit? It attempts to generalize @baxpr Python code (which was not designed for diffusion data). Here are details regarding the implementation and the implications:
In brief, while this method seems to provide a robust solution for the problem (as long as point 5 does not pose an issue), the ordering for DTI data may be different than previous versions. This makes regression testing tricky. Also, since Philips often stores the b=0 volume as the first to disk, but gives it the highest GradientOrientationNumber, there may be complications with scripts that assume that the first volume is the reference volume (e.g. if one does not specify the |
|
Thanks! Yes, that works for my test fmri. @neurolabusc I've sent you a diffusion scan that might help. We quit making assumptions about ordering of diffusion volumes in the niftis a couple years ago due to these issues. Philips does not permit multiple b=0 to be specified, as far as I know - the data set I just sent you is how we have worked around that using a negligible b value and some arbitrary specified direction. ... That is an enhanced DICOM though, generated in a different way. I'll see if I can get a classic set for you. |
|
Unfortunately it broke diffusion. A scan with 1 b=0 followed by 32 b=999, and an apparent ADC probably at the end came out with the b=0 volume distributed among the b=999 volumes 2 slices at a time, i.e. 2 of the slices were replaced by the b=0 slices of the correct z. The bumped slices were placed at the highest z. The "ADC" file is mostly the average of the b=999 volumes (which is normal - it wasn't really the ADC to begin with), except that two of its slices have also been replaced with b=0 and moved to the top of the image. The array shape, .bval, and .bvec files were not changed. Although the ImageType says ["ORIGINAL", "PRIMARY", "M", "SE", "M", "SE"], I think this series should really have been called DERIVED, since the SeriesDescription and ProtocolName say "Reg - AX_DTI SENSE", i.e. it was the MoCo version of the scan. Rob |
|
I should have given the dim field from the nii header: But again, it's probably best not to focus on this particular one yet. All I know is it used to work. |
|
Good news - enhanced DICOM (but a different scan, that really is ORIGINAL data) seems to be unharmed. The other series had been unenhanced with dcuncat. |
|
|
Hi all, I'm the researcher who has been working with @drmclem, whose DICOM data set triggered this issue. I've caught up with the thread this morning. I've just cloned and run the development branch of dcm2niix following the adjustments by @neurolabusc. Can confirm that it has fixed the issue for our DICOMs. So, thank you very much @neurolabusc for the fix and @baxpr for your initial Python implementation too. |
|
Paul Morgan at Nottingham University has provided images for a new validation repository dcm_qa_philips_asl. Paul provided both the enhanced and classic images, but the combined set seems to overwhelm Github, so this repository is only the classic images. I have updated dcm2niix to handle these images. This should now deal with diffusion, ASL and fMRI data. The ASL data has a few quirks (as described with the validation), so the dcm2niix commit includes a lot of heuristics to handle that modality. Hopefully others can test this on a broader set of images. The ordering of ASL volumes with respect to repeat, phase and presence of label is arbitrary, so I just copied the order observed for the enhanced images. It would be great if Philips could provide a bit more ASL sequence details for these DICOMs. BEP005 provides a community wishlist that will allow automated analyses for this modality. |
|
I no longer think dcuncat is the likely reason why the development version of dcm2niix missorts the slices in some of our test dMRI series. The affected series are from Philips 2, 3, and 4 software, and have been through at least one deidentification step (dcanon by me, and possibly others before they were sent to our lab), and I have lost track of whether the scanner stored them in classic or enhanced format, if I ever knew. However, in the old days it was standard practice for people here to run dcuncat on enhanced DICOM as soon as it arrived, so unfortunately for the data I have in hand the scanner software version is strongly conflated with whether it has been dcuncatted. A Philips 5 dMRI that we scanned here works much better, although development dcm2niix prints some spurious error messages when working with the original enhanced version. It looks like it has well behaved volume indices, while the "volume indices" for the older scans go from 1 to 65537 and do not work with the new scheme. |
|
I am closing this issue. The current development version comprehensively addresses this issue. In cases where all the useful tags are stripped the algorithm resorts to assuming that the instance number is truthful, resolving the regression for the (over aggressively) anonymized data from @captainnova . However, this fixes raises deeper concerns regarding how Philips ASL volumes should be ordered. This problem is introduced in issue 533. |
* commit 'v1.0.20200427-207-g66a4fd0': (76 commits) Philips ASL in temporal order (rordenlab#533) Optional compilation to disable kludge for issue 532, e.g. "make CFLAGS=-DmyDisableGEPEPolarFlip" (rordenlab#532) 0020,9157 if dcuncat, use 0020,0013 if 0019,10A2; 0020,0100; 2005,1063; 2005,1413 do not vary (rordenlab#529) Add BIDS ArterialSpinLabelingType field Diagnostics for issue 529 (rordenlab#529) Flip row order for kGE_EPI_PEPOLAR_REV Detect and correct GE epi_pepolar sequence (rordenlab#532) More CT meta data, detect manufacturer MEDISO Philips ASL volume order (rordenlab#529) ensureSequentialSlicePositions verbosity Handle Philips images where 2005,1063 is set to zero for all volumes, see Magdeburg_2014 from dcm_qa_philips Philips MR 51.0 @baxpr kludge (rordenlab#529) Remove redundancy Kludge for Philips random instance numbers (rordenlab#529) BUG: Fix preprocessor conflict. Support ancient Linux, tested on holy-build-box (rordenlab#531) undefine DT_NONE (https://neurostars.org/t/adjusting-for-negative-mosaicrefacqtimes-issue-271-warning-but-with-normal-slice-times/19866/3) UIH new TotalReadoutTIme formula (rordenlab#531) UIH uodates Allow acquisition number (0020,0012) in filename (rordenlab#526) ...
|
@drmclem and @sandeepganji please take my tone as constructive. However, I do think that the Philips enhanced DICOM places a tremendous burden on users. I was originally lured to South Carolina by the opportunity to use the brilliant Philips MRI systems. Years later, our entire hospital system is moving exclusively to Siemens. I do think Philips continues to make outstanding hardware, but data storage, transfer and display is a major factor for clinical sales. As a consumer, I have a vested interest in seeing competitive alternatives. I do urge Philips to examine their competitors modern enhanced DICOM implementations. Here is data from Canon saved as both enhanced and classic DICOM. The uncompressed classic data requires 458mb, the enhanced is just 76mb. Likewise, dcm2niix is able to convert the clean enhanced DICOM in about 1/10th the time of classic data. The meta data is human readable, instead of overwhelming humans and machines with redundant data. |
We have a classic DICOM data set which I think has both the images is indirect order and is also a dynamic scan (something I would expect to be typical for an fmri for example), However the converted files have the slices scrambled.
Looking at the logs it would appear that re-ordering is only being applied to the conventional scans not the dynamics.
A working one ... (single scan)
Not working - no test done and no re-ordering ?
Would you expect this to work ?
Does this code imply that it only is applied to 4d scans with 1 in the 4th dimension ? (latest git hub version)
Best wishes
Matthew
The text was updated successfully, but these errors were encountered: