Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove minc, add a minc based on SeriesUID #138

Merged
merged 17 commits into from Nov 1, 2016

Conversation

gluneau
Copy link
Contributor

@gluneau gluneau commented Jun 29, 2016

New script to delete minc file based on SeriesUID

uploadNeuroDB/minc_deletion.pl -profile prod -seriesuid 1.3.12.2.1107.5.2.32.35045.201507232316035928639007.0.0.0 confirm

./uploadNeuroDB/minc_deletion.pl -help                                                                                              
*******************************************************************************
Minc Deletion
*******************************************************************************

Author  :   Gregory Luneau
Date    :   July 2016
Version :   1 revision  3


The program does the following:

Deletes minc files from Loris by:
  - Moving the existing files to an archive directory.
    .mnc .nii .jpg .header .raw_byte.gz
  - Deleting all related data from 2 database tables.
    parameter_file & files
  - Deletes data from files_qcstatus & feedback_mri_comments
    database tables if the -delqcdata is set. In most cases
    you would want to delete this when the images changes.
  - Deletes mri_acquisition_dates entry if it is the last file
    removed from that session.

Use the argument "select" to view the record that could be removed
from the database.  Use "confirm" to acknowledge that the data in 
the database will be deleted once the script executes.


Summary of options:

-- Basic options ---------------------------------------------------------------
   -profile       config file in ../dicom-archive/.loris_mri [default: ""]
   -seriesuid     Only deletes this SeriesUID [default: ""]
   -fileid        Only deletes this FileID [default: ""]
   -delqcdata     Deletes QC data
   -nodelqcdata   opposite of -delqcdata [default]

usage: ../uploadNeuroDB/minc_deletion.pl [options] select|confirm
       ../uploadNeuroDB/minc_deletion.pl -help to list options

Also, modification of tarchiveLoader so that a unique file can be inserted based on SeriesUID

uploadNeuroDB/tarchiveLoader -globLocation -profile null_grads -seriesuid 1.3.12.2.1107.5.2.32.35045.201507232316035928639007.0.0.0 DCM_2015-07-23_ImagingUpload-10-38-WiBmLY.tar

@ted-strauss-K1
Copy link
Contributor

                              ���
                                 ���
                                  |||
                                  |||
                                  |||
                         oooo$$$$$$$$$$$$oooo
                      oo$$$$$$$$$$$$$$$$$$$$$$$$o
                   oo$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$o
                 o$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$o
               o$$$$$$          $$$$$$$          $$$$$$o
              o$$$$$$$$        $$$$$$$$$        $$$$$$$$$o
             $$$$$$$$$$$      $$$$$$$$$$$      $$$$$$$$$$$$
           $$$$$$$$$$$$$$    $$$$$$ $$$$$$    $$$$$$$$$$$$$$
          o$$$$$$$$$$$$$$$ $$$$$$$   $$$$$$$ $$$$$$$$$$$$$$$
          o$$$$$$$$$$$$$$$$$$$$$$     $$$$$$$$$$$$$$$$$$$$$$$
          $$$$$$$$$$$$$$$$$$$$$$       $$$$$$$$$$$$$$$$$$$$$$
          $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$" "$$$
          $$$$$  $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$   o$$
           $$$$    $$$$  ""$$$$$$$$$$$$$$$$$""  $$$$   o$$$
            "$$$$o         $$$$  $$$$$   $$$          o$$$
               $$$$o   .     ""  """""   ""       .  $$$"
                "$$$$o $$$oo                   oo$$o$$$
                  "$$$$$$$$$    o$$$$   $$$$   $$$$$""
                     ""$$$$$$ooo$$$$$ooo$$$$$$$$$""
                        ""$$$$$$$$$$$$$$$$$$$$
                                """$$$$"""

                             Happy Halloween!

@ted-strauss-K1
Copy link
Contributor

 #
 ##
 ###
  ####
   #####
   #######
    #######
    ########
    ########
    #########
    ##########
   ############
 ##############
################
 ################
   ##############
    ##############                                              ####
    ##############                                           #####
     ##############                                      #######
     ##############                                 ###########
     ###############                              #############
     ################                           ##############
    #################      #                  ################
    ##################     ##    #           #################
   ####################   ###   ##          #################
        ################  ########          #################
         ################  #######         ###################
           #######################       #####################
            #####################       ###################
              ############################################
               ###########################################
               ##########################################
                ########################################
                ########################################
                 ######################################
                 ######################################
                  ##########################      #####
                  ###  ###################           ##
                  ##    ###############
                  #     ##  ##########
                            ##    ###
                                  ###
                                  ##
                                  #

@ted-strauss-K1
Copy link
Contributor

                         ______
                         ___)    (___
                      __/     __     \__
                     /     \_/  \_/     \
                    /      / \__/ \      \
                   |          /\          |
                  /       ---(  )---       \
                 |            \/            |
                 | ........................ |
                / ._________/    \_________. \
               | ./ _________    _________ \. |
               |./ //#/\###())~~(()###/\#\\ \.|
               |/ //#([])##// ~~ \\##([])#\\ \|
               | //###\/##//  ~~  \\##\/###\\ |
               | \\######//   ~~   \\######// |
                \             /\             /                  .^. .^.
                 )      __   !II!   __      (                   |~| |~|
                /' .___//    !II!    \\___. '\                  |~| |~|
               (  ' .__/     [II]     \__. '  )         __      |~| |~|
                \  '          YY          '  /          \)\_    |~| |~|
                 \  '  |   ________   |  '  /            \~~\   |~~V~~|
                  \  ' | _/'      '\_ | '  /              \~~\_ |~~|~~~\
                   \  '|/____________\|'  /                \~~~\~~~~~~~~)
                    \   V___======___V   /                  \~~~\~~~~~~/
                     \  V   \____/   V  /                    \~~~|~~~~~\
                      \ V   ~~  ~~   V /                      )~~|~~~~~~)
                       \V  ~~\~~/~~  V/                       \~~~~~~~~/
                       _V ~~~~\/~~~~ V_                        \~~~~~~|
                   ___/ V\____/\____/V \_____                   \~~~~~|
                __/ :   V    :  :    V   :  :\___                |====|
              _/' .' .' V  .' .' .'  V .' .' .' .\_              |.' .|
             /:  :  :  :  :  :  :  :  :  :  :  :  :\             |  : |
            /. `. `. `. `. `. `. `. `. `. `. `. `. `\            |`. `|
           /  :  :  :  :  :  :  :  :  :  :  :  :  :  \           |  : |
          / .' .' .' .' .' \\  '  // ' .' .' .' .' .' \          |.' .|
         / :  :  :  :  :    ))   ((   :  :  :  :  :  : \         |  : |
        |`. `. `. `. `. `. ((  .  )) . `. `. `. `. `. `.\        |`. `|
        |  :  :  :  :  :    ))   ((   :  :  :  :  :  :  :\       |  : |
       / .' .'  ' .' .' .' ((  '  )) ' .' .' .'  ' .' .' .|      /.' .|
      | :  : //  :  :  :    ))   ((   :  :  :   \\:  :  : |     |:  : |
      |. `. || `. `. `. `. ((  .  )) . `. `. `.  ||`. `. `.\    | `. `\
     /  :  :||   :  :  :    ))   ((   :  :  :  : ||  :  :  :|   |:  :  |
    | .' .' || .' .' .' .' ((  '  )) ' .' .' .'  ||.' .' .' |   | .' .'|
    |:  :  :||   :  :  :    ))   ((   :  :  :  : ||  :  :  : \  |:  :  |
   /. `. `. || `. `. `. `. ((  .  )) . `. `. `.  ||\. `. `. `.\ | `. `.|
  |  :  :  : \\  :  :  :  :  :  : _   :  :  :   //  \:  :  :  :\|:  :  |
  |.' .' .' .'\\  .' .' .' . _ ' ._\ ' .' .' . //|   \.' .' .' .| .' .'|
./:  :  :  :  :\\   :  :  : /_  :  \\ :  :   _// |    \ :  :  : |:  :  |
|. `. `. `. __ `\\_  `. `. // `. `. \\______/_/ `|     | `. `. `| `. `.|
| :/ :  :  /  |  \_\______// :  :  : \______/  : |     |:  :  : |:  :  |
.'|.' .' ./   |.' .\______/.' .' .' .' .' .' .' .|      \.' .' .' .' .'|
  |  :  :|    \  :  :  :  :  :  :  :  :  :  :  :/        \ : o   :  :  |
`/ `. `. |     |. `. `. `. `. `. `. `. `. `. `.|          | o  `. `. `.|
| :  :  /      | :  :  :  :  :  :  :  :  :  :  |          | o :  :  :  |
|' .' .|       |' .' .' .' .' .' .' .' .' .' .'|           \ o .' .' .'|
  :  : |       | :  :  :  :  :  :  :  :  :  :  :\           \ o  :  :  |
`. `. /       /`. `. `. I  `. `. `. `. ` I . `. `\           \ oo `  o.|
  :  |       /:  :  :  : I   :  :  :  :  I  :  :  \           \_ oooo  |
.' .'|      /' .' .' .'  I .' .' .' .' . I ' .' .' \            \_____/
  : /      /  :  :  :  : I   :  :  :  : I   :  :  : \
`. |      | `. `. `. `.  I `. `. `. `.  I `. `. `. `.\
  :|      |:  :  :  :  : I   :  H H   : I   :  :  :  :\
.' |     /' .' .' .' .' . I ' . H H .'  I .' .' .' .' .\
  :|    |  :  :  :  :  :   I :  H H    I :  :  :  :  :  \
`. |    |`. `. `. `. `. `. I  ` L L `. I  `. `. `. `. `. \
  :|    |  :  :  :  :  :  : I    H H  I  :  :  :_ :  :  : \
.' |    |.' .' .' .' .' .' . I ' H H I .' .' .' .\ .' .' .'\
  :|    |  :  :___  :  :  :   I  H HI :  :  :  :  \  :  :  :\
`. |    |`. `.|   |. `. `. `.  I H H . `. ` _`. `.|\. `. `. `|
  :|    /  :  |   | :  :  :  :  IH H  :  : | _ :  | \:  :  : |
.' |   | .' .'|   |' .' _  .' .' H H ' .'  ||   .'|  \.' .' .|
  :|   |:  :  |   | :  _ |   :   H H  :   //   :  |   | :  : |
`. /   | `. `./   |. `. || `. `. H H . ` //  `. `.|   |. `. `|
  |    |:  : |    | :  : \\  :   H H  :_//  :  :  |   | :  : |
.'|    / .' .|    /' .' . \\  .' H H ' _/ .' .' .'\   |' .' .|
  |   | :  : |   |  :  :   \\    H H  :  :  :  :  :|  | :  : |
`.|   |. `. `|   |`. `. `.  \\_. H H . `. `. `. `. |  |. `. `|
  |   | :  :/   |:  :  :  :  \_  H H  :  :  :  :  :|  | :  : |
.'|   |' .' |  |' .' .' .' .' .' H H ' .' .' .' .' |  |' .' .|
  |   | :  :|  | :  :  :  :  :   H H  :  :  :  :  :|  | :  : |
`.|  /`. `. |  |. `. `. `. `. `. `. `. `. `. `. `. |   \ `. `|
  / |:  :  :/  / :  : __  :  :  :  :  :  :  :__   :\    |  :  \
.|  | .' .'|  |.' .' |XX\' .' .' .' .' .' .'/XX|.' .\   |.' .'|
.|  /:  :  |  |  :  :  \X\:  :  :  :  :  : /X/ :  :  |  |  :  |
`| |. `. `.|  |`. `. `. \X\_. `. `. `. `._/X/`. `. `.|  |`. `.|
.| | :  :  |  |  :  :  : \XX\_  :  :  :_/XX/:  :  :  |  |  :  |
.| |' .' ./   /.' .' .' .' \XX\' .' .'/XX/.' .' .' .'|  |.' .'|
.| | :  :|   |:  :  :  :  :  \X\_____/X/ :  :  :  :  |  |  :  |
`| |. `. |   | `. `. `. `. `. |XXXXXXX|`. `. `. `. `.|  \`. `.|
.| | :  :|   |:  :  :  :  :  :|XX| |XX|  :  :  :  :  |   | :  |
.| |' .' |   | .' .' .' .' .'  \ |V| / .' .' .' .' .'|   |' .'|
.| / :  :|   |:  :  :  :  :  : /\/ \/\:  :  :  :  :  |   | :  |
=||`. `./    | `. `. `. `. `. |       |`. `. `. `. `.|   |. `.|
~\|  : |     |:  :  :  :  :  :|       |  :  :  :  :  |   | :  |
~~|.' .|      \.' .' .' .' .' |       |.' .' .' .' .'|    \ .' \
~~|  : |       | :  :  :  :  :|       |  :  :  :  :  |     \  : \
~~|`. `|       |. `. `. `. `. |       |`. `. `. `. `.|      \. `.|
_~|  : |        \:  :  :  :  :|       |  :  :  :  : /        |:  |
.||.' .|         |.' .' .' .' |       |.' .' .' .' |         |===|
.||  : \         |  :  :  :  :|       |  :  :  :  :|         /~~~\
  V\. `.|        |`. `. `. `. |       |`. `. `. `. |        |~~~~~\
    |===|        |  :  :  :  :|       |  :  :  :  :|        \~~~~~~)
    /~~~\        |.' .' .' .' |       |.' .' .' .' |         )~~~~/
   /~~~~~|        \ :  :  :  :|       |  :  :  :  /         /~_~~~\
  (~~~~~~/         | `. `. `. |       |`. `. `. `|         |~| |~~~)
   \~~~~(          |:  :  :  :|       |  :  :  : |         |~| |~~/
   /~~~_~\         | .' .' .' |       |.' .' .' .|          V  |~|
  (~~~| |~|        |:  :  :  :|       |  :  :  : |             |~|
   \~~| |~|        | #. `. `. `\     / `. `. `.#`|             |~|
    |~|  V        / : #:  : #:  )   ( : #:  : #:  \            |~|
    |~|           )' .# .' #' .(     ) .'#.' .# .'(             V
    |~|           \ : ###### :  )   ( :  ######:  /
    |~|           /. `# `. #. `(     ) `.#`. `# `.\
     V            ) : #:  :# :  )   ( :  #  : #:  (
                  \' .# .' .# ./     \ .# .' .# .'/
                   |:# :  :  :|       |  :  :  # |
                   | `. `. `. |       |`. `. `. `|
                   |:  :  :  :|       |  :  :  : |
                   | .' .' .' \       /.' .' .' .|
                   |:  :  :  : |     |:  :  :  : |
                  /. `/   |`. `|     | `. |   \ `.\
                 |  :/     \ : |     |:  /     \  :|
                 |.'[  ( )  ] .|     | .[  ( )  ]' |
                 |  [       ]: |     |: [       ] :|
                 |`. \     /. `|     | `.\     /`. |
                  \ : \   |  : |     |:  :|   /:  /
                   \ .' .' .' /       \.' .' .' ./
                    |  :  :  |         | :  :  :|
                    |`. `. `.|         |. `. `. |
                    |  :  :  |         | :  :  :|
                     \' .' .'|         |' .' .'/
                      \:  :  |         | :  : /
                       |`. `.|         |. `. |
                       |  :  |         | :  :|
                       |.' .'|         |' .' |
                       |  :  |         | :  :|
                       |`. `.|         |. `. |
                      /:  :  |         | :  : \
                     |' .' .'|         |' .' .'|
                     | :  :  |         | :  :  |
                     |. `. `.|         |. `. `.|
                     | :  :  |         | :  :  |
                     |' .' .'|         |' .' .'|
                    /  :  :  |         | :  :  :\
                   / `. `. `./         \. `. `. `\
                  / :  :  : |           |:  :  :  \
                 /.' .' _/ .|           | .\_.' .' \M
                /:  : _/  :/             \  :\_:  : \E
               /. `._/ |`./               \. | \_. `.\P
              /  :_/  /: /                 \: \  \_  :\H
             / ._/   /' /                   \ '\   \_ .\
            /__/    |  /                     \  |    \__\
                    |_/                       \_|

@@ -744,10 +775,10 @@ sub dicom_to_minc {
############################################################
#### use some other converter if specified in the config ###
############################################################
if ($converter ne 'dcm2mnc') {
if ($converter !~ /dcm2mnc/) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change allows project to modify their prod files to use a different version of dcm2mnc and still use the same command format. For example dcm2mnc4ng does match the proper command string now.

@cmadjar
Copy link
Collaborator

cmadjar commented Jun 30, 2016

Wow! Someone is having fun!!!

Regarding the pull request. I did not look at it closely but I thought you would like to know.

A unique minc is defined by the combination of the SeriesUID field and the EchoTime field. This is particularly important when dealing with acquisitions having multiple echoes (like the field maps, the MP2RAGE for example). Therefore, you might want to modify your pull request to include the EchoTime field as well.

Feel free to contact me if you have questions. I'll also be back next week.

Cheers!!

Cécile

@driusan
Copy link
Collaborator

driusan commented Jun 30, 2016

I'm not sure I follow @cmadjar.. if they have multiple echos in a single acquisition, how can the echo be part of the acquisition identification?

@cmadjar
Copy link
Collaborator

cmadjar commented Jul 7, 2016

Here is an example that could help clarifying. A field map acquisition on Siemens scanners typically results in three images:

  • two magnitude images with the same SeriesUID but different echo time
  • a phase difference with a different SeriesUID

In the case of the magnitude files, if you only check the SeriesUID, one of the image will never be inserted because the first one with the same SeriesUID was already inserted. This is why you need the EchoTime in addition to the SeriesUID in order to check for file uniqueness.

screen shot 2016-07-07 at 9 47 41 am

This is also true for other modalities using multi-echos. We have now a new modality (quantitative T2* map) in PREVENT-AD using 12 different echo times but the SeriesUID remains the same for the 12 images.

We can talk more during the next meeting.

use NeuroDB::MRIProcessingUtility;

my $profile = 'prod'; # this should never be set unless you are in a
# stable production environment
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

based on the comment, shouldn't this not be set by default?

@gluneau gluneau removed their assignment Jul 18, 2016
my $dbh = &NeuroDB::DBI::connect_to_db(@Settings::db);


sub selORdel {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a little confusing to have a function with the same name as a variable, when they're two different things (I'm a little surprised perl allows it at all)

@driusan
Copy link
Collaborator

driusan commented Jul 20, 2016

@cmadjar @MounaSafiHarab Can one of you double check that the changes to uploadNeuroDB/NeuroDB/MRIProcessingUtility.pm aren't going to negatively affect anything else?

Other than that, this looks fine to me.

"name of config file in ../dicom-archive/.loris_mri"
],
["-seriesuid", "string", 1, \$seriesuid, "Only deletes this SeriesUID"
],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gluneau
can you please add description of the confirm and check if seriesUID is not set?

@MounaSafiHarab
Copy link
Contributor

MounaSafiHarab commented Aug 4, 2016

@gluneau

  • Check mri_acquisition_dates table (if only file exists and is being removed, the table entry needs to be removed)
  • Test if processed data exist?

@cmadjar cmadjar self-assigned this Aug 4, 2016
usage: $0 [options] select|confirm
$0 -help to list options

USAGE
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gluneau
nice help section!
It would be nice to explain what the select and confirm options are supposed to do. It is not very clear.

@MounaSafiHarab
Copy link
Contributor

@gluneau
Decision is to move this to 17.0 to have @cmadjar test it more thoroughly.

@cmadjar
Copy link
Collaborator

cmadjar commented Oct 14, 2016

I used this command using the tarchiveLoader and it reinserted everything. Shouldn't it have inserted only the scan corresponding to that seriesUID?

tarchiveLoader -globLocation -profile prod tarchive_file.tar -seriesuid 1.3.12.2.1107.5.2.32.35442.2016071310164138453406379.0.0.0

@gluneau
Copy link
Contributor Author

gluneau commented Oct 14, 2016

@cmadjar, I found that same bug at the same time!

I think at some point I was overzealous and I removed the parenthesis, but the are essential and not optional!

line 123 of MRIProcessingUtility.pm

my ($seriesuid) = @_;

@cmadjar
Copy link
Collaborator

cmadjar commented Oct 14, 2016

let me know when you have taken care of it and I'll test the insertion once more :)

@cmadjar
Copy link
Collaborator

cmadjar commented Oct 17, 2016

Great job! Thank you :)

@samirdas
Copy link
Contributor

samirdas commented Oct 21, 2016

@gluneau
to do more changes then @cmadjar to retest

@MounaSafiHarab
Copy link
Contributor

MounaSafiHarab commented Oct 27, 2016

@gluneau

A thought that just crossed my mind:
when you remove a minc, shouldn't you update the Number_of_MincCreated/Number_of_MincInserted in the mri_upload table? (and perhaps when re-adding the minc as well)?
those display in the imaging_uploader front-end table...

print "\nDelete from DB";
# Delete from DB
selORdel("parameter_file","Value");
selORdel("files_qcstatus","QCStatus");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

look at the delete of lines 238 and 239;
before deleting from these two tables; you should have a user input to confirm this is to be deleted
(can map the information and then when re-inserting the new file, can use SeriesUID and EchoTime to re-insert the QC information for the newly inserted FileID)

@MounaSafiHarab
Copy link
Contributor

@cmadjar @driusan @samirdas

so Greg added the flag; is this good to go for you @cmadjar? is this what you wanted to see?
the decision in the imaging meeting was for me to merge but I want to make sure you are Okay with it first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants