Skip to content

GUI Manual

Jump to bottom
Iosif Spartalis edited this page Sep 30, 2021 · 55 revisions

GUI User Guide

The tool has a tab for each data type (tabular, DICOM) and the usage of the GUI is a straightforward procedure.

CSV Tab

tabular tab

Data Validation

First we have to load the dataset csv file.

If we need to validate the data, we need to have a json file containing the table schema (metadata) of the dataset csv file. This json must may follows either a modified frictionless data table-schema specifications or MIP's Data-Catalogue specification schema.

About frictionless schema, the modified schema of that json file can be found here. In short, it is a simple modification that adds MIPType property in the field json object. The acceptable values for this property are text, numerical, integer, nominal and date, depending on the original field object type property value.

Also, in the case where the dataset belongs to a certain MIP's pathology, there is the option to download the dataset's CDE schema from Data Catalogue. In that case, the user can save the schema file in a local drive.

For the report, there are two options file formats:

  • pdf
  • excel (xlsx)

outlier threshold input field is related with the outlier detection for numerical variables of the incoming dataset. The way that the Data Quality Control tool handles the outlier detection of a certain numerical variable, is that first calculates the mean and the standard deviation based on the valid values of that column and then calculates the upper and the lower limit by the formula: upper_limit = mean + outlier threshold * standard deviation, lower_limit = mean - outlier threshold * standard deviation. If any value is outside those limits then it is considered as an outlier.

The report file will be saved in the given output folder, by clicking Create Report button. The name of the output file will be:

  • <dataset>_report.pdf

Data Cleaning

After reviewing the Data Validation report created in the previous step (Please refer to the Data Validation Report wiki section for further details), we can proceed with the data cleaning operation. We repeat the previous steps and select the Perform Data Cleaning check button. The cleaned dataset file will be saved in the report folder using the original dataset name with the addition of the suffix '_corrected'.

Infer Tab

Inference tab

In this tab we can infer a dataset's schema and save it to the local disk. The schema could be saved in two formats:

  1. Frictionless spec json
  2. Data Catalogue's spec Excel (xlsx) file, that can be used for creating a new CDE pathology version.

In the infer option section, we give the number of rows that the tool will based on for the schema inference. Also, we declare the maximum number of categories that a nominal MIPType variable can have.

If we choose the Data Catalogue's excel as an output, the tool offers the option of suggesting CDE variables for each column of the incoming dataset. This option is possible, only when a CDE dictionary is provided. This dictionary is an excel file that contains information for all the CDE variables that are included or will be included in the MIP (this dictionary will be available in the Data Catalogue in the near future). The tool calculates a similarity measure for each column based on the column name similarity (80%) and the value range similarity (20%). The similarity measure takes values between 0 and 1. In the field similarity threshold we can define the minimum similarity measure between an incoming column and a CDE variable that need to be met in order the tool to suggest that CDE variable as a possible correspondence. The tool stores those CDE suggestions in the excel file in the column named CDE and also stores the corresponding concept path under the column conceptPath.

DICOM Tab

image dicom GUI

Data Validation

We select the Dicom Root Folder where all the DICOM are stored. It is assumed that for each patient there is a subfolder containing all the MRI dcm files, note that a patient could have more than one MRI. Then, we select the the Output Report Folder where the report files will be placed. If the folder does not exist, the tool will create it. Then, we press the Create Report button.

The tool creates in the <report folder>, a pdf report file (dicom_report.pdf) and, depending of the results, also creates the following csv files :

  • validsequences.csv
  • invalidsequences.csv
  • invaliddicoms.csv
  • notprocessed.csv
  • mri_visits.csv

The above files are created even if no valid/invalid sequences/dicoms files have been found. In such case, the files will be empty.

validsequences.csv

If there are valid sequences then the tool will create this csv file. A sequence is 'valid' if it meets the minimum requirements found here. This file contains all the valid MRI sequences that found in given DICOM folder with the following headers discribing each sequence:

PatientID, StudyID, SeriesNumber, SeriesDescription, SeriesDate

The value of the sequence tags SeriesDescription and SeriesDate are dirived from the headers in the dicom files - more specifically, the value of a sequence tag is the most frequent value of this particular tag found in the sequence's dicom files.

invalidsequences.csv

If there are invalid sequences the tool will create this csv file with the following headers:

PatientID, StudyID, SeriesNumber, Slices, Invalid_dicoms, SeriesDescription, Error1, Error2, Error3, Error4, Error5, Error6

  • Slices is the number of dicom files that the current sequence is consist of (sum of valid and invalid dicoms).
  • Invilid_dicoms is the number of invalid dicom files the current sequence.
  • Error1 - Error6 is an error description that explains the reason why the sequence is characterized as 'invalid'

invaliddicoms.csv

If a dicom file does not have at least one of the mandatory tags as described in the MIP specification found here, then it will be characterized as 'invald'. If there are invalid dicoms in the DICOM dataset, the tool will create this csv file with the following headers:

Folder, File, PatientID, StudyID, SeriesNumber, InstanceNumber, MissingTags

  • MissingTags is a list of the missing mandatory DICOM tags.

notprocessed.csv

If in the given root folder are some files that the QC tool can not process (not dicom files, corrupted dicom files etc), the tool will create this csv file with the following headers describing the location of those files:

Folder, File

mri_visits.csv

This file contains MRI visit information for each patient. This file is necessary for the HBP MIP DataFactory's Step3_B and it has the following headers:

PATIENT_ID, VISIT_ID, VISIT_DATE

Data Cleaning

If we want to filter out the invalid MRI sequences and reorganize the dcm files of the valid MRIs in a suitable folder structure for importing them into LORIS-for-MIP, repeat the previous step and select the Reorganize files for Loris pipeline check button.

For the LORIS pipeline the dcm files are reorganized and stored in a folder structure <ouput_report_folder>/<patientid>/<patientid_visitcount>. All the dcm sequence files that belong to the same scanning session (visit) are stored in the common folder <patientid_visitcount>.

Clone this wiki locally