Preprocessing and Quality Control of Targeted NMR Data (Bruker IVDr) with the nPYc-toolbox.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing and Quality Control of Targeted NMR Data (Bruker IVDr) with the nPYc-toolbox"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial demonstrates how to use the NMR targeted data processing modules of the nPYc-Toolbox, to \n",
    "import and perform some basic quality control of outputs from the Bruker IVDr targeted quantification methods and generate a final high quality dataset ready for data modelling.\n",
    "\n",
    "This tutorial should be performed after **Preprocessing and Quality Control of NMR Data with nPYc-Toolbox**, as we will exclude quantification outputs from spectra which failed basic quality control checks. \n",
    "\n",
    "Details of how to install all of the required dependencies and to set up your computing environment can be found here [Installing the nPYc-Toolbox](https://npyc-toolbox.readthedocs.io/en/latest/tutorial.html#Installing-the-nPYc-Toolbox).\n",
    "\n",
    "The dataset used in this example (DEVSET U 1D NMR) is comprised of six samples of pooled human urine, aliquoted, and independently prepared and measured by 1H NMR spectroscopy. Each source sample was separately prepared and assayed thirteen times. A pooled QC sample (study reference, SR) and independent external reference (long-term reference, LTR) of a comparable matrix was also acquired to assist in assessing analytical precision. See [Tutorial Datasets](https://npyc-toolbox.readthedocs.io/en/latest/tutorial.html#tutorial-datasets) and the the Metabolights Study [MTBLS694](https://www.ebi.ac.uk/metabolights/MTBLS694) for details of the study, and [Recommended Study Design Elements](https://npyc-toolbox.readthedocs.io/en/latest/studydesign.html) for details of the various QC samples acquired.\n",
    "\n",
    "The [nPYc-toolbox-tutorials](https://github.com/phenomecentre/nPYc-toolbox-tutorials) contain all of the data required to run the tutorial Jupyter notebooks, full details of which are given [here](https://npyc-toolbox.readthedocs.io/en/latest/tutorial.html#preprocessing-and-quality-control-of-nmr-targeted-data-with-the-npyc-toolbox)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Import the nPYc-Toolbox and Configure the Jupyter Notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Import the nPYc-Toolbox\n",
    "import nPYc\n",
    "\n",
    "# Import enumerations for sample type\n",
    "from nPYc.enumerations import VariableType, DatasetLevel, AssayRole, SampleType\n",
    "\n",
    "# Import normalisation objects for data normalisation\n",
    "from nPYc.utilities.normalisation import NullNormaliser, TotalAreaNormaliser, ProbabilisticQuotientNormaliser\n",
    "\n",
    "# Import matplotlib plotting, configure the Jupyter notebook to plot inline\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# Set up plotly to work in offline mode with the notebook\n",
    "from plotly.offline import init_notebook_mode, iplot\n",
    "init_notebook_mode(connected=True)\n",
    "\n",
    "# Set up the pandas numerical display precision\n",
    "import pandas\n",
    "pandas.set_option('display.float_format', '{:.2f}'.format)\n",
    "\n",
    "# Set up to hide warnings (particularly Depreciation, RunTime warnings, these are not for the user to worry about!)\n",
    "# These lines can be commented to show warnings if of interest to advanced users\n",
    "import warnings\n",
    "warnings.filterwarnings(action='ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Import and Preprocess targeted NMR Data (Bruker IVDr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The NMR DevSet Dataset is located in 'DEVSET U 1D NMR raw data files'. This folder contains 94 directories. \n",
    "Each directory contains a .xml file which was obtained by extracting metabolite concentrations from the ‘noesygppr1d’ pulse sequence spectrum with the Bruker IVDr \"BI-QUANT-UR\" software. \n",
    "\n",
    "The second step is to load these .xml files into a nPYc-Toolbox [Dataset](https://npyc-toolbox.readthedocs.io/en/latest/objects.html) object.\n",
    "\n",
    "The \"dataPath\" parameter sets the location of the NMR raw data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "dataPath = r'./DEVSET U 1D NMR raw data files'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The “sop” parameter points to a configuration file (encoded in JSON format) which contains a set of pre-set parameters to use during data import see [Configuration Files](https://npyc-toolbox.readthedocs.io/en/latest/configuration/configuration.html) for full details.\n",
    "\n",
    "The nPYc-Toolbox contains three configuration files for Bruker IVDr methods, 'BrukerQuant-UR' (urine small molecules), 'BrukerBI-QUANT-PS' (serum/plasma small molecules) and 'BrukerBI-LISA' (serum/plasma) lipoprotein.\n",
    "For this urine biofluid dataset, we will load data from the 'BI-QUANT-UR' IVDr method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "sop = 'BrukerQuant-UR'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The .xml files from the BI-QUANT-UR method contain values in absolute and relative (normalised to creatinine) units.\n",
    "The 'unit' parameter can be used to select either the values 'mmol/L' or in 'mmol/mol Crea'. We will extract only the absolute units in this tutorial.\n",
    "\n",
    "The following line creates an object representing the targeted dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "IVDrTargetedData = nPYc.TargetedDataset(dataPath, fileType='Bruker Quantification', sop=sop, unit='mmol/L')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Each nPYc-Toolbox Dataset object contains a name that can be changed as shown in the next cell. \n",
    "This name will be used in the summary and visualisation reports and in the file names of the exported data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "IVDrTargetedData.name = 'nPYc Targeted NMR Tutorial dataset'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Import Sample Metadata and Match to Acquired Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default way to add sample metadata is to prepare a CSV file which follows the set of conventions as described in [CSV Template for Metadata Import](https://npyc-toolbox.readthedocs.io/en/latest/samplemetadata.html#csv-template-for-metadata-import) and match it with the acquired data using the 'addSampleInfo' method.\n",
    "\n",
    "Based on the corresponding entries in the sample metadata CSV file, the acquired samples are categorised into different types, where 'Study Samples' comprise the main core of the study, and the others are acquired for specific roles in characterising data quality. The main QC samples here are the 'Study Reference' samples, which comprise a pool of study samples and are used to assess analytical stability across the run. For interest we have also included some 'Long Term Reference' samples (a QC sample external to the study) and a blank, for full details see [Recommended Study Design Elements](https://npyc-toolbox.readthedocs.io/en/latest/studydesign.html).\n",
    "\n",
    "Although optional, this is recommended in order to make optimal use of the quality control features and visualisations provided by the nPYc-Toolbox.\n",
    "\n",
    "An example CSV file is provided, as given in 'DEVSET U 1D NMR IVDr Basic CSV.csv:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "IVDrTargetedData.addSampleInfo(descriptionFormat='Basic CSV', filePath=r'DEVSET U 1D NMR IVDr Basic CSV.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As described in [Datasets](https://npyc-toolbox.readthedocs.io/en/latest/objects.html), the spectral data, sample metadata and feature metadata can be inspected directly using:\n",
    "\n",
    "```\n",
    "dataset.intensityData\n",
    "dataset.sampleMetadata\n",
    "dataset.featureMetadata\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Generate Quality Control Reports"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The nPYc-Toolbox offers a series of reports, pre-set visualisations comprised of text, figures and tables to describe and summarise the characteristics of the dataset, and help the user assess the overall impact of quality control decisions (i.e. excluding samples or features and changing filtering criteria). \n",
    "\n",
    "For full details see [Reports](https://npyc-toolbox.readthedocs.io/en/latest/reports.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sample Summary Report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first report can be used to check the expected samples against those acquired, in terms of numbers, sample type, and any samples either missing from acquisition or not recorded in the sample metadata CSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "nPYc.reports.generateReport(IVDrTargetedData, 'sample summary')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, for this dataset there are two samples with entries in the sample metadata CSV file, but missing from acquisition. This allows the user to quickly assess the completeness of the dataset and, for example, investigate why these samples were missing.\n",
    "\n",
    "From the 'sample summary' report, it can be seen that corresponding information is missing from the sample metadata CSV file for one sample. This sample is listed has having unknown type, and missing information. As above, this allows the user to quickly determine whether information should be added to the sample metadata CSV file for this sample, or whether the spectrum should be excluded from the final dataset (see '5. Exclude Samples and/or Features if Required' for details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature Summary Report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The feature summary report provides visualisations summarising the quality and distribution of values across samples for each individual feature.\n",
    "\n",
    "In order, for an NMR targeted Dataset these consist of:\n",
    "- Tables containing summary information about each of the quantified features (Name, and if available, reference ranges). These are further broken down by the type of quantification (not applicable for targeted NMR datasets).\n",
    "- The residual standard deviation (RSD) observed for each feature per Sample Type/Assay Role (Figure 2)\n",
    "- Violinplots showing the distribution across the different sample types for each chemical compound (Figure 3)\n",
    "- A table with the RSD values for each feature, calculated separately for the Study Pool, External Reference and Study Sample types (Table 2)\n",
    "\n",
    "For each parameter, acceptable default values are pre-defined in the configuration SOP, see [Built-in Configuration SOPs](https://npyc-toolbox.readthedocs.io/en/latest/configuration/builtinSOPs.html). If different values are required, these can be set by the user in the SOP directly, or by updating the 'Attribute', either at import (as above), or by subsequent direct modification in the pipeline (see [Datasets](https://npyc-toolbox.readthedocs.io/en/latest/objects.html) for more details).\n",
    "\n",
    "Any samples failing any of the above criteria are flagged at the end of the report. \n",
    "\n",
    "For full details of each of the above, see [Feature Summary Report: NMR Targeted Datasets](https://npyc-toolbox.readthedocs.io/en/latest/reports.html#feature-summary-report-nmr-targeted-datasets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "nPYc.reports.generateReport(IVDrTargetedData, 'feature summary')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a TargetedNMR dataset, sample exclusions are performed based on the assessment of the raw full high resolution NMR spectra. Further down, we will filter\n",
    "one of the samples highlighted in the 'Preprocessing and quality control of NMR data with the nPYc-Toolbox' tutorial as failing the solvent suppression check.\n",
    "\n",
    "The violinplots can be used to exclude features with a very high proportion of zeros. We will show how to exclude these on the 'Excluding Specific Samples and/or Features' section. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Exporting Reports"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default all reports are output directly to the notebook (as above), however, if html copies are required these can be automatically saved to the save directory by adding the optional input argument \"destinationPath\".\n",
    "\n",
    "For example, to save to the path defined in \"saveDir\":  \n",
    "\n",
    "```\n",
    "saveDir = '/path to save outputs'\n",
    "nPYc.reports.generateReport(IVDrTargetedData, 'feature summary', destinationPath=saveDir)\n",
    "\n",
    "``` "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Exclude Samples and/or Features if Required"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataset objects contain two internal 'mask' vectors, the 'sampleMask' and 'featureMask', which store whether a sample or feature respectively should be used when calculating QC metrics, visualised in the reports and finally exported, see [Sample and Feature Masks](https://npyc-toolbox.readthedocs.io/en/latest/objects.html#sample-and-feature-masks).\n",
    "\n",
    "There are several functions which modify these masks, which are useful at various stages of quality control and in preparing a final dataset for export."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mask Samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The 'updateMasks' function can be used to automatically mask samples (and/or features).\n",
    "\n",
    "For now, we do not mask any features, setting \"filterFeatures=False\".\n",
    "\n",
    "Here, by setting preferences with the 'sampleTypes' argument, any samples which are not required can be masked. In this example, we limit our dataset to study samples ('SampleType.StudySample, AssayRole.Assay') and study reference samples ('SampleType.StudyPool, AssayRole.PrecisionReference') by running the following (see [Sample and Feature Masks](https://npyc-toolbox.readthedocs.io/en/latest/masks.html) and [Enumerations](https://npyc-toolbox.readthedocs.io/en/latest/enumerations.html) for more details)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "IVDrTargetedData.updateMasks(sampleTypes=[SampleType.StudySample, SampleType.StudyPool], assayRoles=[AssayRole.Assay, AssayRole.PrecisionReference], filterFeatures=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results of masking can be summarised using the 'sample summary' report."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "nPYc.reports.generateReport(IVDrTargetedData, 'sample summary')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Excluding Specific Samples and/or Features "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The 'updateMasks' function works to mask samples or features not meeting specific criteria, in addition to this, the nPYc-Toolbox also contains two additional methods to mask specific samples or features directly, 'excludeSamples' and 'excludeFeatures' respectively, see [Sample and Feature Masks](https://npyc-toolbox.readthedocs.io/en/latest/masks.html).\n",
    "\n",
    "Each of these functions takes three input arguments; firstly, a list of sample or feature identifiers; secondly, the name of the column in 'sampleMetadata' (for 'excludeSamples') or 'featureMetadata' (for 'excludeFeatures') where these identifiers can be found; and finally an optional message as to why these samples or features have been flagged for exclusion.\n",
    "\n",
    "The IVDr algorithms will attempt generate a .xml file for all NMR raw data directories. However, if the high resolution NMR spectrum which was used to derive is not of high quality, we recommend excluding the quantification results from the TargetedDataset.\n",
    "\n",
    "In the next cell, we use the 'excludeSamples' syntax to exclude a sample that failed the quality control in the **Preprocessing and Quality Control of NMR Data with the nPYc-Toolbox** tutorial, and another\n",
    "sample for which no metadata was provided (which was highlighted in the 'Sample Summary' report."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "IVDrTargetedData.excludeSamples(['PipelineTest_Urine_300K_RFT_290118/580'], on='Sample File Name', message='Incomplete solvent suppression')\n",
    "IVDrTargetedData.excludeSamples(['PipelineTest_Urine_300K_RFT_290118/930'], on='Sample File Name', message='No metadata provided')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Finally, we can use the 'excludeFeatures' command to exclude all features containing only 0 values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%% \n"
    }
   },
   "outputs": [],
   "source": [
    "zero_value_features = IVDrTargetedData.featureMetadata[IVDrTargetedData.intensityData.sum(axis=0) == 0]['Feature Name']\n",
    "IVDrTargetedData.excludeFeatures(zero_value_features, on='Feature Name', message='Features with only zero values')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Permanently Exclude Masked Samples/Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once satisfied with the sample and feature masks, exclusions can be applied (permanently removed from the dataset) using the 'applyMasks' function.\n",
    "\n",
    "This method should be used only when it is absolutely certain that the masked features and samples are to be removed, as the excluded data will otherwise have to be re-imported.\n",
    "\n",
    "Before masks have been applied, however, feature/sample masking can be changed by first re-setting the masks to include all samples/features:\n",
    "\n",
    "```\n",
    "IVDrTargetedData.initialiseMasks() \n",
    "```\n",
    "\n",
    "Then different feature/sample exclusions can be applied as required.\n",
    "\n",
    "For details see [Sample and Feature Masks](https://npyc-toolbox.readthedocs.io/en/latest/masks.html)\n",
    "\n",
    "In this case we are happy with the masking, and so features and samples can be permanently excluded using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "IVDrTargetedData.applyMasks()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# 6. Analytical Multivariate Quality Control"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The nPYc-Toolbox provides the capacity to generate a principal component analysis (PCA) model of the data (via the [pyChemometrics](https://github.com/phenomecentre/pyChemometrics) module), and subsequently, to use this to assess data quality, identify potential sample and feature outliers, and determine any potential analytical associations with the main sources of variance in the data ([Multivariate Analysis](https://npyc-toolbox.readthedocs.io/en/latest/multivariate.html))\n",
    "\n",
    "A PCA model can be generated using 'exploratoryAnalysisPCA', and there are a number of parameters which can be optimised depending on the dataset (see [PCA Model](https://npyc-toolbox.readthedocs.io/en/latest/multivariate.html#pca-model) for full details).\n",
    "\n",
    "One key parameter is 'scaling', which divides each column in the data matrix by its respective standard deviation raised to a power of the scaling parameter. This parameter can range in value between 0 and 1, and recommended values are 0 for mean-centering only, 0.5 for Pareto scaling and 1 for unit variance (UV) scaling. The outcome of PCA model will vary based on the scaling method selected, and different scaling functions can be appropriate depending on the data itself and the question being asked of the data, see [van der Berg et al. 2006](https://www.ncbi.nlm.nih.gov/pubmed/16762068)\n",
    "\n",
    "The default scaling is unit variance (\"scaling=1\"), which scales every variable to have a variance of one, and thus all variables (despite their different magnitudes) become equally important in the model.\n",
    "\n",
    "Each model is cross-validated using 7-fold cross-validation and the recommended number of principal components automatically estimated based on two criteria, when either one of these is met no more components will be added and the PCA model will be returned. There criteria are:\n",
    "1. \"minQ2\": Q2 is the variance predicted by each component (from cross-validation), when adding a component does not improve Q2 by at least this value (default \"minQ2=0.05\") then no more components will be added.\n",
    "2. \"maxComponents\": this defines the maximum number of components (default \"maxComponents=10\") returned by the model (regardless of Q2 increases).\n",
    "\n",
    "Again these parameters can be amended by adding them as input arguments to 'exploratoryAnalysisPCA'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "PCAmodel = nPYc.multivariate.exploratoryAnalysisPCA(IVDrTargetedData, scaling=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The analytical multivariate report provides visualisations summarising the largest sources of variance in the dataset (from the PCA model generated) with particular emphasis on any potential analytical sources, as defined in 'analyticalMeasurements' in the [Built-in Configuration SOPs](https://npyc-toolbox.readthedocs.io/en/latest/configuration/builtinSOPs.html).\n",
    "\n",
    "These consist of:\n",
    "- Scree plot of variance (Figure 1)\n",
    "- Scores plots coloured by sample type (Figure 2)\n",
    "- Strong sample outliers (Figure 3)\n",
    "- DmodX sample outliers (Figure 4)\n",
    "- Loadings plots (Figure 5)\n",
    "- Distribution of analytical parameters (Figure 6)\n",
    "- Heatmap of potential associations between analytical parameters and the main sources of variance (Figures 7 and 8)\n",
    "- Scores plots coloured by analytical parameters with potential association (Figures 9-11)\n",
    "\n",
    "For full details of each of the above, see [Multivariate Analysis Report](https://npyc-toolbox.readthedocs.io/en/latest/multivariate.html#multivariate-analysis-report)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "nPYc.reports.multivariateReport(IVDrTargetedData, PCAmodel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There appears to be slightly more dispersion between the SR samples in the targeted dataset than when compared to that of the full profiling data. However, no strong associations are observed in the heatmaps between analytical parameters and samples scores (main sources of variance in the data), we can conclude that the data is of high quality, and ready to be exported."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Finalise and Export Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once no further exclusions or preprocessing is required, the final dataset can be exported.\n",
    "\n",
    "The 'final report' compiles information about the samples acquired, and the overall quality of the dataset taking key figures from feature and multivariate reports to give an overall summary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nPYc.reports.generateReport(IVDrTargetedData, 'final report', pcaModel=PCAmodel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Subsequently, datasets can be exported in a variety of formats with the 'exportDataset' method (see [Exporting Data](https://npyc-toolbox.readthedocs.io/en/latest/exportingdata.html)).\n",
    "\n",
    "By default datasets are exported to the current working directory, however, if files are required to be exported to a defined path, this can be done by adding the optional input argument \"destinationPath\".\n",
    "\n",
    "For example, to save to the path defined in \"saveDir\":  \n",
    "\n",
    "```\n",
    "saveDir = '/path to save outputs'\n",
    "IVDrTargetedData.exportDataset(saveFormat='UnifiedCSV', destinationPath=saveDir)\n",
    "\n",
    "```\n",
    "\n",
    "To export a single CSV file, which contains a row for every sample, and a column for every feature, alongside all of the sample and feature specific metadata, set \"saveFormat=UnifiedCSV\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "IVDrTargetedData.exportDataset(saveFormat='UnifiedCSV', destinationPath='.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, to export the 'final report':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "IVDrTargetedData.Attributes['figureFormat'] = 'svg'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "nPYc.reports.generateReport(IVDrTargetedData, 'final report', pcaModel=PCAmodel, destinationPath='.')"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}