docs/source/_example_MEGMA/example_00_Introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b7414202-021e-4763-990a-fd27acdbb913",
   "metadata": {},
   "source": [
    "\n",
    "## 1. Introduction\n",
    "\n",
    "\n",
    "* [1.1 MEGMA introduction](#1.1-MEGMA-introduction)\n",
    "* [1.2 Metagenomic cross nation datasets and tasks](#1.2-Metagenomic-cross-nation-datasets-and-tasks)\n",
    "* [1.3 MEGMA fitting and AggMapNet training strategy](#1.3-MEGMA-fitting-and-AggMapNet-training-strategy)\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5090cd5a-dfe8-47a1-be28-2e409f40cec2",
   "metadata": {},
   "source": [
    "### 1.1 MEGMA introduction\n",
    "**MEGMA** is short for metagenomic **M**icrobial **E**mbedding, **G**rouping, and **M**apping **A**lgorithm (MEGMA) , which is a further step development of **AggMap** [(v.1.1.7)](https://pypi.org/project/aggmap/1.1.7/) that specific for metagenomic data learning. **MEGMA** is developed to transform the tabular metagenomic data into spatially-correlated color image-like 2D-representations, named as the 2D-microbiomeprints (3D tensor data in the form of row, column and channel, or feature maps, **Fmaps**). 2D-microbiomeprints are multichannel feature maps (Fmaps) and are the inputs of ConvNet-based AggMapNet models. \n",
    "\n",
    "![MEGMA](https://raw.githubusercontent.com/shenwanxiang/aggmap-megma/master/cover/f2.png)\n",
    "\n",
    "**MEGMA** is released in the **aggmap** package, in this example, we will show how to employ the **aggmap** package to perform **MEGMA** on the cross-nation datasets of metagenomics."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "667d37b2-1ab9-4686-8575-f4022e2ed5b7",
   "metadata": {},
   "source": [
    "### 1.2 Metagenomic cross nation datasets and tasks\n",
    "\n",
    "This metagenomic cross-nation data covers 849 gut microbial species from fecal samples of total 575 subjects (Colorectal cancers, CRCs or healthy controls, CTRs) compiled by [Wirbel et al.(Nat. Med. 25, 679-689, 2019)](https://www.nature.com/articles/s41591-019-0406-6). It contains five metagenomic datasets from five separate studies in five nations (**FRA**: France, **AUS**: Australia, **DEU**: Germany, **CHN**: China, and **USA**: America). \n",
    "             \n",
    "<div align=\"center\">\n",
    "   <span style=\"font-weight:bold;\">Table 1. Metagenomic cross-nation data.</span>\n",
    "</div>\n",
    "\n",
    "\n",
    "\n",
    "|Dataset|# Case (CRC)|# Control (CTR)|# Species|Download|\n",
    "|:----|:----|:----|:----|:----|\n",
    "|**AUS**|46|63|849|[label](https://raw.githubusercontent.com/shenwanxiang/bidd-aggmap/master/docs/source/_example_MEGMA/dataset/AUS_dfy.csv), [data](https://raw.githubusercontent.com/shenwanxiang/bidd-aggmap/master/docs/source/_example_MEGMA/dataset/AUS_dfx.csv)|\n",
    "|**CHN**|74|54|849|[label](https://raw.githubusercontent.com/shenwanxiang/bidd-aggmap/master/docs/source/_example_MEGMA/dataset/CHN_dfy.csv), [data](https://raw.githubusercontent.com/shenwanxiang/bidd-aggmap/master/docs/source/_example_MEGMA/dataset/CHN_dfx.csv)|\n",
    "|**DEU**|60|60|849|[label](https://raw.githubusercontent.com/shenwanxiang/bidd-aggmap/master/docs/source/_example_MEGMA/dataset/DEU_dfy.csv), [data](https://raw.githubusercontent.com/shenwanxiang/bidd-aggmap/master/docs/source/_example_MEGMA/dataset/DEU_dfx.csv)|\n",
    "|**FRA**|53|61|849|[label](https://raw.githubusercontent.com/shenwanxiang/bidd-aggmap/master/docs/source/_example_MEGMA/dataset/FRA_dfy.csv), [data](https://raw.githubusercontent.com/shenwanxiang/bidd-aggmap/master/docs/source/_example_MEGMA/dataset/FRA_dfx.csv)|\n",
    "|**USA**|52|52|849|[label](https://raw.githubusercontent.com/shenwanxiang/bidd-aggmap/master/docs/source/_example_MEGMA/dataset/USA_dfy.csv), [data](https://raw.githubusercontent.com/shenwanxiang/bidd-aggmap/master/docs/source/_example_MEGMA/dataset/USA_dfx.csv)|\n",
    "\n",
    "\n",
    "The tasks are to build classfication models to classify CRCS from CTRs, and to identify the important microbial species that contribute to CRCs. Each microbe is a feature point in the classfication model, therefore, we need to calculate the feature importance to get the importance score of the microbes. Since we have five independent datasets that from five countries, we can build the model on one country data and test the model performance on the rest of the countries, we called this a study-to-study transfer (**STST**, model trained on one nation data and tested on the rest nations) experiment. The **STST** can tell us the real performance of the metagenomc machine learning models in the application of the external data. Therefore, in this example, the study-to-study transfer will be used for the evaluation of the performance of the metagenomic prediction models and the consistency of the across-nation biomarker discovery. Note that since **MEGMA** is an unsupervised learning method, we can use all unlabeled metagenomic data to fit `megma` (AKA, to train `megma`). Using all unlabeled data makes it more accurate and robust when calculating pairwise distances between microbes. Because the feature points in a country's metagenomic data may have very low variance, the correlation between microbes cannot be accurately calculated."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6606602c-127a-43fd-9a70-74892e5d15f1",
   "metadata": {},
   "source": [
    "### 1.3 MEGMA fitting and AggMapNet training strategy\n",
    "\n",
    "\n",
    "Because MEGMA is unsupervised learning method, therefore, in this stduy, we can use all of the metagenomic unlabelled abudance data to pre-fit MEGMA, we named this MEGMA as `megma_overall`. As shown in the figure below, the `megma_overall` can be fitted on the abundance data of the 5 countries: AUS, CHN, FRA, DEU, and USA. After that, the `megma_overall` can be used to transform the Fmaps for each country, and then we can build the AggMapNet based on the generated Fmaps of each country.\n",
    "\n",
    "\n",
    "Except for that, we can also try to use unlabelled abudance data of one country only to fit the `megma`, for example if we use the USA abundance data, we can named our megma as `megma_usa`. We called this as country-specific megma, because megma is fitted on each country only. In the STST test stage, the Fmaps for the rest countries will be transformed from this megma. The country-specific `megma` maybe not as robust as the overall `megma`, because the samples to fit the country-specific megma is lower than the samples to fit the overall megma.\n",
    "\n",
    "\n",
    "![MEGMA_Agm](https://raw.githubusercontent.com/shenwanxiang/bidd-aggmap/master/docs/source/_example_MEGMA/images/MEGMA-AggMapNet.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}