Hub tutorial updates

scvi_hub_intro_and_download.ipynb

@@ -47,7 +47,9 @@
    "id": "14f43c53",
    "metadata": {},
    "source": [
-    "## Overview"
+    "## Overview\n",
+    "\n",
+    "**TL;DR**: Hugging Face is a Cloud-based platform that's ideal for storage and sharing of pre-trained ML models. We provide an interface to it in scvi-tools and store pre-trained scvi-tools models on the Hugging Face Model Hub. Read on for more details or jump [here](#Download-models) for practical examples."
    ]
   },
   {
@@ -167,7 +169,7 @@
    "id": "ac80797a",
    "metadata": {},
    "source": [
-    "#### \\_scvi_required_metadata.json\n",
+    "### \\_scvi_required_metadata.json\n",
     "\n",
     "If you have been exploring the demo model, you  might have noticed a file under \"Files and versions\" named \\_scvi_required_metadata.json. As its name implies, this file contains metadata that is required for scvi-tools to be able to accomplish certain tasks such as loading your model. As we will see in the scvi-hub upload tutorial, we provide utilities to auto-generate this file, much like we do for the Model Card.\n",
     "\n",
@@ -208,7 +210,7 @@
    "source": [
     "### Code overview\n",
     "\n",
-    "Below we provide a brief overview of the main classes and functions that you will be interacting with. You can of course read more in our scvi-tools API docs \\[link to the hub section TODO after 1.0 release\\].\n",
+    "Below we provide a brief overview of the main classes and functions that you will be interacting with. You can of course read more in our scvi-tools API docs.\n",
     "\n",
     "- `scvi.hub.HubModel`. This class represents a pretrained scvi-tools model on the HF Model Hub. For example the demo-model we saw earlier would be abstracted as a HubModel. You will mainly use this class to:\n",
     "  - Push a new model to the HF Model Hub.\n",
@@ -222,40 +224,30 @@
   {
    "attachments": {},
    "cell_type": "markdown",
-   "id": "845a6dd6",
+   "id": "ac5fdf39",
    "metadata": {},
    "source": [
-    "## Imports\n",
+    "## Download models\n",
     "\n",
     "Before we move on to downloading models, let's first add the Python imports we need."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
-   "id": "012a0495",
+   "execution_count": null,
+   "id": "512390c6",
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "Global seed set to 0\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "from scvi.hub import HubModel"
    ]
   },
   {
    "attachments": {},
    "cell_type": "markdown",
-   "id": "ac5fdf39",
+   "id": "6bf06051",
    "metadata": {},
    "source": [
-    "## Download models\n",
-    "\n",
     "Now, let's use scvi-tools to download a model. We'll use the demo-model we saw previously."
    ]
   },
@@ -386,13 +378,41 @@
     "</div>"
    ]
   },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "aef14732",
+   "metadata": {},
+   "source": [
+    "## Large training data\n",
+    "\n",
+    "Depending on the size of the dataset used to train the model, there can be different scenarios, described below.\n",
+    "\n",
+    "1. The model and its full training data are stored in the HF Hub Model.\\\n",
+    "   👉 In this case:\\\n",
+    "   `hmo.model` and `hmo.load_model()` will load the model using the full training data.\\\n",
+    "   `hmo.adata` and `hmo.read_adata()` will load the full training data.\\\n",
+    "   `hmo.large_training_adata` and `hmo.read_large_training_adata()` will no-op (and return None) since there is no link to the full training data.\n",
+    "1. The model and a minification of the training data are stored in the HF Hub Model. Optionally, there may be a link to the full (i.e., non-minified) training data in the Hub Model metadata.\\\n",
+    "   👉 In this case:\\\n",
+    "   `hmo.model` and `hmo.load_model()` will load the model using the minified data.\\\n",
+    "   `hmo.adata` and `hmo.read_adata()` will load the minified data.\\\n",
+    "   `hmo.large_training_adata` and `hmo.read_large_training_adata()` will download and load into memory the full training data if there is a link to it, else it will no-op (and return None).\n",
+    "1. Only the model is stored in the HF Hub Model. There should be a link to the full training data in the Hub Model metadata.\\\n",
+    "   👉 In this case:\\\n",
+    "   `hmo.model` and `hmo.load_model()` will download the full training data and load the model using the latter.\\\n",
+    "   `hmo.adata` and `hmo.read_adata()` will no-op (and return None).\\\n",
+    "   `hmo.large_training_adata` and `hmo.read_large_training_adata()` will download and load into memory the full training data."
+   ]
+  },
   {
    "attachments": {},
    "cell_type": "markdown",
    "id": "a5e0366f",
    "metadata": {},
    "source": [
     "## Versioning\n",
+    "\n",
     "HF uses Git to seamlessly handle versioning. So if you want to download a specific version of a model, you can do so by specifying the commit hash. You can pass this into the pull function via the revision argument."
    ]
   },

scvi_hub_upload_and_large_files.ipynb

@@ -662,7 +662,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "scvi-tools-dev",
+   "display_name": "scvi-hub",
    "language": "python",
    "name": "python3"
   },
@@ -676,11 +676,11 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:36:39) [GCC 10.4.0]"
+   "version": "3.9.13 | packaged by conda-forge | (main, May 27 2022, 17:01:00) \n[Clang 13.0.1 ]"
   },
   "vscode": {
    "interpreter": {
-    "hash": "b5142939ddaa1787bd1bfcf4c0ad4d35be0fa2237c553f986d37efcb39f03f79"
+    "hash": "c7ad0fb529154325adae80858adf69263139128ad86800d0218f40cb1980a4c2"
    }
   }
  },