examples/create_mobie_project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create MoBIE Project\n",
    "\n",
    "Create an example MoBIE project with the python mobie package.\n",
    "See [the installation instructions](https://github.com/mobie/mobie-utils-python) to set up the python package.\n",
    "For more details on the MoBIE and the MoBIE project structure check out [the MoBIE README](https://github.com/mobie/mobie#data-storage).\n",
    "\n",
    "The data used in this example is part of the publication [Seipin and Nem1 establish discrete ER subdomains to initiate yeast lipid droplet biogenesis](https://doi.org/10.1083/jcb.201910177) and can be downloaded from [here](https://oc.embl.de/index.php/s/IV1709ZlcUB1k99)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# general imports\n",
    "import os\n",
    "import imageio\n",
    "import mobie\n",
    "\n",
    "# the location of the data\n",
    "# adapt these paths to your system and the input data you are using\n",
    "\n",
    "# location of the input data. \n",
    "# the example data used in this notebook is available via this link:\n",
    "# https://oc.embl.de/index.php/s/IV1709ZlcUB1k99\n",
    "example_input_data = \"/home/pape/Work/data/mobie/mobie-example-data\"\n",
    "\n",
    "# the location of the mobie project that will be created\n",
    "# we recommend that the mobie project folders have the structure <PROECJT_ROOT_FOLDER/data>\n",
    "# the folder 'data' will contain the sub-folders for individual datasets\n",
    "mobie_project_folder = \"/home/pape/Work/data/mobie/mobie_example_project/data\"\n",
    "\n",
    "# name of the dataset that will be created.\n",
    "# one project can contain multiple datasets\n",
    "dataset_name = \"example-dataset\"\n",
    "dataset_folder = os.path.join(mobie_project_folder, dataset_name)\n",
    "\n",
    "# the platform and number of jobs used for computation.\n",
    "# choose 'local' to run computations on your machine.\n",
    "# for large data, it is also possible to run computation on a cluster;\n",
    "# for this purpose 'slurm' (for slurm cluster) and 'lsf' (for lsf cluster) are currently supported\n",
    "target = \"local\"\n",
    "max_jobs = 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize the dataset\n",
    "\n",
    "First, we need to initialize the dataset. This step includes generating the top-level project folder (if it's not present already), the subfolders for the new dataset and adding the \"default\" image for this dataset.\n",
    "All these steps are performed by the function `add_image`.\n",
    "\n",
    "This function accepts input image data in different formats. The input data is specified with the arguments\n",
    "`input_path`, which specifies the file path and `input_key`, which specifies the internal path or search patterns.\n",
    "- tif images (2d or 3d) - for this option set `input_key=''`\n",
    "- folder with image files - for this option `input_key` needs to be the glob pattern for the image files, e.g `input_key='*.tif'` to load all tif files\n",
    "- hdf5 file - `input_key` needs to be the internal file path\n",
    "- n5 or zarr file - `input_key` needs to be the internal file path\n",
    "\n",
    "The input files will be copied into the project folder in the [bdv.n5 dataformat](https://github.com/bigdataviewer/bigdataviewer-core/blob/master/BDV%20N5%20format.md) and an image pyramid will be created through consecutive downsampling.\n",
    "\n",
    "To efficiently process large files the inputs should be in hdf5, n5 or zarr format.\n",
    "Note that all inputs need to be either 2d or 3d images (volumes).\n",
    "Multi-channel images (volumes) should be seperated into their channels and then each channel added individually (see `Adding image data` below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The 'default' image for our example dataset is a 2d EM slice showing an overview of the dataset.\n",
    "input_file = os.path.join(example_input_data, \"em_overview.tif\")\n",
    "\n",
    "# This is the name that will be given to the image source in mobie.\n",
    "raw_name = \"raw\"\n",
    "# The name of the menu from which the image can be added to the viewer.\n",
    "# Here, we choose \"em\", because this is an EM image slice.\n",
    "menu_name = \"em\"\n",
    "\n",
    "# We need some metadata to create the n5-file in big-data-viewer format:\n",
    "# - unit: the phyiscal unit of the coordinate system\n",
    "# - resolution: the size of one voxel in the physical unit, this needs to be a tuple/list of length 3,\n",
    "#               specifying the size for each of the 3 spatial dimensions\n",
    "# - chunks: the size of the chunks (in voxels) that are used to store the output file.\n",
    "#           good choices are usually (1, 512, 512) for 2d data and (64, 64, 64) for 3d data\n",
    "# - scale_factors: the scale factors used for downsampling the input when creating the image pyramid\n",
    "#                  this needs to be a list, where each entry specifies the scale factors for the 3 axes.\n",
    "# Note that axes are listed in the order ZYX for the resolution, chunks and scale factors\n",
    "# (in the java implementation of mobie / big-data-viewer the axis convention is XYZ).\n",
    "# Also note that the values for all three axes (ZYX) need to be specified. In the case of 2d data, the value\n",
    "# for Z should be set to 1.\n",
    "unit = \"nanometer\"\n",
    "resolution = (1., 10., 10.)\n",
    "chunks = (1, 512, 512)\n",
    "scale_factors = 4 * [[1, 2, 2]]\n",
    "\n",
    "mobie.add_image(\n",
    "    input_path=input_file, \n",
    "    input_key='',  # the input is a single tif image, so we leave input_key blank\n",
    "    root=mobie_project_folder,\n",
    "    dataset_name=dataset_name,\n",
    "    image_name=raw_name,\n",
    "    menu_name=menu_name,\n",
    "    resolution=resolution,\n",
    "    chunks=chunks,\n",
    "    scale_factors=scale_factors,\n",
    "    is_default_dataset=True,  # mark this dataset as the default dataset that will be loaded by mobie\n",
    "    target=target,\n",
    "    max_jobs=max_jobs,\n",
    "    unit=unit\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding image data\n",
    "\n",
    "After a dataset is created, we can add additional images to the dataset with the `add_image` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, we add two EM tomograms that are available in the example dataset.\n",
    "# These tomograms show small areas in higher detail and in 3d.\n",
    "\n",
    "# These are the two file names for the tomograms.\n",
    "tomo_names = [\"27_tomogram.tif\", \"29_tomogram.tif\"]\n",
    "\n",
    "# We choose chunks and scale factors for 3d data, taking\n",
    "# into account that the tomograms have a larger extent in the\n",
    "# XY plane than in Z\n",
    "unit = \"nanometer\"\n",
    "resolution = [5., 5., 5.]\n",
    "chunks = (32, 128, 128)\n",
    "scale_factors = [[1, 2, 2], [1, 2, 2],\n",
    "                 [1, 2, 2], [1, 2, 2],\n",
    "                 [2, 2, 2]]\n",
    "\n",
    "# The tomograms need to be placed at the correct position w.r.t.\n",
    "# the 2d em overview. This is achieved via an affine transformation,\n",
    "# that has been determined externally and will be applied on the fly by big-data-viewer.\n",
    "# Each affine transformation contains 12 parameters.\n",
    "# Note that the transformations here have to be specified in the 'native' axis order for bdv/mobie\n",
    "# this means they are given in XYZ order, unlike the other parameters that are given in ZYX\n",
    "transformations = [\n",
    "    [5.098000335693359, 0.0, 0.0, 54413.567834472655,\n",
    "     0.0, 5.098000335693359, 0.0, 51514.319843292236,\n",
    "     0.0, 0.0, 5.098000335693359, 0.0],\n",
    "    [5.098000335693359, 0.0, 0.0, 39024.47988128662,\n",
    "     0.0, 5.098000335693359, 0.0, 44361.50386505127,\n",
    "     0.0, 0.0, 5.098000335693359, 0.0]\n",
    "]\n",
    "\n",
    "# if you have transformations that were determined with some tool in python, they are usually given in the ZYX order.\n",
    "# you can use the following code to translate them to the correct XYZ order:\n",
    "\n",
    "# # assume 'trafo' is a transformation in zyx axis order:\n",
    "# from mobie.utils import transformation_to_xyz\n",
    "# trafo = transformation_to_xyz(trafo)\n",
    "\n",
    "# it might also be necessary to change the transformation order\n",
    "# (this is because some tools specify transformations in forward order, some in backward order)\n",
    "# in this case use\n",
    "# trafo = transformation_to_xyz(trafo, invert=True)\n",
    "\n",
    "# (in practice you probably need to figure this out with trial and error)\n",
    "# (sorry if this is confusing, but automatically changing this for the transformations can lead to \n",
    "# a lot of subtle errors, so I decided against doing this)\n",
    "\n",
    "# add the two tomograms\n",
    "for name, trafo in zip(tomo_names, transformations):\n",
    "    im_name = os.path.splitext(name)[0]\n",
    "    im_path = os.path.join(example_input_data, name)\n",
    "    \n",
    "    # we need to pass additional 'view' arguments for the tomograms.\n",
    "    # view arguments can modify the viewer state for loading the image source\n",
    "    # here, we adjust the contrast limits to load the tomograms with\n",
    "    # the correct contrast already and we set the affine trasnformtaiton\n",
    "    # that will map the tomograms to the correct position via sourceTransforms\n",
    "    im = imageio.volread(im_path)\n",
    "    min_val, max_val = im.min(), im.max()\n",
    "    view = mobie.metadata.get_default_view(\n",
    "        \"image\", im_name,\n",
    "        source_transform={\"parameters\": trafo},\n",
    "        contrastLimits=[min_val, max_val]\n",
    "    )\n",
    "    mobie.add_image(\n",
    "        input_path=im_path,\n",
    "        input_key=\"\",\n",
    "        root=mobie_project_folder,\n",
    "        dataset_name=dataset_name,\n",
    "        image_name=im_name,\n",
    "        menu_name=\"em\",  # also put the tomo sources in the em menu\n",
    "        resolution=resolution,\n",
    "        scale_factors=scale_factors,\n",
    "        transformation=trafo,\n",
    "        chunks=chunks,\n",
    "        target=target,\n",
    "        max_jobs=max_jobs,\n",
    "        view=view,\n",
    "        unit=unit\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Next, we add a fluorescence image that is also part of the example dataset.\n",
    "\n",
    "input_path = os.path.join(example_input_data, \"fluorescence_downsampled.tif\")\n",
    "\n",
    "# The name of the image in mobie.\n",
    "im_name = \"fluorescence\"\n",
    "# We choose 'lm' as menu name, because this is a lightmicroscopy source\n",
    "menu_name = \"lm\"\n",
    "\n",
    "# This is again a 2d image, so we set all values for Z to 1.\n",
    "unit = \"nanometer\"\n",
    "resolution = [1., 100., 100.]\n",
    "scale_factors = [[1, 2, 2], [1, 2, 2], [1, 2, 2]]\n",
    "chunks = (1, 512, 512)\n",
    "\n",
    "# we set the default display color to green.\n",
    "view = mobie.metadata.get_default_view(\"image\", im_name, color=\"green\")\n",
    "\n",
    "mobie.add_image(\n",
    "    input_path=input_path,\n",
    "    input_key=\"\",\n",
    "    root=mobie_project_folder,\n",
    "    dataset_name=dataset_name,\n",
    "    image_name=im_name,\n",
    "    menu_name=menu_name,\n",
    "    resolution=resolution,\n",
    "    scale_factors=scale_factors,\n",
    "    view=view,\n",
    "    chunks=chunks,\n",
    "    target=target,\n",
    "    max_jobs=max_jobs,\n",
    "    unit=unit\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# as last image, we add a binary mask for the foreground in the image\n",
    "input_path = os.path.join(example_input_data, \"em_mask.tif\")\n",
    "mask_name = \"mask\"\n",
    "\n",
    "# again, the mask is 2d\n",
    "unit = \"nanometer\"\n",
    "chunks = [1, 256, 256]\n",
    "resolution = [1., 160., 160.]\n",
    "scale_factors = [[1, 2, 2]]\n",
    "\n",
    "mobie.add_image(\n",
    "    input_path=input_path,\n",
    "    input_key=\"\",\n",
    "    root=mobie_project_folder,\n",
    "    dataset_name=dataset_name,\n",
    "    image_name=mask_name,\n",
    "    menu_name=\"em\",\n",
    "    resolution=resolution,\n",
    "    chunks=chunks,\n",
    "    scale_factors=scale_factors,\n",
    "    unit=unit\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding segmentation data\n",
    "\n",
    "In addition to image data and masks, MoBIE supports segmentations, which contain label masks for different objects\n",
    "(e.g. organs, cells, ultrastructure) in the volume. For segmentations, MoBIE also supports tables, which contain additional properties for the objects in the segmentation.\n",
    "The function `add_segmentation` copies the input data and also generates the default table for the segmentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# we add a segmentation for several of the cells visible in the em-overview image\n",
    "input_path = os.path.join(example_input_data, \"em_segmentation.tif\")\n",
    "segmentation_name = \"cells\"\n",
    "\n",
    "unit = \"nanometer\"\n",
    "resolution = [1., 30., 30.]\n",
    "chunks = [1, 256, 256]\n",
    "scale_factors = [[1, 2, 2], [1, 2, 2], [1, 2, 2], [1, 2, 2]]\n",
    "\n",
    "mobie.add_segmentation(\n",
    "    input_path=input_path,\n",
    "    input_key=\"\",\n",
    "    root=mobie_project_folder,\n",
    "    dataset_name=dataset_name,\n",
    "    segmentation_name=segmentation_name,\n",
    "    menu_name=\"em-segmentation\",\n",
    "    resolution=resolution,\n",
    "    chunks=chunks,\n",
    "    scale_factors=scale_factors,\n",
    "    add_default_table=True  # add the default table with the properties mobie needs to interact with table and segmentation\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Updating views\n",
    "\n",
    "The `create_view` function can be used to create new views or update existing views (if `overwrite=True` is set).\n",
    "Here, we use it to update the default view for our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we update the default bookmark so that both the raw data \n",
    "# and the segmentation are loaded upon opening the dataset\n",
    "source_list = [[raw_name], [segmentation_name]]\n",
    "settings = [ \n",
    "    {\"color\": \"white\", \"contrastLimits\": [0., 255.]},\n",
    "    {\"lut\": \"glasbey\", \"opacity\": 0.75}\n",
    "]\n",
    "\n",
    "mobie.create_view(dataset_folder, \"default\",\n",
    "                  sources=source_list, display_settings=settings,\n",
    "                  overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Publishing the project \n",
    "\n",
    "The project created above will be located on the local filesystem at `mobie_project_folder`.\n",
    "In order to share it with collaborators or make the data public, MoBIE can also read data stored in a\n",
    "[AWS S3](https://aws.amazon.com/s3/) compatible object store.\n",
    "For this, some additional metadata is necessary, that can be generated via `add_remote_project_metadata`.\n",
    "\n",
    "The data then needs to be uploaded to the s3 storage by some appropriate tool and the metadata needs to be uploaded to github to make it accessible for MoBIE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to generate the metadata for publishing the project, the\n",
    "# following information is needed:\n",
    "# - bucket_name: the name of the bucket in the object store\n",
    "# - service_endpoint: the address of the service endpoint used.\n",
    "#                     this allows specifying object stores that are different from aws\n",
    "#                     here, we use the object store located at EMBL Heidelberg as service endpoint.\n",
    "#                     to use an aws s3 endpoint, set it to https://s3.amazonaws.com \n",
    "bucket_name = \"my-test-bucket\"\n",
    "\n",
    "service_endpoint = \"https://s3.embl.de\"\n",
    "\n",
    "mobie.metadata.add_remote_project_metadata(\n",
    "mobie_project_folder,\n",
    "    bucket_name,\n",
    "    service_endpoint\n",
    ")\n",
    "\n",
    "# Once the metadata is generated, you can upload your project. \n",
    "# MoBIE can access projects directly from an s3 compatible object store.\n",
    "# Optionally the metadata can be uploaded to github to have it under version control;\n",
    "# the github repository can also be used as entry point for the MoBIE viewer.\n",
    "\n",
    "# 1.) Upload the complete folder at \"mobie_project_folder\" to the s3 bucket.\n",
    "# There are several tools available to achieve this, for example\n",
    "# aws s3 sync (https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html)\n",
    "# The sync command would look something like this (assuming the file paths used in this example)\n",
    "# $ aws s3 sync /home/pape/Work/data/mobie/mobie_example_project/data https://s3.embl.de/my-test-bucket\n",
    "\n",
    "# 2.) (OPTIONAL!) Create a github repository for this project and upload the metadata to it:\n",
    "# - Go to https://github.com/ and log into or create your account\n",
    "# - Create a new empty (!) repository, e.g. called \"my-mobie-project\"\n",
    "# - Go to /home/pape/Work/data/mobie_example_project in a terminal (again assuming the filepaths used in the example notebook)\n",
    "# - Initialize git via \n",
    "#   $ git init\n",
    "# - Add the repository you just created as remote via\n",
    "#   $ git remote add origin https://github.com/<USERNAME>/my-mobie-project\n",
    "# - Tell git to ignore the image data files (n5 files) by creating a file \".gitignore\" and adding the line \"*.n5\"\n",
    "#   This is very important, because otherwise we would add all the image data to git.\n",
    "# - Add the metadata to git via\n",
    "#   $ git add .\n",
    "# - Upload the data to github via\n",
    "#   $ git push origin master"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Troubleshooting\n",
    "\n",
    "TODO"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}