.loom, an efficient file format for large omics datasets
loompy, a Python library for creating and working with data in
loom-viewer, a web app for browsing large single-cell RNA-seq datasets
This repository is the loom-viewer part. Read more about .loom and loompy here.
You can choose between installation through PyPi, or build
loom-viewer from source.
Installing through pip
- Install the Python 3.x version of Anaconda.
Note: loompy and loom-viewer only work with Python 3
- Open a terminal.
Windows users: open
Programs > Anaconda 3 > Anaconda Promp.
- Install Loom with the following command:
pip install loompy pip install loom-viewer
Tip: Loom is under development and updated often. To ensure you have the latest version, do this:
pip install -U loom-viewer
Installing from source
- Install the Python 3.x version of Anaconda.
Note: loompy and loom-viewer only work with Python 3
- If you haven't already, install
loompy, either from source or through PyPI:
pip install loompy
- Install depedencies:
- npm ( ubuntu:
sudo apt-get install npm)
- webpack ( ubuntu:
sudo apt-get install webpack)
- Clone this repository with git
git clone https://github.com/linnarsson-lab/loom-viewer.git
- Navigate to the newly created
loom-viewerfolder, and install all required node packages:
- Run the build script. On Mac/Linux:
To build the production version, type:
Tip: when developing, you only modified the Python code, and want to skip building the client side, type:
The Loom CLI tool
After installation, you should have access to the
loom CLI from the terminal ("Anaconda Prompt" on Windows). Test if everything works correctly by typing:
To learn more about the loom tool, type:
For help with individual commands, just type:
loom [command] --help
Getting started with the viewer
To open the Loom Viewer locally, type in
loom without any extra flags:
This will start a local server, and automatically open
localhost:8003 in your default browser. Since this is the first time running the tool, it will fail to find any datasets, showing an empty list and an error message about fetch being broken on Safari (we are working on that one).
Close the server for now (CTRL+C on Linux/Windows, CMD+C on Mac).
To view our Loom files, we have to go through the following steps:
- Store the Loom files in a directory where the viewer can find it
a. (optional) To see heatmap tiles, generate them with
loom tile <loom filename>b. (optional) To quickly access gene expression data, pre-expand the rows with
loom expand -r <loom filename>
- Start the loom server, or if it is already running, refresh the page
Where the server looks for Loom files
The first time you run
loom-datasets folder will be created in your home folder. This is where the
loom-viewer server looks for Loom files by default.
You can explicitly point the loom CLI to a different path with
loom --dataset-path <your path here>
The dataset folder is organised in projects. These are sub-folders in the root dataset folder. Folders represent individual projects. The Loom files are stored in these project folders: (all other files are ignored)
~/ ├─ loom-datasets/ . ├─ # Project 1 (folder) │ ├─ # dataset1.loom (loom file) │ └─ # dataset2.loom (loom file) └─ # Project 2 (folder) ├─ # dataset3.loom (loom file) └─ # dataset4.loom (loom file)
Either create a Loom file with
loompy, or download one from a loom viewer hosted on-line.
If you want, you can open it now and browse it on-line!
On the right you can see a cloud icon that links to the original loom file. Right-click and save the file to an appropriately labeled project folder in the
~/ ├─ loom-datasets/ . ├─ Linnarsson Lab . └─ Cortex.loom
Once we start a local loom server (see below) and open
localhost:8003, the resulting view should be a single project,
Linnarsson Lab, with a single loom file,
Cortex.loom. Opening it should give the same results as the on-line versoin.
Generating heatmap tiles with
By default, the server generates and caches the metadata, attributes, and individual rows and columns as they are requested (see below).
However, to view a heat map of the entire data matrix, the associated image tiles must be pre-generated from the Loom file (the reason we do not do this on-the-fly is that generating tiles requires iterating over all the data in a Loom file. For bigger files is so slow that it would freeze the system if done while running a server).
Note: try to avoid running a server at the same time you generate tiles, as this can lead to unexpected HDF5 behaviour due to multiple programs accessing the same file at once.
Tile generation is done through the
loom tile command. For a list of possible flags, type
loom tile --help:
usage: loom tile [-h] [--project [PROJECT [PROJECT ...]]] [-A] [-t] [file [file ...]] positional arguments: file Loom file(s) to expand. Expands all files matching the provided file names. To avoid this, use an absolute path to specify a single file. optional arguments: -h, --help show this help message and exit --project [PROJECT [PROJECT ...]] Project(s) for which to expand all files. -A, --all Expand all loom files. -t, --truncate Remove previously expanded tiles if present (False by default)
Note that the
loom tile command will automatically search all projects for all matching file names, and generate tiles for each. Using unique file names is encouraged!
# Generate tiles for all files named `Cortex.loom` or `Cortex_allgenes.loom loom tile Cortex.loom Cortex_allgenes.loom # Generate tiles for `/home/me/loom-datasets/Published/Cortex.loom` # on Windows, this would be something like: # loom tile C:\Users\Job van der Zwan\loom-datasets\Published\Cortex.loom loom tile /home/me/loom-datasets/Published/Cortex.loom # Generate tiles for all files in the `Published` project: loom tile --project Published
Note: the tiles are not automatically updated when changing data in a loom file, meaning the viewer may show the old data! To fix this, re-do this step with the added -t flag:
# -t for "truncate", telling loom to overwrite the old files loom tile -t [filenames and project folders]
Because generation can be slow for larger Loom files, the command checks if the relevant
<filename>.loom.tile subfolder exists, skipping generation if it does. This means that if the tile command is aborted without finishing, the remaining tiles will not be added when running
loom tile again! To override this, run:
loom tile -t <filename.loom>. Alternatively, delete the subfolder in question.
Pre-generating/updating JSON files for metadata, attributes and rows with
Similar to pre-generating image tiles, is possible to pre-generate all metadata, like attributes and gene rows, for a Loom file. This step used to be mandatory, but now the server generates the required JSOn on the fly.
There are two situations in which this is still relevant.
First, if you change data in a loom file, previously generated JSON files for that data becomes outdated. Currently the loom server cannot detect this change, so the JSON files have to manually be updated.
Second, because opening and closing an HDF5 connection gets slower as file-size increases, this may still be useful when serving very large loom files on-line.
For either of these scenarios we use the
loom expand command.
usage: loom expand [-h] [--project [PROJECT [PROJECT ...]]] [-A] [-C] [-t] [-m] [-a] [-r] [-c] [file [file ...]] positional arguments: file Loom file(s) to expand. Expands all files matching the provided file names. To avoid this, use an absolute path to specify a single file. When combined with --clear it clears all expanded files instead. optional arguments: -h, --help show this help message and exit --project [PROJECT [PROJECT ...]] Project(s) for which to expand all files (or clear expansion with --clear). -A, --all Expand all loom files (or clear expansion with --clear). -C, --clear Remove previously expanded files. -t, --truncate Replace previously expanded files if present (False by default). Only does something in combination with expansion (-m, -a, -r or -c). -m, --metadata Expand metadata (False by default) -a, --attributes Expand attributes (False by default) -r, --rows Expand rows (False by default) -c, --cols Expand columns (False by default)
It is nearly identical to the
loom tile command, except that we also need to specify whether we want to generate JSON files for metadata, attributes, rows and/or columns.
# Generate JSON for metadata, attributes and rows # for all files named `Cortex.loom` or `Cortex_allgenes.loom, # and overwrite ay previous JSON files encountered loom expand -mart Cortex.loom Cortex_allgenes.loom # Generate attributes JSON files for /home/me/loom-datasets/Published/Cortex.loom # on Windows, this would be something like: # loom expand -a C:\Users\Job van der Zwan\loom-datasets\Published\Cortex.loom loom expand -a /home/me/loom-datasets/Published/Cortex.loom # Generate row JSON files for all files in the `Published` project: loom expans -r --project Published
The website is stuck with the message that fetch is broken on Safari, but I am not using Safari
The loom server cannot find any loom files, but the website mistakenly thinks it failed to fetch the list of loom files (which (for now) is guaranteed to happen on Safari anyway, hence the warning message).
See above for where and how to store your loom files so that the server can find them.
If this did not solve your problem, open an issue.
There are no heatmap tiles in the viewer
If you have not done so yet, generate the tiles for the heatmap first with
loom tile <filename.loom>. See above for more details.
If you have done so but something went wrong, you can try again with
loom tile -t <filename.loom>.
Generating tiles fails with
ERROR - module 'scipy.misc' has no attribute 'toimage'
It is likely that you are missing the required image library. This can happen if you use MiniConda, which has fewer default packages. Try installing it with
conda install pillow.
Why is pre-generation required for serving the metadata or viewing heatmap tiles data
There are two reasons for this:
First, h5py can not safely cope with multiple people accessing the same Loom file simultanously. This is not a problem when exploring loom files locally (unless you open the same loom file in multiple tabs), but when using the
loom-viewer as a server to set up a website for sharing loom-files, two or more people opening the same loom files will cause the server to crash.
Second, loading the genes from the loom file and converting it to either JSON or a PNG is a fairly slow process. The viewer is mainly intended to explore and share finished loom files that do not change, making it wasteful to do this work multiple times. Serving pre-generated static files is thousands of times faster.
This limits the usefulness as an off-line viewer somewhat for now, but we have plans to make the server capable of detecting when you changed something in a loom file, and automatically update the data in question. This would make this process easier, but for now you have to manually trigger an update to the generated JSON files and image tiles when changing data in the a loom file.