Skip to content

Updated content to add dask distributed to the material #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jul 3, 2025
Merged

Updated content to add dask distributed to the material #9

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
df2284e
Updated notebooks to use dask distributed extension
fercer Jul 3, 2025
d42de88
Merge branch 'main' into dev-content
fercer Jul 3, 2025
f6fa53b
Added changes to the requirements.md
fercer Jul 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion notebooks/Solved_Zarr_and_Dask_for_large-scale_imaging-Part-1.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2313,6 +2313,15 @@
"import tifffile"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"input_path = r\"C:\\Users\\Public\\Documents\\WSI_example\\CMU-1.svs\""
]
},
{
"cell_type": "code",
"execution_count": 48,
Expand All @@ -2332,7 +2341,7 @@
}
],
"source": [
"tif_store = tifffile.imread(r\"C:\\Users\\Public\\Documents\\WSI_example\\CMU-1.svs\", aszarr=True)\n",
"tif_store = tifffile.imread(input_path, aszarr=True)\n",
"tif_store"
]
},
Expand Down
1,170 changes: 755 additions & 415 deletions notebooks/Solved_Zarr_and_Dask_for_large-scale_imaging-Part-2.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

67 changes: 10 additions & 57 deletions notebooks/Zarr_and_Dask_for_large-scale_imaging-Part-1.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -256,23 +256,6 @@
"- [ ] Create a delayed function (decorated with `@dask.delayed`) that can be applied lazily"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n",
"def grid_x(height, width, offset = 0):\n",
" x = np.arange(offset, offset + width)\n",
" return np.tile(x, (height, 1))\n",
"\n",
"\n",
"def grid_y(height, width, offset = 0):\n",
" y = np.arange(offset, offset + height)\n",
" return np.tile(y[:, None], (1, width))"
]
},
{
"cell_type": "code",
"execution_count": null,
Expand All @@ -288,13 +271,6 @@
"## 1.4 Stack, Concatenate, and Block"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- [ ] Find a way to create a $1000\\times1000$ pixels image by joining multiple $500\\times500$ pixels blocks "
]
},
{
"cell_type": "code",
"execution_count": null,
Expand Down Expand Up @@ -331,6 +307,15 @@
"import tifffile"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"input_path = r\"C:\\Users\\Public\\Documents\\WSI_example\\CMU-1.svs\""
]
},
{
"cell_type": "code",
"execution_count": null,
Expand Down Expand Up @@ -463,31 +448,6 @@
"ℹ Dask arrays already work with `skimage` functions without calling `.compute()`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from dask_image import ndfilters"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from skimage.filters import threshold_multiotsu"
]
},
{
"cell_type": "code",
"execution_count": null,
Expand All @@ -502,13 +462,6 @@
"- [ ] Visualize the results using `Matplotlib`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
Expand All @@ -523,7 +476,7 @@
"provenance": []
},
"kernelspec": {
"display_name": "scale-up",
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
Expand Down
130 changes: 121 additions & 9 deletions notebooks/Zarr_and_Dask_for_large-scale_imaging-Part-2.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,42 @@
"# 2. Compute on Zarr arrays"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2.1 Set up a Dask cluster"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- [ ] Use the Jupyter's Dask extension to start a distributed cluster\n",
"- [ ] Click the \"+ New\" button at the bottom of the plugin\n",
"\n",
"![image](dask_extension.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- [ ] Click the \"<>\" button to inject the code needed to connect with this cluster\n",
"\n",
"![image](dask_extension_ready.png)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"executionInfo": {
"elapsed": 976,
Expand All @@ -78,6 +111,22 @@
"import skimage"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Change this to the actual path where the image is stored"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"input_path = r\"C:\\Users\\Public\\Documents\\WSI_example\\CMU-1_Crop.ome.zarr\""
]
},
{
"cell_type": "code",
"execution_count": null,
Expand Down Expand Up @@ -124,7 +173,12 @@
"outputs": [],
"source": [
"from cellpose import models, transforms\n",
"import torch"
"import torch\n",
"\n",
"gpu = torch.cuda.is_available()\n",
"model_type = \"cyto3\"\n",
"\n",
"cellpose_model = models.CellposeModel(gpu=gpu, model_type=model_type)"
]
},
{
Expand All @@ -147,7 +201,24 @@
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
"source": [
"img_t = transforms.convert_image(da_sel.compute(), channel_axis=2, channels=[0, 0])\n",
"img_t = transforms.normalize_img(img_t, invert=False, axis=2)\n",
"\n",
"labels, _, _ = cellpose_model.eval(img_t[None, ...], diameter=None, flow_threshold=None, channels=[0, 0])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "iRTdgTzi_KfU"
},
"outputs": [],
"source": [
"plt.imshow(da_sel)\n",
"plt.imshow(skimage.color.label2rgb(labels), alpha=0.5)"
]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -249,7 +320,7 @@
},
{
"cell_type": "code",
"execution_count": 78,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
Expand Down Expand Up @@ -377,7 +448,7 @@
},
{
"cell_type": "code",
"execution_count": 267,
"execution_count": null,
"metadata": {
"id": "2yzM2YNRcAFI"
},
Expand Down Expand Up @@ -443,9 +514,39 @@
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"metadata": {
"cellView": "form",
"id": "Bh6c9EVy9hST"
},
"outputs": [],
"source": []
"source": [
"with ProgressBar():\n",
" for s in range(1, 6):\n",
" da_labels = da.from_zarr(\"CMU-1_Crop_labels_cellpose_cyto3.zarr\", component=str(s - 1))\n",
"\n",
" da_labels_ds = da_labels[::2, ::2]\n",
" da_labels_ds = da_labels_ds.rechunk()\n",
"\n",
" da_labels_ds.to_zarr(\n",
" \"CMU-1_Crop_labels_cellpose_cyto3.zarr\",\n",
" component=str(s),\n",
" write_empty_chunks=False,\n",
" compressor=zarr.Blosc(clevel=9),\n",
" overwrite=True\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "Bh6c9EVy9hST"
},
"outputs": [],
"source": [
"z_labels = zarr.open(\"CMU-1_Crop_labels_cellpose_cyto3.zarr\", mode=\"a\")"
]
},
{
"cell_type": "markdown",
Expand All @@ -456,7 +557,7 @@
},
{
"cell_type": "code",
"execution_count": 282,
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "Bh6c9EVy9hST"
Expand All @@ -483,7 +584,7 @@
" \"path\" : str(s),\n",
" \"coordinateTransformations\": [\n",
" {\n",
" \"scale\" : [ 1.0, 1.0, 1.0, 4.942E-4 / (2**s), 4.942E-4 / (2**s)],\n",
" \"scale\" : [4.942E-4 / (2**s), 4.942E-4 / (2**s)],\n",
" \"type\" : \"scale\"\n",
" }\n",
" ]\n",
Expand All @@ -495,6 +596,17 @@
"]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "giJcwW_F3vrF"
},
"outputs": [],
"source": [
"z_labels = zarr.open(\"CMU-1_Crop_labels_cellpose_cyto3.zarr\", mode=\"r\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand Down
Binary file added notebooks/dask_extension.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added notebooks/dask_extension_ready.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
11 changes: 6 additions & 5 deletions requirements.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,12 @@ Additionally, install Jupyterlab to follow the notebooks, and its Dask extension
conda install -c conda-forge jupyterlab=4.4.3 dask-labextension=7.0.0
```

## A test image
## Test images

We'll work with big microscopy images. An example of that kind of image can be found [here](https://openslide.cs.cmu.edu/download/openslide-testdata/Aperio/CMU-1.svs).
So it is a good idea to have it downloaded as well.
We'll work with big microscopy images: whole slide images. These are images obtained by scanning slides on which thin slices of tissue have been mounted. A number of examples of these kind of image can be found in the [OpenSlide](https://openslide.org) test data, for example in the [Aperio SVS format](https://openslide.cs.cmu.edu/download/openslide-testdata/Aperio/). These are TIFF variants that can be read with the [`Tifffile` Python library](https://github.com/cgohlke/tifffile).

A crop of that image can be found [here](https://drive.google.com/file/d/17owNcq_Or6aBAyUVE33fyHSS0VvKuHSw/view?usp=sharing), along with its `Zarr` version [here](https://drive.google.com/file/d/1BmNxOrO3vOFPR-PCnV00DYgFsD1sDu47/view?usp=sharing).
For this workshop, we recommend downloading a smaller crop of the CMU-1 image from [our google drive](https://drive.google.com/file/d/17owNcq_Or6aBAyUVE33fyHSS0VvKuHSw/view?usp=sharing), along with [its `Zarr` version](https://drive.google.com/file/d/1BmNxOrO3vOFPR-PCnV00DYgFsD1sDu47/view?usp=sharing).

A smaller example image can be found [here](https://openslide.cs.cmu.edu/download/openslide-testdata/Aperio/CMU-1-Small-Region.svs), and its corresponding `Zarr` version [here](https://drive.google.com/file/d/1MifgafB5mhVAvqjzEAAR_zQajAU5Dcya/view?usp=drive_link).
A alternative even smaller example image can be found [in the OpenSlide test data](https://openslide.cs.cmu.edu/download/openslide-testdata/Aperio/CMU-1-Small-Region.svs), and its corresponding `Zarr` version can also be downloaded from [our google drive](https://drive.google.com/file/d/1MifgafB5mhVAvqjzEAAR_zQajAU5Dcya/view?usp=drive_link).

**Please pre-download either pair of images before the tutorial!**