Processing large images in linux

ImageMagick

Increase the file size that can be processed

ImageMagick is a suite of programs to process image files.

Limits to image size are set by file policy.xml (in Ubuntu 17.10, /etc/ImageMagick-6/policy.xml). To deal with large images, you'll have to increase those limits, from something like this

<policymap>
  <!-- <policy domain="resource" name="temporary-path" value="/tmp"/> -->
  <policy domain="resource" name="memory" value="256MiB"/>
  <policy domain="resource" name="map" value="512MiB"/>
  <policy domain="resource" name="width" value="16KP"/>
  <policy domain="resource" name="height" value="16KP"/>
  <policy domain="resource" name="area" value="128MB"/>
  <policy domain="resource" name="disk" value="1GiB"/>

to

<policymap>
  <!-- <policy domain="resource" name="temporary-path" value="/tmp"/> -->
  <policy domain="resource" name="memory" value="5GiB"/>
  <policy domain="resource" name="map" value="5GiB"/>
  <policy domain="resource" name="width" value="45GP"/>
  <policy domain="resource" name="height" value="45GP"/>
  <policy domain="resource" name="area" value="5GB"/>
  <policy domain="resource" name="disk" value="20GiB"/>

Getting image metadata

identify -verbose file.tif

Extract one level from multilevel microscopy images

First, inspect the levels available

$ identify "file.ndpi"
file.ndpi[0] TIFF 80640x41984 80640x41984+0+0 8-bit sRGB 233.7MB 0.000u 0:00.019
file.ndpi[1] TIFF 20160x10496 20160x10496+0+0 8-bit sRGB 233.7MB 0.000u 0:00.000
file.ndpi[2] TIFF 5040x2624 5040x2624+0+0 8-bit sRGB 233.7MB 0.000u 0:00.000
file.ndpi[3] TIFF 1260x656 1260x656+0+0 8-bit sRGB 233.7MB 0.000u 0:00.000
file.ndpi[4] TIFF 315x164 315x164+0+0 8-bit sRGB 233.7MB 0.000u 0:00.000
file.ndpi[5] TIFF 1145x391 1145x391+0+0 8-bit sRGB 233.7MB 0.000u 0:00.000
file.ndpi[6] TIFF 572x195 572x195+0+0 8-bit Grayscale Gray 233.7MB 0.000u 0:00.000

Extract the level of interest, e.g. level 4

convert file.ndpi[4] file_4.tif

If the output is going to be large, you need to save as BigTIFF

convert file.ndpi[0] TIFF64:file_0.tif

Adjust intensity levels automatically

convert file.tif -channels rgb -auto-level tiff64:file_out.tif

NDPITools

NDPITools are two small binaries and an ImageJ plugin for NDPI files produced by Hammamatsu microscopes.

To install in Ubuntu, download the binaries to a local directory

sudo mkdir /opt/ndpitools
cd /opt/ndpitools
sudo wget https://www.imnc.in2p3.fr/pagesperso/deroulers/software/ndpitools/download/ndpitools-1.7.2/lin64/ndpi2tiff
sudo wget https://www.imnc.in2p3.fr/pagesperso/deroulers/software/ndpitools/download/ndpitools-1.7.2/lin64/ndpisplit

Change permissions to make them executable

sudo chmod +x ndpi2tiff ndpisplit

Edit ~/.bashrc to add the binaries to the path

export PATH="/opt/ndpitools:$PATH"

Split NDPI file into separate levels

Running the command

ndpisplit file.ndpi

would produce something like this

file_macro.tif
file_map.tif
file_x0.078125_z0.tif
file_x0.3125_z0.tif
file_x1.25_z0.tif
file_x20_z0.tif
file_x5_z0.tif

Convert NDPI to standard TIFF

Running the command

ndpi2tiff file.ndpi

would produce a file

file.tif