invalid tile byte count when using writing tif file

[stephandooper]

When trying to write a black-white tif file (a tumor mask for breast cancer H&E slides) I am running into an error which I have been trying to solve for the last few hours, but to no avail. I was wondering if you would happen to know what could be the cause of this. I have also tried to solve it by replacing it with a conda installation, but with the same result.

• example tif file and code included below

Installation information

   linux: 5.8.14-arch1-1
   python: 3.8
   vips: https://github.com/libvips/libvips/releases/download/v8.11.2/vips-8.11.2.tar.gz
   libjpeg: libjpeg-turbo8-dev
   ibgtk2.0-dev
   libgsf-1-dev
   libtiff5-dev
   pyvips: 2.1.15

sample code:

mask_fname = 'tumor_010_mask.tif'
image = pyvips.Image.new_from_file(mask_fname)

# does not work, gives TIFFFillTile errors
arr = np.ndarray(buffer=image.write_to_memory())

# does not work, gives TIFFFillTile errors
image.write_to_file('test.tif')

Whenever I try to run the above sample code, I get the following errors:

---------------------------------------------------------------------------
Error                                     Traceback (most recent call last)
<ipython-input-29-21eb559af30f> in <module>
      1 image = pyvips.Image.new_from_file(mask_fname, access=pyvips.Access.SEQUENTIAL)
      2 
----> 3 arr = np.ndarray(buffer=image.write_to_memory(),
      4                            dtype=dict_inverse(dtype_to_format)[image.format],
      5                            shape=[image.height, image.width, image.bands])

/usr/local/lib/python3.8/site-packages/pyvips/vimage.py in write_to_memory(self)
    619         pointer = vips_lib.vips_image_write_to_memory(self.pointer, psize)
    620         if pointer == ffi.NULL:
--> 621             raise Error('unable to write to memory')
    622         pointer = ffi.gc(pointer, glib_lib.g_free)
    623 

Error: unable to write to memory
  TIFFFillTile: 0: Invalid tile byte count, tile 0
TIFFFillTile: 0: Invalid tile byte count, tile 1
TIFFFillTile: 0: Invalid tile byte count, tile 2
TIFFFillTile: 0: Invalid tile byte count, tile 3
TIFFFillTile: 0: Invalid tile byte count, tile 4
TIFFFillTile: 0: Invalid tile byte count, tile 5
TIFFFillTile: 0: Invalid tile byte count, tile 6
.
.
.
TIFFFillTile: 0: Invalid tile byte count, tile 189
TIFFFillTile: 0: Invalid tile byte count, tile 190

example.zip

[jcupitt]

Hi @stephandooper,

I think your TIFF file must be corrupt or truncated. It looks like a huge pyramidal TIFF, but it's only 1.7mb. Can any program load it?

$ vips copy tumor_010_mask.tif x.png
(vips:2501111): VIPS-WARNING **: 13:54:51.287: error in tile 0 x 0
...

[stephandooper]

hi @jcupitt
Thank you for your quick reply.
I have been able to open it using ASAP's multiresolutionimageinterface.
code snippet below.

The thing about this particular tif is that there is only a very small region which contains tumor (image in the attachment). I have overlayed the tif file I send as an example with the actual tissue using ASAP, which also loads it without problem in the viewer.
The tif file itself only contains binary values (0,1), where > 99% of the file has the value 0.

More general: I have an entire folder with these masks and pyvips is not able to load any of them. I think that these masks have been generated using the writer that comes with ASAP. Could it be that pyvips does not support it?

reader = mir.MultiResolutionImageReader()
image = reader.open('tumor_mask_010.tif)
print('Image dimensions at level 0: {}'.format(image.getDimensions()))
extracted_patch = image.getUCharPatch(42245, 169268, 1500, 1500, 3) # x at level 0, y at level 0, width, height, level
plt.imshow(extracted_patch)

Image dimensions at level 0: (97792, 215552)

output of the program:

[jcupitt]

ImageMagick fails too:

$ convert tumor_010_mask.tif[6] x.png
convert-im6.q16: 0: Invalid tile byte count, tile 0. `TIFFFillTile' @ error/tiff.c/TIFFErrors/606.
...

The [6] means read page 6 of the TIFF, which is one of the lower resolution layers and not too large.

I think this is a non-standard TIFF file, unfortunately. You could ask the ASAP devs for their opinion.

[stephandooper]

I have had a discussion with one of the developers this afternoon. His hypothesis is that it has to do something with the lzw compression that is being used for mask files when saving them to tiff, which can be obtained from tiffinfo:

(truncated to avoid overflowing)

tiffinfo tumor_010_mask.tif
TIFF Directory at offset 0x5122 (20770)
  Image Width: 97792 Image Length: 215552
  Tile Width: 512 Tile Length: 512
  Resolution: 41136.8, 41136.8 pixels/cm
  Bits/Sample: 8
  Sample Format: unsigned integer
  Compression Scheme: LZW
  Photometric Interpretation: min-is-black
  Orientation: row 0 top, col 0 lhs
  Samples/Pixel: 1
  SMin Sample Value: 0
  SMax Sample Value: 2
  Planar Configuration: single image plane

I will try re-generating the masks using pyvips as the writer instead of asap and see what it produces. I think this should be possible

[jcupitt]

I did a little debugging and it breaks here in libvips:

https://github.com/libvips/libvips/blob/master/libvips/foreign/tiff2vips.c#L1789

It's just calling TIFFReadTile() in the most vanilla way you can. I don't think any program that uses libtiff to read TIFFs will be able to read these files.

Perhaps ASAP is using a different TIFF library? Maybe bitmiracle?

Tracing a little way into libtiff, it ends up calling TIFFFillTile here:

https://gitlab.com/libtiff/libtiff/-/blob/master/libtiff/tif_read.c#L1127

Which then tries to fetch the number of byes in the compressed tile:

https://gitlab.com/libtiff/libtiff/-/blob/master/libtiff/tif_dirread.c#L6386

Which is zero, so you get the error.

I think ASAP's tiff writer has not set the byte count for the compressed tile.

[jcupitt]

Or they could be using a very old version of libtiff, I suppose.

[stephandooper]

From the github page it appears that it is dependent on libtiff 4.0.1 and higher: https://github.com/computationalpathologygroup/ASAP

I am not really familiar with the libtiff version numbers (libtiff, libtiff4, libtiff5, etc) so I am not sure whether using an old version of libtiff is the culprit.

[jcupitt]

Sorry, I keep making very small posts. In pyvips you can create an image like this with:

Python 3.9.5 (default, May 11 2021, 08:20:37) 
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyvips
>>> x = pyvips.Image.new_from_file("Mirax2.2-1.mrxs")
>>> x.width
101832
>>> x.height
219976
>>> x.bands
4
>>> mask = x[1] > 128
>>> mask.write_to_file("x.tif", compression="lzw", tile_width=512, tile_height=512, pyramid=True)
>>>

The tile looks like this:

$ tiffinfo x.tif 
TIFF Directory at offset 0x173812d0 (389550800)
  Image Width: 101832 Image Length: 219976
  Tile Width: 512 Tile Length: 512
  Resolution: 42757.8, 42674.7 pixels/cm
  Bits/Sample: 8
  Sample Format: unsigned integer
  Compression Scheme: LZW
  Photometric Interpretation: min-is-black
  Orientation: row 0 top, col 0 lhs
  Samples/Pixel: 1
  Planar Configuration: single image plane
  Predictor: horizontal differencing 2 (0x2)
TIFF Directory at offset 0x246a3f1a (610942746)
  Subfile Type: reduced-resolution image (1 = 0x1)
  Image Width: 50916 Image Length: 109988
  Tile Width: 512 Tile Length: 512
  Resolution: 42757.8, 42674.7 pixels/cm
  Bits/Sample: 8
  Compression Scheme: LZW
  Photometric Interpretation: min-is-black
  Orientation: row 0 top, col 0 lhs
  Samples/Pixel: 1
  Planar Configuration: single image plane
  Predictor: horizontal differencing 2 (0x2)
.... etc.

And can be opened like this (reading page 7 to keep the size reasonable):

$ python3
Python 3.9.5 (default, May 11 2021, 08:20:37) 
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyvips
>>> x = pyvips.Image.new_from_file("x.tif", page=7)
>>> x.write_to_file("x.png")
>>> 

To make:

[stephandooper]

Then it seems that something is going wrong within the ASAP pipeline. As a sanity check I have read an original tissue file into asap and then wrote it back to tif.

The result is that pyvips is able to read the resulting array into memory, and write it to tif without any problem. The only repeating problem is that the masks generated by the asap pipeline are able to be processed into pyvips.

As a workaround, I have been able to come up with a rather intensive workaround: I can read the culprit masks into memory using ASAP, and then read the resulting array into pyvips using read_from_buffer. Then, I can simply save the resulting array using pyvips. The resulting file will have no problem being handled by pyvips.

As for the problem itself: I suspect you are right in saying that it has something to do with ASAP. I will try to investigate this further and post an issue when I am able to reproduce the problem.

Since the problem probably lies somewhere within ASAP I will close this issue.