TIFFFillTile: Read error when writing a TIFF file on a per-tile basis.

[MartinBuessemeyer]

Hello People,

firstly, thanks a lot for providing the python bindings for vips. They are really helpful 😄.

I am facing a library issue when trying to write a tiff image in tile-based fashion.

The use case:
I get tiles generated by some kind of source (e.g. a CV deep learning model). Since the composed image would be too large to fit in RAM, I want to write it to disk on a per-tile basis, but when reading the image again, I am getting a TIFFFillTile: Read error at row 896, col 896, tile 16; got 0 bytes, expected 49152.

I broke down the issue to a minimal code example:

def test_isolate_error():
    from tempfile import NamedTemporaryFile
    import pyvips as vips

    def numpy_to_vips(array: np.ndarray) -> vips.Image:
        array = array.transpose((1, 0, 2))
        h,w,c = array.shape
        array = array.ravel()
        return vips.Image.new_from_memory(
            data=array.data,
            width=w,
            height=h,
            bands=c,
            format="uchar",
        )
    tiff_size = 1024
    tile_size = 512
    tiling = True
    path = Path(NamedTemporaryFile(suffix=".tiff").name)

    vips_img = vips.Image.black(tiff_size, tiff_size, bands=3)
    vips_img.write_to_file(str(path), tile=tiling)

    for x in range(0, tiff_size, tile_size):
        for y in range(0, tiff_size, tile_size):
            region_img = numpy_to_vips(np.ones((tiff_size, tiff_size, 3), dtype=np.uint8) * 255)
            vips_img.insert(region_img, x, y)
            vips_img.write_to_file(str(path), tile=tiling)
            vips_img = vips.Image.new_from_file(str(path))

This code is failing for me in the latest ubuntu docker image with the most recent vips release with the following error message:

Error Message:
>           raise Error('unable to call {0}'.format(operation_name))
E           pyvips.error.Error: unable to call VipsForeignSaveTiffFile
E             TIFFFillTile: Read error at row 896, col 896, tile 16; got 0 bytes, expected 49152

Of course, it might be the case that my code for writing the image on a per-tile basis is just wrong, and I would need to write it differently. In both cases, help would be appreciated 👍.

Thanks for your time,
Martin

[jcupitt]

Hi @MartinBuessemeyer,

Sorry, I don't think this is going to work :( The best you can do with pyvips is something like (untested!!):

tiles = [some expression to yield a list of numpy tiles]
# you can use `new_from_array` to make a vips image from a numpy array
# they will share memory buffers, so there's no copying and it should be efficient
image = pyvips.Image.arrayjoin([pyvips.Image.new_from_array(tile) for tile in tiles],
                               across=tile_across)
image.write_to_file("x.tif", tile=True, pyramid=True)

But this will unfortunately force the whole image into ram. You could test it and see how memory performance scales, but I fear it may not be good enough.

We've been talking about a better way to generate images like this, but it's not trivial and no one's done the work.

[MartinBuessemeyer]

Thanks for the quick response and your help. I'll give it a try 👍.
Feel free to close the issue if you consider this as done :).