Demo: Hang / TIFFReadDirectory Error when creating an empty file

[8ern4ard]

Mac OS Monterey M1
gdal 3.4.2 py39h4aaee43_0 conda-forge
rasterio 1.2.10 py39h4f00f16_5 conda-forge

This code hangs for me and does not finish

import rasterio
from rasterio.crs import CRS
from rasterio.transform import Affine

profile = {'driver': 'GTiff', 'dtype': 'uint16', 'nodata': 0.0, 'width': 10978, 'height': 10598, 'count': 18,
           'crs': CRS.from_epsg(32632), 'transform': Affine(10.0, 0.0, 500000.0, 0.0, -10.0, 5200020.0),
           'blockxsize': 1024, 'blockysize': 1024, 'tiled': True, 'interleave': 'band'}

with rasterio.open('/Users/me/Desktop/bug.tif', 'w', **profile):
    pass

When I interrupt and re-try (so the empty and corrupted file exists already):

Traceback (most recent call last):
  File "/Users/me/Library/Application Support/JetBrains/PyCharmCE2021.2/scratches/bug.py", line 11, in <module>
    with rasterio.open('/Users/me/Desktop/bug.tif', 'w', **profile):
  File "/Users/me/mambaforge/envs/automap21/lib/python3.9/site-packages/rasterio/env.py", line 437, in wrapper
    return f(*args, **kwds)
  File "/Users/me/mambaforge/envs/automap21/lib/python3.9/site-packages/rasterio/__init__.py", line 230, in open
    s = writer(path, mode, driver=driver,
  File "rasterio/_io.pyx", line 1122, in rasterio._io.DatasetWriterBase.__init__
  File "rasterio/_io.pyx", line 70, in rasterio._io._delete_dataset_if_exists
  File "rasterio/_shim.pyx", line 78, in rasterio._shim.open_dataset
  File "rasterio/_err.pyx", line 216, in rasterio._err.exc_wrap_pointer
rasterio._err.CPLE_AppDefinedError: /Users/me/Desktop/bug.tif: TIFFReadDirectory:Failed to read directory at offset 272647686

Process finished with exit code 1

It does work however, if:
i) blockxsize / blockysize is 245, or
ii) tiled = false, or
iii) count = 1 or even 20 (in my quick test it hangs only with 17, 18 !)

[rouault]

ok, I can reproduce with:

$ gdal_create test.tif -outsize 10978 10598 -bands 18 -co tiled=yes -co blockxsize=1024 -co blockysize=1024 -co interleave=band -ot uint16
ERROR 1: TIFFResetField:Value exceeds 32bit range of output type.
ERROR 1: TIFFWriteDirectorySec:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.
ERROR 1: TIFFResetField:test.tif: Can not read TIFF directory count
[ hangs ]

[rouault]

Fix in #5481
Workaround: add a BIGTIFF='YES' explicit option to the profile

[8ern4ard]

Thanks for your time! Your gdal error was much more explanatory
ERROR 1: TIFFWriteDirectorySec:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.