Open big tif file with C++ API got some error

[shrikedehyperion]

#1627 Expected behavior and actual behavior.

I expected to open this TIF file and using RasterIO method to read it into memory,

file links:
https://drive.google.com/open?id=1ndGyRuOXMGLb6vjFzfLNTbsfS5Rxmx2c

the code sample I use is from GDAL tutorial, like this:

#include <iostream>  
#include <gdal_priv.h>
#include <cpl_conv.h>

using namespace std;

int main()
{
    const char* pszFile;
    GDALAllRegister();
    pszFile = "/*.tif";
    GDALDataset* poDataset = (GDALDataset*)GDALOpen(pszFile, GA_ReadOnly);
    GDALRasterBand* poBand = poDataset->GetRasterBand(1);
    float* pafScanline;
    int nXSize = poBand->GetXSize();
    pafScanline = (float*)CPLMalloc(sizeof(float) * nXSize);
    poBand->RasterIO(GF_Read, 0, 0, nXSize, 1,
    pafScanline, nXSize, 1, GDT_Float32,
     0, 0);
    system("pause");
    return 0;
}

I compile this program,nothing wrong, then run it, nothing wrong until the RAsterIO method began to work, and it gives me error list below:

ERROR 6: Unhandled type for StripOffset/StripByteCount
ERROR 1: /*.tif, band 1: IReadBlock failed at X offset 0, Y offset 0: Unhandled type for StripOffset/StripByteCount

I then try some other IMG file like some smaller JPG file, all is good and no error.
In the same time, I am using GDAL python API, then I use Python Open this TIF file and use ReadRaster method with same parameters, it works, the tif file is correctly read into memory and output right,

import gdal
ds = gdal.Open('/*.tif')
band = ds.GetRasterBand(1)
nXsize = band.XSize
data = band.ReadRaster(xoff=0, yoff=0, xsize=nXsize, ysize =1, buf_xsize=nXsize, buf_ysize=1)

data output:

b'\xe6\x02\xce\x02\xc7\x02\x8e\x02B\x02\xfc\x01\x12\x02\xf5\x01\xd2\x01\xc3\x01\x04\x02\xfc\x01\xc9\x01\x90\x02\xf9\x02\xe4\x02p\x02\x1b\x02\xe8\x01\x02\x02\x00\x0...

I began to suspect if it is some dependencies I forget to set when I build gdal for C++, when I install GDAL with Conda,it tells me there are more packages needs to installed, which is:

bzip2-1.0.8                |       he774522_0         150 KB
cfitsio-3.470              |       he774522_2         592 KB
curl-7.67.0                |       h2a8f88b_0         126 KB
expat-2.2.5                |       he025d50_0          91 KB
freexl-1.0.5               |       hfa6e2cd_0          57 KB
gdal-3.0.2                 |   py37hdf43c64_0         1.5 MB
geos-3.8.0                 |       h33f27b4_0         1.0 MB
geotiff-1.5.1              |       h5770a2b_0         140 KB
hdf4-4.2.13                |       h712560f_2         2.7 MB
hdf5-1.10.4                |       h7ebc959_0        19.2 MB
kealib-1.4.7               |       h07cbb95_6         158 KB
krb5-1.16.4                |       hc04afaa_0         832 KB
libboost-1.67.0            |       hd9e427e_4        31.8 MB
libcurl-7.67.0             |       h2a8f88b_0         284 KB
libgdal-3.0.2              |       h1155b67_0         8.4 MB
libkml-1.3.0               |       he5f2a48_4        54.9 MB
libnetcdf-4.6.1            |       h411e497_2         593 KB
libpng-1.6.37              |       h2a8f88b_0         598 KB
libpq-11.2                 |       h3235a2c_0         3.5 MB
libspatialite-4.3.0a       |       h7ffb84d_0         4.0 MB
libssh2-1.8.2              |       h7a1dbc1_0         186 KB
libtiff-4.1.0              |       h56a325e_0         997 KB
m2w64-expat-2.1.1          |                2         160 KB
m2w64-gettext-0.19.7       |                2         4.2 MB
m2w64-libiconv-1.14        |                6         1.5 MB
m2w64-xz-5.2.2             |                2         395 KB
mkl-service-2.3.0          |   py37hb782905_0         200 KB
mkl_fft-1.0.15             |   py37h14836fe_0         137 KB
mkl_random-1.1.0           |   py37h675688f_0         270 KB
numpy-1.17.4               |   py37h4320e6b_0           5 KB
numpy-base-1.17.4          |   py37hc3f5095_0         4.8 MB
openjpeg-2.3.0             |       h5ec785f_1         226 KB
pcre-8.43                  |       ha925a31_0         511 KB
postgresql-11.2            |       h3235a2c_0        20.0 MB
proj-6.2.1                 |       h9f7ef89_0         8.4 MB
six-1.13.0                 |           py37_0          27 KB
tbb-2018.0.5               |       he980bc4_0         167 KB
tiledb-1.6.3               |       h7b000aa_0         1.6 MB
xerces-c-3.2.2             |       ha925a31_0         4.1 MB

And when I build GDAL for visual studio, I just build GDAL, SQlite and Proj,I don't know which exact package cause the different result in two API, hope you guys could answer my question.

Operating system

Microsoft Windows10.0 17763 64bit
visual studio 2019 community version

GDAL version and provenance

GDAL C++ API version 3.0.2 with sqlite3 and proj6.2.1

GDAL python API version 3.0.2, other dependencies is mention above

[rouault]

I'm not sure why you get it working in Python. Presumably because you don't use the same GDAL version as in C++, and such version being an older version that didn't use some recent optimizations. I get errors with GDAL master both in C++ and Python, which is consistent.

The error comes from the fact that this file is non conformant with the BigTIFF specification.
The data type for the StripByteCounts uses incorrectly SLONG8 (signed long8),

StripByteCounts (279) SLONG8 (17) 98820<21960 21960...

whereas acoording to https://www.awaresystems.be/imaging/tiff/bigtiff.html only unsigned types (LONG8, LONG, SHORT) are allowed for that tag.
So I'd suggest to reach to the data producer so that they fix their file.
There are also more minor issues reported by libtiff on that file

Warning 1: TIFFFetchNormalTag:ASCII value for tag "GeoASCIIParams" contains null byte in value; value incorrectly truncated during reading due to implementation limitations
Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples.

[shrikedehyperion]

I'm not sure why you get it working in Python. Presumably because you don't use the same GDAL version as in C++, and such version being an older version that didn't use some recent optimizations. I get errors with GDAL master both in C++ and Python, which is consistent.

The error comes from the fact that this file is non conformant with the BigTIFF specification.
The data type for the StripByteCounts uses incorrectly SLONG8 (signed long8),

StripByteCounts (279) SLONG8 (17) 98820<21960 21960...

whereas acoording to https://www.awaresystems.be/imaging/tiff/bigtiff.html only unsigned types (LONG8, LONG, SHORT) are allowed for that tag.
So I'd suggest to reach to the data producer so that they fix their file.
There are also more minor issues reported by libtiff on that file

Warning 1: TIFFFetchNormalTag:ASCII value for tag "GeoASCIIParams" contains null byte in value; value incorrectly truncated during reading due to implementation limitations
Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples.

With Python API, I use Conda create a separate environment with python 3.7 and nothing else but gdal3.0.2 and its dependencies, It successed reading that TIF, I could list all the package I have in that virtual environment:

# Name                    Version                   Build  Channel
blas                      1.0                         mkl
bzip2                     1.0.8                he774522_0
ca-certificates           2019.11.27                    0
certifi                   2019.11.28               py37_0
cfitsio                   3.470                he774522_2
curl                      7.67.0               h2a8f88b_0
expat                     2.2.5                he025d50_0
freexl                    1.0.5                hfa6e2cd_0
gdal                      3.0.2            py37hdf43c64_0
geos                      3.8.0                h33f27b4_0
geotiff                   1.5.1                h5770a2b_0
hdf4                      4.2.13               h712560f_2
hdf5                      1.10.4               h7ebc959_0
icc_rt                    2019.0.0             h0cc432a_1
icu                       58.2                 ha66f8fd_1
intel-openmp              2019.4                      245
jpeg                      9b                   hb83a4c4_2
kealib                    1.4.7                h07cbb95_6
krb5                      1.16.4               hc04afaa_0
libboost                  1.67.0               hd9e427e_4
libcurl                   7.67.0               h2a8f88b_0
libgdal                   3.0.2                h1155b67_0
libiconv                  1.15                 h1df5818_7
libkml                    1.3.0                he5f2a48_4
libnetcdf                 4.6.1                h411e497_2
libpng                    1.6.37               h2a8f88b_0
libpq                     11.2                 h3235a2c_0
libspatialite             4.3.0a               h7ffb84d_0
libssh2                   1.8.2                h7a1dbc1_0
libtiff                   4.1.0                h56a325e_0
libxml2                   2.9.9                h464c3ec_0
lz4-c                     1.8.1.2              h2fa13f4_0
m2w64-expat               2.1.1                         2
m2w64-gcc-libgfortran     5.3.0                         6
m2w64-gcc-libs            5.3.0                         7
m2w64-gcc-libs-core       5.3.0                         7
m2w64-gettext             0.19.7                        2
m2w64-gmp                 6.1.0                         2
m2w64-libiconv            1.14                          6
m2w64-libwinpthread-git   5.0.0.4634.697f757               2
m2w64-xz                  5.2.2                         2
mkl                       2019.4                      245
mkl-service               2.3.0            py37hb782905_0
mkl_fft                   1.0.15           py37h14836fe_0
mkl_random                1.1.0            py37h675688f_0
msys2-conda-epoch         20160418                      1
numpy                     1.17.4           py37h4320e6b_0
numpy-base                1.17.4           py37hc3f5095_0
openjpeg                  2.3.0                h5ec785f_1
openssl                   1.1.1d               he774522_3
pcre                      8.43                 ha925a31_0
pip                       19.3.1                   py37_0
postgresql                11.2                 h3235a2c_0
proj                      6.2.1                h9f7ef89_0
python                    3.7.6                h60c2a47_2
setuptools                44.0.0                   py37_0
six                       1.13.0                   py37_0
sqlite                    3.30.1               he774522_0
tbb                       2018.0.5             he980bc4_0
tiledb                    1.6.3                h7b000aa_0
tk                        8.6.8                hfa6e2cd_0
vc                        14.1                 h0510ff6_4
vs2015_runtime            14.16.27012          hf0eaf9b_1
wheel                     0.33.6                   py37_0
wincertstore              0.2                      py37_0
xerces-c                  3.2.2                ha925a31_0
xz                        5.2.4                h2fa13f4_4
zlib                      1.2.11               h62dcd97_3
zstd                      1.3.7                h508b16e_0

As for my original problem, I'll just try use an older version of GDAL.