All pixels shift to the right in very large rasters #53
Comments
|
@hampmark I encountered this error myself for the first time the other day. I'll do some digging to see if I can figure out what is going on but it would be useful for me to know what file format the input and output files were in, e.g. GeoTIFF, and what front-end you were using for WhiteboxTools. |
|
I used the python bindings to WhiteboxTools on a Windows 10 machine (Python 3.5-64bit, Whitebox tools 1.0.0). I believe all tested so far are GeoTIFF, and I append the information on 3 rasters used where the first (info_INDATA.png) is used to produce a breached DEM (info_BREACHED.png) which in turn was used as indata to create D8_flowacc (info_FLOWACC.png). All 3 raster are shifted one pixel for every operation performed. In this case it means that the D8_flowacc raster is shifted 2 pixels to the right of the first indata raster.
|
|
I'm sorry to be a bother, but would you mind running the PrintGeoTiffTags tool on both the input and output files and post the tool output? It provides a level of detail on the data that I will need to diagnose the issue. Thanks. |
|
This is also a possible duplicate of Issue #45 although that bug report documents a half-pixel shift instead. |
|
Here is the log with the detailed information: Processing DEM: //MARK-VVANALYS8/D$/HM/aros_fromfaruk/outdata/bigger2m/372820_6246331_428933_6320224_2m_dem_crop.tif
Little-endian byte order used GDAL_NODATA (code=42113 type=DT_ASCII count=5 offset=960051501): -999 Processing DEM: //MARK-VVANALYS8/D$/HM/aros_fromfaruk/outdata/bigger2m/Breached_DEMs/WBT372820_6246331_428933_6320224_2m_dem_crop.tif
Little-endian byte order used Processing DEM: //MARK-VVANALYS8/D$/HM/aros_fromfaruk/outdata/bigger2m/Breached_DEMs/D8_Flowacc/WBT372820_6246331_428933_6320224_2m_dem_crop.tif
Little-endian byte order used |
|
I think that something wrong is happening when writing Float64 GeoTIFF or simply writing BigTIFF variant. In my current workflow, I execute in order It is important to note that the extent of all output files are the same as the original DEM, so the shift to the right makes the last colum to right vanishing while the first colum to the left being filled with NoData. Here is some data to try (link valid for the next 14 days) and my current code. I'll do more tests to try find if the issue is with Float64 TIFF or with BigTIFF, (but I think it's with BigTIFF as e same workflow with a subset of the DEM doesn't produce any shift). import os
import subprocess
wbt_dir = "WhiteboxTools_1.2.0" # WhiteboxTools directory
outdir = "shifting_problems" # working directory
dem_path = os.path.join(outdir, "DEM.tif")
# Fill Single-Cell Pits (FSCP)
outFSCP = os.path.join(outdir, "DEM_1_FSCP.tif")
cmd = " ".join([wbt_dir + "\\whitebox_tools.exe",
"-r=FillSingleCellPits",
"--dem='" + dem_path + "'",
"--output='" + outFSCP + "'",
"-v"])
subprocess.run(cmd)
# Breach Single-Cell Pits (BSCP)
outFBSCP = os.path.join(outdir, "DEM_2_FBSCP.tif")
cmd = " ".join([wbt_dir + "\\whitebox_tools.exe",
"-r=BreachSingleCellPits",
"--dem='" + outFSCP + "'",
"--output='" + outFBSCP + "'",
"-v"])
subprocess.run(cmd)
# Breach Depressions Least-Cost
outFBSCP_BREACH = os.path.join(outdir, "DEM_3_FBSCP_BREACH.tif")
cmd = " ".join([wbt_dir + "\\whitebox_tools.exe",
"-r=BreachDepressionsLeastCost",
"--dem='" + outFBSCP + "'",
"--output='" + outFBSCP_BREACH + "'",
"--radius='50'",
"--min_dist",
"--fill",
"-v"])
subprocess.run(cmd)
# Breach Depressions Fast
outFBSCP_BREACH_BFAST = os.path.join(outdir, "DEM_4_FBSCP_BREACH_BFAST.tif")
cmd = " ".join([wbt_dir + "\\whitebox_tools.exe",
"-r=BreachDepressions",
"--dem='" + outFBSCP_BREACH + "'",
"--output='" + outFBSCP_BREACH_BFAST + "'",
"--flat_increment='0.0001'",
"--fill_pits",
"-v"])
subprocess.run(cmd) |
|
After some more tests, I'm now pretty confident that the issue is with the TIFF writer when the output file exceed 4GB and that WhiteboxTools is forced to write a BigTIFF. As long as a normal TIFF below 4GB is written, there is no shift. I also tested a BigTIFF below 4GB as input and the output file of WhiteboxTools returned a normal TIFF (also below 4GB) without any shift. So it seems to read BigTIFF just fine. |
|
To make sure, I tried My workaround for now is to use DEP as input/output and then at the end I convert the files I need to FLT and then to GeoTIFF (BigTIFF) with |
When using either BreachDepressions or D8FlowAccumulation on a large raster exceeding 500 000 000 pixels, the resulting raster will be shifted one pixel to the right.
The text was updated successfully, but these errors were encountered: