Both IDW interpolation and clip raster with mask indicate 100% complete, but hangs forever

[qgib]

Author Name: Tony Walters (@tonywsea)
Original Redmine Issue: 21660
Affected QGIS version: 3.6.0
Redmine category:processing/qgis

---

By clicking "cancel" many times eventually it will abort.
Clean install of 3.6 on new Intel 8700 6 core system.
Multi-core (6) rendering enabled.

link to geopackage: https://we.tl/t-D6RQZztc9k

[qgib]

Author Name: Tony Walters (@tonywsea)

---

Went back to a laptop with 3.4 Madeira and IDW interpolation works correctly. Also ran 3.6 on the laptop and failed there as well.

Status bar jumps immediately to 99% finished, but hangs there. Normally it shows the processing progression from 1% onwards.

Cheers,
T

[qgib]

Author Name: Giovanni Manghi (@gioman)

---

I tried on both Windows and Linxu on 3.6 and can't replicate (but maybe depends on what column is used to interpolate? what do you used?).

Also where is the data to test the raster clip?

---

• status_id was changed from Open to Feedback

[qgib]

Author Name: Tony Walters (@tonywsea)

---

Hi Giovanni,

I almost feel embarrassed like I'm reporting errors that are unique to my install.

Here is a typical log report:

---

Processing algorithm…
Algorithm 'IDW interpolation' starting…
Input parameters:
{ 'DISTANCE_COEFFICIENT' : 10, 'EXTENT' : '377194.8269227125,423458.1550825302,4893116.891602981,4926995.061894643 [EPSG:2961]', 'INTERPOLATION_DATA' : 'F:\GIS\South Shore\SouthShoreModel\Analysis\IPT_Feb_28\IPT_MODC_unique.shp::::0::::2::~::0::|:', 'OUTPUT' : 'C:/Users/tonwa/AppData/Local/Temp/processing_011cc56ee4374c2a9fa07ac9ac6993c7/8a3fb811d84e41e39011912aca925068/OUTPUT.tif', 'PIXEL_SIZE' : 0.1 }

Execution failed after 21.03 seconds

Loading resulting layers
Algorithm 'IDW interpolation' finished

I have tried using layer extent and also canvas extent with no success. The reason it fails in this case in 21.03 seconds is I click repeatedly on Cancel and it eventually finishes. If I didn't cancel it, it runs all night/forever. Example of attribute file is attached. Interpolation attribute is 'Download_S".

When I selected "use layer extent" this morning, the following Python error:

2019-03-27T10:17:53 WARNING Traceback (most recent call last):
File "C:/PROGRA1/QGIS31.4/apps/qgis-ltr/./python/plugins\processing\gui\ExtentSelectionPanel.py", line 167, in useLayerExtent
dlg = LayerSelectionDialog(self)
File "C:/PROGRA1/QGIS31.4/apps/qgis-ltr/./python/plugins\processing\gui\ExtentSelectionPanel.py", line 72, in init
QgsMapLayerProxyModel.HasGeometry | QgsMapLayerProxyModel.RasterLayer | QgsMapLayerProxyModel.MeshLayer)
AttributeError: type object 'QgsMapLayerProxyModel' has no attribute 'MeshLayer'

This all used to work fine originally.

I did a clean install of 3.4 LTR as well and this is where the above data is coming from.

shapefile link https://we.tl/t-paTmZGMOAl

I also ran Grass r.surf.idw on the same shapefile. It did complete successfully with log as follows:

Processing algorithm…
Algorithm 'v.surf.idw' starting…
Input parameters:
{ '-n' : False, 'GRASS_MIN_AREA_PARAMETER' : 0.0001, 'GRASS_RASTER_FORMAT_META' : '', 'GRASS_RASTER_FORMAT_OPT' : '', 'GRASS_REGION_CELLSIZE_PARAMETER' : 0, 'GRASS_REGION_PARAMETER' : None, 'GRASS_SNAP_TOLERANCE_PARAMETER' : -1, 'column' : 'DOWNLOAD_S', 'input' : 'C:\Users\tonwa\Documents\GIS Projects\Common\NS\IPT_MODC_unique_Feb28.shp|layername=IPT_MODC_unique_Feb28', 'npoints' : 12, 'output' : 'C:/Users/tonwa/AppData/Local/Temp/processing_011cc56ee4374c2a9fa07ac9ac6993c7/087bbb358e62426d9473024b209d1629/output.tif', 'power' : 2 }

g.proj -c proj4="+proj=utm +zone=20 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs"
v.in.ogr min_area=0.0001 snap=-1.0 input="C:\Users\tonwa\Documents\GIS Projects\Common\NS\IPT_MODC_unique_Feb28.shp" layer="IPT_MODC_unique_Feb28" output="vector_5c9b7ad2a60e33" --overwrite -o
g.region n=4966726.636450228 s=4924757.6693206765 e=419913.4332330831 w=369936.10964671226 res=100.0
v.surf.idw input=vector_5c9b7ad2a60e33 npoints=12 power=2 column=DOWNLOAD_S output=output11521d64d6944babac5fbd964e884062 --overwrite
g.region raster=output11521d64d6944babac5fbd964e884062
r.out.gdal -t -m input="output11521d64d6944babac5fbd964e884062" output="C:\Users\tonwa\AppData\Local\Temp\processing_011cc56ee4374c2a9fa07ac9ac6993c7\087bbb358e62426d9473024b209d1629\output.tif" format="GTiff" createopt="TFW=YES,COMPRESS=LZW" --overwrite
Starting GRASS GIS...
WARNING: Concurrent mapset locking is not supported on Windows
Executing <C:\Users\tonwa\AppData\Local\Temp\processing_011cc56ee4374c2a9fa07ac9ac6993c7\grassdata\grass_batch_job.cmd> ...
C:\PROGRA1\QGIS31.4\bin>chcp 1252 1>NUL
C:\PROGRA1\QGIS31.4\bin>g.proj -c proj4="+proj=utm +zone=20 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs"
Default region was updated to the new projection, but if you have multiple mapsets g.region -d should be run in each to update the region from the default
Projection information updated
C:\PROGRA1\QGIS31.4\bin>v.in.ogr min_area=0.0001 snap=-1.0 input="C:\Users\tonwa\Documents\GIS Projects\Common\NS\IPT_MODC_unique_Feb28.shp" layer="IPT_MODC_unique_Feb28" output="vector_5c9b7ad2a60e33" --overwrite -o
Over-riding projection check
Check if OGR layer <IPT_MODC_unique_Feb28> contains polygons...
0..2..4..6..8..10..12..14..16..18..20..22..24..26..28..30..32..34..36..38..40..42..44..46..48..50..52..54..56..58..60..62..64..66..68..70..72..74..76..78..80..82..84..86..88..90..92..94..96..98..100
Creating attribute table for layer <IPT_MODC_unique_Feb28>...
Column name <WEB100_S99> renamed to <WEB100_S99_1>
Column name <WEB100_S99> renamed to <WEB100_S99_2>
Column name <WEB100_S99> renamed to <WEB100_S99_3>
Importing 909 features (OGR layer <IPT_MODC_unique_Feb28>)...
0..2..4..6..8..10..12..14..16..18..20..22..24..26..28..30..32..34..36..38..40..42..44..46..48..50..52..54..56..58..60..62..64..66..68..70..72..74..76..78..80..82..84..86..88..90..92..94..96..98..100

Building topology for vector map vector_5c9b7ad2a60e33@PERMANENT...
Registering primitives...
C:\PROGRA1\QGIS31.4\bin>g.region n=4966726.636450228 s=4924757.6693206765 e=419913.4332330831 w=369936.10964671226 res=100.0
C:\PROGRA1\QGIS31.4\bin>v.surf.idw input=vector_5c9b7ad2a60e33 npoints=12 power=2 column=DOWNLOAD_S output=output11521d64d6944babac5fbd964e884062 --overwrite
907 points loaded
Interpolating raster map (420 rows, 500 columns)...
0..2..4..6..8..10..12..14..16..18..20..22..24..26..28..30..32..34..36..38..40..42..44..46..48..50..52..54..56..58..60..62..64..66..68..70..72..74..76..78..80..82..84..86..88..90..92..94..96..98..100
v.surf.idw complete.
C:\PROGRA1\QGIS31.4\bin>g.region raster=output11521d64d6944babac5fbd964e884062
C:\PROGRA1\QGIS31.4\bin>r.out.gdal -t -m input="output11521d64d6944babac5fbd964e884062" output="C:\Users\tonwa\AppData\Local\Temp\processing_011cc56ee4374c2a9fa07ac9ac6993c7\087bbb358e62426d9473024b209d1629\output.tif" format="GTiff" createopt="TFW=YES,COMPRESS=LZW" --overwrite
ERROR 6: SetColorTable() only supported for Byte or UInt16 bands in TIFF format.
Checking GDAL data type and nodata value...
2..5..8..11..14..17..20..23..26..29..32..35..38..41..44..47..50..53..56..59..62..65..68..71..74..77..80..83..86..89..92..95..98..100
Using GDAL data type
Exporting raster data to GTiff format...
WARNING: Too many values, color table cut to 65535 entries
2..5..8..11..14..17..20..23..26..29..32..35..38..41..44..47..50..53..56..59..62..65..68..71..74..77..80..83..86..89..92..95..98..100
r.out.gdal complete. File <C:\Users\tonwa\AppData\Local\Temp\processing_011cc56ee4374c2a9fa07ac9ac6993c7\087bbb358e62426d9473024b209d1629\output.tif> created.
C:\PROGRA1\QGIS31.4\bin>exit
Execution of <C:\Users\tonwa\AppData\Local\Temp\processing_011cc56ee4374c2a9fa07ac9ac6993c7\grassdata\grass_batch_job.cmd> finished.
Cleaning up default sqlite database ...
Cleaning up temporary files...
Execution completed in 20.67 seconds
Results:
{'output': <QgsProcessingOutputLayerDefinition {'sink':C:/Users/tonwa/AppData/Local/Temp/processing_011cc56ee4374c2a9fa07ac9ac6993c7/087bbb358e62426d9473024b209d1629/output.tif, 'createOptions': {'fileEncoding': 'System'}}>}

Loading resulting layers
Algorithm 'v.surf.idw' finished

Even this algorithm seems to have a problem with selecting "use layer extent", but is ok with "select minimum extent from all layers"

The issue with clip raster with vector layer may have been mis-reported. I started another of this task early this week and it did complete successfully, although took 2 full days of compute on a very fast Intel processor.

[qgib]

Author Name: Giovanni Manghi (@gioman)

---

Tony Walters wrote:

Hi Giovanni,

I almost feel embarrassed like I'm reporting errors that are unique to my install.

it is quite normal that there could be local issues. This does not means is the case, but the fact I can't replicate (also on the same OS) is usually a strong evidence,

I did a clean install of 3.4 LTR as well and this is where the above data is coming from.

the pyhton error was already fixed, you will not see anymore in the next point release for 3.4

Anyway... in the original description you state that you were seeing the problems in 3.6 and not 3.4.5... now you say another thing. You see the issues in both?

shapefile link https://we.tl/t-paTmZGMOAl

do you see the issues with the shape, the gpkg or both?
What is the field to be used to interpolate?

The issue with clip raster with vector layer may have been mis-reported. I started another of this task early this week and it did complete successfully, although took 2 full days of compute on a very fast Intel processor.

it depends on the datasets you used, can you link them?

[qgib]

Author Name: Alexander Bruy (@alexbruy)

---

Please provide test data, link from the bugreport expired.

---

• category_id was changed from Processing/Core to Processing/QGIS

[alexbruy]

@tonywsea can you check with master and if issue still exists provide test data and steps to reproduce?

[alexbruy]

Works for me with master/3.4/3.8.