[Processing] gdal gives no output in standalone script

[jfbourdon]

Describe the bug
I'm unable to use any GDAL algorithm in a standalone script (QGIS 3.12.1 on Windows 10). All processing.run(native: ) and processing.run(saga: ) algorithms work without any problem, but when using a processing.run(gdal: ) algorithms, I only get an empty result (no error). However, the same processing(gdal: ) algorithm works when I'm using the python console within QGIS.

Thinking that it may be related to environment variables, I used os.environ inside QGIS' python console to see what variables were loaded, then loaded the exact same ones in my standalone script but it didn't work. I asked for help on SO some time ago got no answer.

How to Reproduce
Create a py file, change the inpath and outpath variables and run it with QGIS' python (note that the paths are with an OSGeo4W64 installation at C:/OSGeo4W64):

import os

os.environ['PATH'] = ";".join([
    "C:/OSGeo4W64/apps/qgis/bin",
    "C:/OSGeo4W64/apps/Python37",
    "C:/OSGeo4W64/apps/Python37/Scripts",
    "C:/OSGeo4W64/apps/qt5/bin",
    "C:/OSGeo4W64/apps/Python27/Scripts",
    "C:/OSGeo4W64/bin",
    "C:/WINDOWS/system32",
    "C:/WINDOWS",
    "C:/WINDOWS/system32/WBem",
    "C:/OSGeo4W64/apps/msys/bin",
    "C:/OSGeo4W64/apps/Python37/lib/site-packages/pywin32_system32",
    "C:/OSGeo4W64/apps/Python37/lib/site-packages/numpy/.libs",
    "C:/OSGeo4W64/apps/Python37/lib/site-packages/scipy/.libs"
])
os.environ['PROJ_LIB'] = 'C:/OSGeo4W64/share/proj'
os.environ['PYTHONHOME'] = 'C:/OSGeo4W64/apps/Python37'
os.environ['PYTHONPATH'] = 'C:/OSGeo4W64/apps/Python37;C:/OSGeo4W64/apps/Python37/Scripts'
os.environ['GDAL_DATA'] = 'C:/OSGeo4W64/share/gdal'
os.environ['GDAL_DRIVER_PATH'] = 'C:/OSGeo4W64/bin/gdalplugins'
os.environ['GDAL_FILENAME_IS_UTF8'] = 'YES'
os.environ['QGIS_PREFIX_PATH'] = 'C:/OSGeo4W64/apps/qgis'
os.environ['HOME']= 'C:/Users/bouj1a'
os.environ['GEOTIFF_CSV'] = 'C:/OSGeo4W64/share/epsg_csv'
os.environ['JPEGMEM']= '1000000'
os.environ['O4W_QT_BINARIES']= 'C:/OSGeo4W64/apps/Qt5/bin'
os.environ['O4W_QT_HEADERS']= 'C:/OSGeo4W64/apps/Qt5/include'
os.environ['O4W_QT_LIBRARIES']= 'C:/OSGeo4W64/apps/Qt5/lib'
os.environ['O4W_QT_PLUGINS']= 'C:/OSGeo4W64/apps/Qt5/plugins'
os.environ['O4W_QT_PREFIX']= 'C:/OSGeo4W64/apps/Qt5'
os.environ['O4W_QT_TRANSLATIONS']= 'C:/OSGeo4W64/apps/Qt5/translations'
os.environ['OSGEO4W_ROOT']= 'C:/OSGeo4W64'
os.environ['QT_PLUGIN_PATH']= 'C:/OSGeo4W64/apps/qgis/qtplugins;C:/OSGeo4W64/apps/qt5/plugins'
os.environ['VSI_CACHE']= 'TRUE'
os.environ['VSI_CACHE_SIZE']= '1000000'

from qgis.core import *
QgsApplication.setPrefixPath("C:/OSGeo4W64/bin", True)

import processing
from processing.core.Processing import Processing
from qgis.analysis import QgsNativeAlgorithms
from PyQt5.QtCore import *

qgs = QgsApplication([], False)
qgs.initQgis()
Processing.initialize()
QgsApplication.processingRegistry().addProvider(QgsNativeAlgorithms())

inpath = 'E:/raster.tif'
outpath = 'E:/polygon.shp'

params = {
    'INPUT':inpath,
    'BAND':1,
    'FIELD':'VALUE',
    'EIGHT_CONNECTEDNESS':False,
    'OUTPUT':outpath 
}
processing.run("gdal:polygonize", params)

# Result (but no file created)
#> {'OUTPUT': 'E:/polygon.shp'}

QGIS and OS versions
QGIS 3.12.2-București
QGIS code revision 8a1fb33
Compiled against Qt 5.11.2
Windows 10 (10.0)

[gioman]

@jfbourdon you should ask in the users or developers mailing list and check if your script is ok.

[earthcli]

@jfbourdon Have you solved this problem and could you share your experience ？
Thanks.

[jfbourdon]

I haven't solved this yet... I rather avoided the problem by only running my scripts with the GUI and/or by directly calling GDAL executables with subprocess.run(). I asked on the dev mailing list (message here) but got no answer unfortunately.

[github-actions]

The QGIS project highly values your report and would love to see it addressed. However, this issue has been left in feedback mode for the last 14 days and is being automatically marked as "stale".
If you would like to continue with this issue, please provide any missing information or answer any open questions. If you could resolve the issue yourself meanwhile, please leave a note for future readers with the same problem and close the issue.
In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this issue.
If there is no further activity on this issue, it will be closed in a week.

[jfbourdon]

As a follow-up, I'm still unable to call any processing.run(gdal: ) in a standalone script, however since 3.14, we can call directly processing tools, including processing plugins, in CLI with qgis_process which allow me to completely replace my standalone scripts and avoid any issue of the sort. So the original issue persists, but there is now a workaround which is even better.

[naf140230]

I have a similar problem. I do not want any workarounds. I want this issue to be solved. It is still occurring in QGIS 3.16 and QGIS 3.18

[gioman]

I have a similar problem. I do not want any workarounds. I want this issue to be solved. It is still occurring in QGIS 3.16 and QGIS 3.18

@naf140230 if you "want" a fix then look here https://qgis.org/en/site/forusers/commercial_support.html

[naf140230]

From a brief glance of the URL, I can see that there are many groups who contributed to QGIS. I will take a more thorough look soon. In the meantime, do you have any recommendations on which specific ones might have the solution I need or at least whom I could inform of the problem?

[TimWillaert]

I will take a more thorough look soon.

Have you gotten any further with this? I've got the same issue.

[naf140230]

I found a work-around that I am currently using.

[naf140230]

Right now, I would like to know a workaround for a SAGA tool that does not work with processing.run. This is a Raster Calculator tool that SAGA provides, and it does not work in standalone with processing.run, unless I am running it from the Python Console within QGIS.

[gioman]

Right now, I would like to know a workaround for a SAGA tool that does not work with processing.run. This is a Raster Calculator tool that SAGA provides, and it does not work in standalone with processing.run, unless I am running it from the Python Console within QGIS

@naf140230 completely unrelated to this ticket. Anyway, are you on macOS?

[naf140230]

No, I am on Windows, and I ended up solving that problem using a similar workaround to the gdal problem.

[wburt]

I have also encountered this issue. It seems to me that in this example the processing algorithm gdal:polygonize executes the command python3 -m gdal_polygonize myraster.tif mypolygon.shp -b 1 -f "ESRI Shapefile" mypolygon VALUE . Using your own feedback sublcass you can print the error (E:\sw_nt\QGIS_3.16\bin\python3.exe: No module named gdal_polygonize)

Getting beyond my skills but my guess the system can't find the python file gdal_polygonize.py in the path. This looks like it may have been fixed by detecting windows installs and executing the .bat to correctly configure the environment.

[mingatgithub]

Encountered this issue today, and accidently solved the problem now.
Solution:
Correctly define os.environ['OSGEO4W_ROOT']= 'path/to/qgis'

e.g., in my case, os.environ['OSGEO4W_ROOT'] =r"C:\Program Files\QGIS 3.34.0"

---

I randomly tested calling the command directly in Jupyter. I noticed the path to the .bat has not been cooreclty assigned.

!gdal_polygonize.bat D:/Project/test/grassdb/MASK.tif -b 1 -f "ESRI Shapefile" D:/Project/test/grassdb/MASK_t1.shp MASK_t1 DN

The system cannot find the path specified. python: can't open file 'd:\apps\Python39\Scripts\gdal_polygonize.py': [Errno 2] No such file or directory

Then I look up to the 'gdal_polygonize.bat', finding out it relies on %OSGEO4W_ROOT% to correctly call the .py
@echo off call "%OSGEO4W_ROOT%\bin\o4w_env.bat" python -u "%OSGEO4W_ROOT%\apps\Python39\Scripts\gdal_polygonize.py" %*

And then I realized I haven't defined os.environ['OSGEO4W_ROOT'] aside from the environment varials like PATH and PYTHONPATH.