[Bug] r.watershed fails on WIN10 with g_spawn: unable to execute command #1579

wjcper2008 · 2021-05-16T03:13:37Z

I try to run the r.watershed in GRASS 7.8.5 or 7.9, but get 'g_spawn: unable to execute command.'

windows 10 2004 version.

I cannot deal with this error. Why cause this issues? Can anyone help me? Thanks a lot!!

r.watershed --o --q -abs elevation=DEM@PERMANENT hbasin=slu_r thresh=1169

WARNING: G_spawn: unable to execute command
WARNING: Subprocess failed with exit code -1073741511

Traceback (most recent call last):
  File "G:\grassData\GrassPy\r.slopeunits.py", line 326, in <module>
    main()
  File "G:\grassData\GrassPy\r.slopeunits.py", line 196, in main
    grass.run_command('r.watershed', elevation=dem, hbasin='slu_r', thresh=thc, flags='abs', **mode)
  File "D:\OSGEO4\apps\grass\grass79\etc\python\grass\script\core.py", line 545, in run_command
    return handle_errors(returncode, result=None, args=args,kwargs=kwargs)
  File "D:\OSGEO4\apps\grass\grass79\etc\python\grass\script\core.py", line 428, in handle_errors
    raise CalledModuleError(module=module, code=code,returncode=returncode)
grass.exceptions.CalledModuleError: Module run `r.watershed --o --q -abs elevation=DEM@PERMANENT hbasin=slu_r thresh=1169` ended with an error.
The subprocess ended with a non-zero return code:
3221225785. See errors above the traceback or in the error output.

System Info:
GRASS version: 7.9.dev
Code revision: f9ed0ac
Build date: 2021-05-13
Build platform: x86_64-w64-mingw32
GDAL: 3.1.4
PROJ: 6.3.2
GEOS: 3.8.1
SQLite: 3.29.0
Python: 3.7.0
wxPython: 4.0.7
Platform: Windows-10-10.0.19041-SP0 (OSGeo4W)

The text was updated successfully, but these errors were encountered:

hellik · 2021-05-16T04:18:23Z

What is the output of g.region -p?

wjcper2008 · 2021-05-16T04:22:22Z

What is the output of g.region -p?

g.region -p
projection: 3
zone: 0
datum: wgs84
ellipsoid: wgs84
north: 30:31:17.732976N
south: 29:23:41.689821N
west: 117:11:52.591641E
east: 118:53:57.596247E
nsres: 0:00:00.998534
ewres: 0:00:00.998534
rows: 4062
cols: 6134
cells: 24916308
(Sun May 16 12:21:41 2021)

wjcper2008 · 2021-05-16T04:29:19Z

What is the output of g.region -p?

Thanks!

I search ''g_spawn: unable to execute command.'' in Google, but nobody encounters this issues. Sadly:(

I install GRASS 7.9 (ver 125) via OSGEO.

GRASS 7.8.5 is still unworkable for r.watershed.

Additionally, I also install the anaconda.

hellik · 2021-05-16T08:07:47Z

r.watershed --o --q -abs elevation=DEM@PERMANENT hbasin=slu_r thresh=1169

does this command work for you outside of a python script with the computational region you've posted above?

wjcper2008 · 2021-05-16T09:57:37Z

r.watershed --o --q -abs elevation=DEM@PERMANENT hbasin=slu_r thresh=1169

does this command work for you outside of a python script with the computational region you've posted above?

I run it directly in GRASS's console: r.watershed --o --q -abs elevation=DEM@PERMANENT hbasin=slu_r thresh=1169.

Still the same error.

Thus, I think there may be some bug in spawn progress. But I don't know how to settle it.

How do I output more detail error log information?

wjcper2008 · 2021-05-16T14:07:20Z

r.watershed --o --q -abs elevation=DEM@PERMANENT hbasin=slu_r thresh=1169

does this command work for you outside of a python script with the computational region you've posted above?

I think there may be some matter (confliction) about 'spawn' in GIS LIB. I just install from OSGEO

hellik · 2021-05-16T17:09:38Z

How do I output more detail error log information?

try following steps

set your computational region by g.region
set debug level by: g.gisenv set=DEBUG=3
r.watershed --o --q -abs elevation=DEM@PERMANENT hbasin=slu_r thresh=1169
post the debug output from the r.watershed command here
set debug level to 0 again: g.gisenv set=DEBUG=0

wjcper2008 · 2021-05-17T07:19:49Z

How do I output more detail error log information?

try following steps

set your computational region by g.region

set debug level by: g.gisenv set=DEBUG=3

r.watershed --o --q -abs elevation=DEM@PERMANENT hbasin=slu_r thresh=1169

post the debug output from the r.watershed command here

set debug level to 0 again: g.gisenv set=DEBUG=0

Thanks for your help! Here is the debug information about r.watershed. But the debug information is little about win_spawn.

Does something wrong about win_spawn?

Need locate virtual memory in My win10? I have 32G DDR4 RAM.

g.region raster=DEM@PERMANENT
g.gisenv set='DEBUG=3'
r.watershed --o --q -abs elevation=DEM@PERMANENT hbasin=slu_r thresh=1169

D1/3: G_set_program_name(): r.watershed
D2/3: G_file_name(): path = G:\grassData\HuangShan\PERMANENT
D1/3: Mode: All in RAM
D3/3: win_spawn: args = C:\WINDOWS\system32\cmd.exe /c
"D:\OSGEO4~1\apps\grass\grass79/etc/r.watershed/ram -s -a -b
elevation=DEM@PERMANENT threshold=1169 half_basin=slu_r
convergence=5"
WARNING: G_spawn: unable to execute command
WARNING: Subprocess failed with exit code -1073741511
D2/3: G_file_name(): path = G:\grassData\HuangShan\PERMANENT\cats\slu_r
D3/3: G_recreate_command()
D2/3: G_file_name(): path =
G:\grassData\HuangShan\PERMANENT\hist\slu_r
D2/3: G_file_name(): path = G:\grassData\HuangShan\PERMANENT
D2/3:   file open: new (mode = w)

hellik · 2021-05-17T16:29:55Z

It looks like an integer/buffer overflow.

Could you share the DEM?

tmszi · 2021-05-17T19:39:03Z

Could @wjcper2008 try derive basins in the basic NC location please?

Download basic NC location

Switch to basic NC location PERMANENT mapset
Execute r.watershed -abs elevation=elevation@PERMANENT threshold=1000 hbasin=hbasin

tmszi · 2021-05-17T19:44:00Z

Could @wjcper2008 give DEM@PERMANENT metadata info (execute command r.info -g DEM@PERMANENT) please?
What kind of OSGeo4W installer do you use 32 bit (osgeo4w-setup-x86.exe) or 64 bit (osgeo4w-setup-x86_64.exe) please?

wjcper2008 · 2021-05-18T00:59:20Z

Could @wjcper2008 give DEM@PERMANENT metadata info (execute command r.info -g DEM@PERMANENT) please?
What kind of OSGeo4W installer do you use 32 bit (osgeo4w-setup-x86.exe) or 64 bit (osgeo4w-setup-x86_64.exe) please?

This is my DEM file https://github.com/wjcper2008/FileN/releases/download/1.0/DEM.zip

Thanks for your kindly help! @tmszi @hellik

I install osgeo4w-setup-x86_64.exe with the latest version GRASS 7.9 dev.

Also try in GRASS 7.8.5 and QGIS with r.watershed. BUT it still doesn't work.

I try the basic NC location, still get the same issues.

Thus, I think there may be some confliction about spawn in My GRASS. Not the problem with the DEM.tif file.

r.info -g DEM@PERMANENT                                                         
north=30.5215924933333
south=29.3949138391667
east=118.899332290833
west=117.1979421225
nsres=0.000277370422000657
ewres=0.000277370421965004
rows=4062
cols=6134
cells=24916308
datatype=FCELL
ncats=0

tmszi · 2021-05-18T05:02:57Z

I was successfull with deriving basin (NC location) with GRASS GIS 7.8.5 (OSGeo4W installer, express GRASS GIS install only), and GRASS GIS 7.9 (Standalone installer) without this error.

Could @wjcper2008 completely uninstall GRASS GIS (OSGeo4W installer), and try install GRASS GIS again please (if you haven't tried it yet)?

wjcper2008 · 2021-05-18T05:16:58Z

I was successfull with deriving basin (NC location) with GRASS GIS 7.8.5 (OSGeo4W installer, express GRASS GIS install only), and GRASS GIS 7.9 (Standalone installer) without this error.

Could @wjcper2008 completely uninstall GRASS GIS (OSGeo4W installer), and try install GRASS GIS again please (if you haven't tried it yet)?

Thanks!
I try install GRASS 7.8.5 and 7.9 with OSGeo4W, standalone, QGIS...

BUT none of them work with r.watershed.

I don't know what reason causes G_spawn: unable to execute command

hellik · 2021-05-18T16:36:48Z

This is my DEM file https://github.com/wjcper2008/FileN/releases/download/1.0/DEM.zip

tested here with this DEM:

r.external input=D:\wd\testrwatershd\DEM.tif output=DEM                         
Reading band 1 of 1...
Link to raster map <DEM> created.

g.region -p -a raster=DEM@data align=DEM@data                                   
projection: 3 (Längen- und Breitengrad)
zone:       0
datum:      wgs84
ellipsoid:  wgs84
north:      30:31:17.732976N
south:      29:23:41.689821N
west:       117:11:52.591641E
east:       118:53:57.596247E
nsres:      0:00:00.998534
ewres:      0:00:00.998534
rows:       4062
cols:       6134
cells:      24916308

r.colors map=DEM@data color=elevation                                           
Color table for raster map <DEM@data> set to 'elevation'

r.watershed --o -abs elevation=DEM hbasin=slu_r thresh=1169                     
ABSCHNITT 1a (von 5): Initiiere den Speicher.
ABSCHNITT 1b (of 5): Ermittle den Fluß aus der Karte heraus.
SECTION 2: A* Search.
ABSCHNITT 3: Akkumuliere Oberflächenabfluß mit SFD.
ABSCHNITT 4: Bestimme Wassereinzugsgebiet.
ABSCHNITT 5: Schließe Karten.

the DEM works here.

hellik · 2021-05-18T16:39:42Z

g.region raster=DEM@PERMANENT
g.gisenv set='DEBUG=3'
r.watershed --o -abs elevation=DEM@PERMANENT hbasin=slu_r thresh=1169

please run it in DEBUG=3 mode again, but without the quiet flag --q

please post then the whole debug output here.

wjcper2008 · 2021-05-19T01:15:13Z

g.region raster=DEM@PERMANENT
g.gisenv set='DEBUG=3'
r.watershed --o -abs elevation=DEM@PERMANENT hbasin=slu_r thresh=1169
please run it in DEBUG=3 mode again, but without the quiet flag --q

please post then the whole debug output here.

Thanks @hellik

The same as before.... win_spawn does not work.

There may be some confiction for win_spawn. :(

(Wed May 19 09:13:07 2021)                                                      
g.region raster=DEM@PERMANENT                                                   
(Wed May 19 09:13:08 2021) (0 秒)                                           
(Wed May 19 09:13:14 2021)                                                      
g.gisenv set=DEBUG=3                                                            
(Wed May 19 09:13:14 2021)  (0 秒)                                           
(Wed May 19 09:13:23 2021)                                                      
r.watershed --o -abs elevation=DEM@PERMANENT hbasin=slu_r thresh=1169           
D1/3: G_set_program_name(): r.watershed
D2/3: G_file_name(): path = G:\grassData\HuangShan\PERMANENT
D1/3: Mode: All in RAM
D3/3: win_spawn: args = C:\WINDOWS\system32\cmd.exe /c
"D:\OSGEO4~1\apps\grass\grass79/etc/r.watershed/ram -s -a -b
elevation=DEM@PERMANENT threshold=1169 half_basin=slu_r
convergence=5"
WARNING: G_spawn: unable to execute command
WARNING: Subprocess failed with exit code -1073741511
D2/3: G_file_name(): path =
G:\grassData\HuangShan\PERMANENT\cats\slu_r
D3/3: G_recreate_command()
D2/3: G_file_name(): path =
G:\grassData\HuangShan\PERMANENT\hist\slu_r
D2/3: G_file_name(): path = G:\grassData\HuangShan\PERMANENT
D2/3:   file open: new (mode = w)
(Wed May 19 09:13:24 2021)  (1 秒)

nilason · 2021-05-19T17:23:33Z

@wjcper2008 Will adding the flag -m (disk swap memory option) to the r.watershed command make any difference?

hellik · 2021-05-19T18:26:07Z

-m flag is worth to be tested

@wjcper2008 when you're starting winGRASS, then please type

echo %PATH%

and

echo %GISBASE%

into the black GRASS GIS windows console and post here the output

wjcper2008 · 2021-05-21T14:46:20Z

echo %GISBASE%

Thanks @hellik @nilason
-m is still not work for r.watershed as Beautify flat areas is not yet supported for disk swap mode

(Fri May 21 22:42:41 2021)                                                      
echo %PATH%                                                                     
D:\OSGEO4~1\apps\grass\grass79\gui\scripts;
D:\OSGEO4~1\apps\grass\grass79\lib;
D:\OSGEO4~1\apps\grass\grass79\bin;
C:\Users\Chen\AppData\Roaming\GRASS7\addons\bin;
D:\OSGEO4~1\apps\Python37;
D:\OSGEO4~1\apps\Python37\Scripts;
D:\OSGEO4~1\apps\Python27\Scripts;
D:\OSGEO4~1\bin;
C:\WINDOWS\system32;C:\WINDOWS;
C:\WINDOWS\system32\WBem;
D:\OSGEO4~1\apps\msys\bin;
D:\OSGEO4~1\apps\Python37\lib\site-packages\pywin32_system32;
D:\OSGEO4~1\apps\Python37\lib\site-packages\pywin32_system32;
D:\OSGEO4~1\apps\Python37\lib\site-packages\numpy\.libs
(Fri May 21 22:42:41 2021)  (0 秒)                                           
(Fri May 21 22:42:59 2021)                                                      
echo %GISBASE%                                                                  
D:\OSGEO4~1\apps\grass\grass79
(Fri May 21 22:43:00 2021) (0 秒)                                           
(Fri May 21 22:43:25 2021)                                                      
r.watershed --o -absm elevation=DEM@PERMANENT hbasin=slu_r thresh=1169          
Beautify flat areas is not yet supported for disk swap mode
WARNING: G_spawn: unable to execute command
WARNING: Subprocess failed with exit code -1073741511
(Fri May 21 22:43:26 2021)  (1 秒)

hellik · 2021-05-21T19:41:25Z

here:

C:\OSGEO4~1\apps\grass\grass79\lib;
C:\OSGEO4~1\apps\grass\grass79\bin;
C:\Users\youruser\AppData\Roaming\GRASS7\addons\bin;
C:\OSGEO4~1\apps\Python37;
C:\OSGEO4~1\apps\Python37\Scripts;{app};
C:\OSGEO4~1\apps\Python27\Scripts;
C:\OSGEO4~1\bin;
C:\WINDOWS\system32;
C:\WINDOWS;
C:\WINDOWS\system32\WBem;
C:\OSGEO4~1\apps\msys\bin;
C:\Program Files\R\R-4.0.3\bin\x64;
C:\OSGEO4~1\apps\Python37\lib\site-packages\pywin32_system32

have you modified %PATH% ?

hellik · 2021-05-21T19:51:20Z

are any other r*.modules working for you?

e.g.

r.slope.aspect
r.stream.extract
r.topidx
r.param.scale
r.flow

wjcper2008 · 2021-05-22T06:34:59Z

are any other r*.modules working for you?

e.g.

r.slope.aspect

r.stream.extract

r.topidx

r.param.scale

r.flow

Thanks @hellik

I check these commands WORK OK!
r.slope.aspect
r.stream.extract
r.topidx

r.flow need the flow accumulation from r.watershed

wjcper2008 · 2021-05-22T06:36:39Z

here:

C:\OSGEO4~1\apps\grass\grass79\lib;
C:\OSGEO4~1\apps\grass\grass79\bin;
C:\Users\youruser\AppData\Roaming\GRASS7\addons\bin;
C:\OSGEO4~1\apps\Python37;
C:\OSGEO4~1\apps\Python37\Scripts;{app};
C:\OSGEO4~1\apps\Python27\Scripts;
C:\OSGEO4~1\bin;
C:\WINDOWS\system32;
C:\WINDOWS;
C:\WINDOWS\system32\WBem;
C:\OSGEO4~1\apps\msys\bin;
C:\Program Files\R\R-4.0.3\bin\x64;
C:\OSGEO4~1\apps\Python37\lib\site-packages\pywin32_system32

have you modified %PATH% ?

The PATH used is in default. I have not modified them any more.
Are there some problems about the PATH?

TillF · 2024-01-11T20:34:53Z

@wjcper2008 Did you find a solution? I have the same problem, switch -m didn't solve the issue.

neteler · 2024-01-11T21:03:20Z

I have the same problem, switch -m didn't solve the issue.

Which GRASS GIS version do you use (on which operating system)?

TillF · 2024-01-12T15:34:21Z

Tested in GRASS 8.2.1 and 8.3.1-1, both installed from standalone-installer on Windows 10 Pro.
However, I have successfully run r.watershed with these GRASS versions on the same data and region with different settings.
I will try to figure out what triggers the different behaviour..

TillF · 2024-01-15T13:19:02Z

I tried to narrow down the error to the minimum reproducable example:
Please use this DEM :
Download

Create location and import DEM:
r.in.gdal input=dem_part.tif output=dem -e location=mynewlocation

After changing to the new loaction, the error occurs when running the following command:
r.watershed elevation=dem drainage=flowdir threshold=200 half_basin=half_basin -s --overwrite
It returns

ABSCHNITT 1a (von 5): Initiiere den Speicher.
ABSCHNITT 1b (of 5): Ermittle den Fluß aus der Karte heraus.
 100%
SECTION 2: A* Search.
ABSCHNITT 3: Akkumuliere Oberflächenabfluß mit SFD.
ABSCHNITT 4: Bestimme Wassereinzugsgebiet.
WARNUNG: G_spawn: unable to execute command
WARNUNG: Unterprozess fehlgeschlagen mit dem Fehlercode -1073741571
WARNUNG: Kategorie-Informationen für [half_basin] in [PERMANENT] fehlen
         oder sind ungültig.

No maps are produced.

The error does not occur when running the command without the 'threshold' argument:
r.watershed elevation=dem drainage=flowdir -s --overwrite
or when the extent of the DEM is reduced significantly at any of its margins.

Tested with GRASS 8.3.1-1, standalone installer, Windows 10 Pro

neteler · 2024-01-15T14:52:21Z

Thanks for providing this example (I just tested this successfully on Linux).
It runs on 8965 rows and 5836 columns = 52,319,740 cells which should not be too large.

I found that this exit code -1073741571 (0xC00000FD) suggests that a stackoverflow has occurred on your Windows system. Maybe e.g. @hellik can test on his Windows box?

hellik · 2024-01-15T15:28:40Z

I confirm this issue with the Data set mentioned above on a win 10 Enterprise Box and the -s option. It works without -s.

IIRC, there may be a related ticket in track.

hellik · 2024-01-15T16:05:09Z

Thanks for providing this example (I just tested this successfully on Linux). It runs on 8965 rows and 5836 columns = 52,319,740 cells which should not be too large.

I found that this exit code -1073741571 (0xC00000FD) suggests that a stackoverflow has occurred on your Windows system. Maybe e.g. @hellik can test on his Windows box?

Yes, it looks like a buffer overflow.

…79 (comment)): now using MFD for constructing EHAs, which looked quite similar ENH: - replaced tryCatch with onError-cleanup function - checked for availability of addons - implemented m-flag for diskswap-option for large grids

hellik · 2024-01-15T20:16:33Z

I confirm this issue with the Data set mentioned above on a win 10 Enterprise Box and the -s option. It works without -s.

IIRC, there may be a related ticket in track.

MFD mode is working

r.watershed elevation=dem@test threshold=200 drainage=flowdir half_basin=half_basin
SECTION 1a (of 5): Initiating Memory.
SECTION 1b (of 5): Determining Offmap Flow.
SECTION 2: A* Search.
SECTION 3a: Accumulating Surface Flow with MFD.
SECTION 3b: Adjusting drainage directions.
SECTION 4: Watershed determination.
SECTION 5: Closing Maps.

hellik · 2024-01-15T20:25:02Z

screenshot is r.watershed MFD mode.

could it be that the DEM has some artefacts for SFD mode not working?

TillF · 2024-01-16T08:46:38Z

The DEM originates from https://www.hydrosheds.org/products/hydrosheds, which is "A void-filled digital elevation model" "derived primarily from the SRTM".
As for artefacts, I don't know which would break the SFD-algorithm.

TillF · 2024-01-16T10:02:20Z

The DEM I provided is just the minimum example to reproduce the error with SFD. As mentioned in your replies, it does not cause the error when using MFD.
However, my original extent is larger and shows the error also when using MFD. If required, I can also provide this full extent DEM (Rows: 29586, Columns: 25868, Total Cells: 765330648).

petrasovaa · 2024-01-26T21:19:43Z

I ran the example on Linux with valgrind, but it didn't detect any errors...

wjcper2008 added the bug Something isn't working label May 16, 2021

wjcper2008 closed this as completed May 16, 2021

wjcper2008 reopened this May 16, 2021

veroandreo changed the title ~~r.watershed fails on WIN10 with g_spawn: unable to execute command~~ [Bug] r.watershed fails on WIN10 with g_spawn: unable to execute command May 16, 2021

tmszi added the Windows Microsoft Windows specific label May 18, 2021

wjcper2008 closed this as completed Jul 31, 2021

wjcper2008 reopened this Jul 31, 2021

[Bug] r.watershed fails on WIN10 with g_spawn: unable to execute command #1579

[Bug] r.watershed fails on WIN10 with g_spawn: unable to execute command #1579

Comments

wjcper2008 commented May 16, 2021 • edited by veroandreo

hellik commented May 16, 2021

wjcper2008 commented May 16, 2021

wjcper2008 commented May 16, 2021 • edited by veroandreo

hellik commented May 16, 2021

wjcper2008 commented May 16, 2021 • edited

wjcper2008 commented May 16, 2021 • edited

hellik commented May 16, 2021

wjcper2008 commented May 17, 2021 • edited

hellik commented May 17, 2021

tmszi commented May 17, 2021 • edited

tmszi commented May 17, 2021 • edited

wjcper2008 commented May 18, 2021 • edited

tmszi commented May 18, 2021 • edited

wjcper2008 commented May 18, 2021

hellik commented May 18, 2021

hellik commented May 18, 2021

wjcper2008 commented May 19, 2021 • edited

nilason commented May 19, 2021

hellik commented May 19, 2021 • edited

wjcper2008 commented May 21, 2021 • edited

hellik commented May 21, 2021

hellik commented May 21, 2021

wjcper2008 commented May 22, 2021

wjcper2008 commented May 22, 2021 • edited

TillF commented Jan 11, 2024

neteler commented Jan 11, 2024

TillF commented Jan 12, 2024 • edited

TillF commented Jan 15, 2024 • edited

neteler commented Jan 15, 2024

hellik commented Jan 15, 2024

hellik commented Jan 15, 2024

hellik commented Jan 15, 2024

hellik commented Jan 15, 2024

TillF commented Jan 16, 2024

TillF commented Jan 16, 2024

petrasovaa commented Jan 26, 2024

wjcper2008 commented May 16, 2021 •

edited by veroandreo

wjcper2008 commented May 16, 2021 •

edited by veroandreo

wjcper2008 commented May 16, 2021 •

edited

wjcper2008 commented May 16, 2021 •

edited

wjcper2008 commented May 17, 2021 •

edited

tmszi commented May 17, 2021 •

edited

tmszi commented May 17, 2021 •

edited

wjcper2008 commented May 18, 2021 •

edited

tmszi commented May 18, 2021 •

edited

wjcper2008 commented May 19, 2021 •

edited

hellik commented May 19, 2021 •

edited

wjcper2008 commented May 21, 2021 •

edited

wjcper2008 commented May 22, 2021 •

edited

TillF commented Jan 12, 2024 •

edited

TillF commented Jan 15, 2024 •

edited