Batch Processing Freezes

[Mikuohatsune53]

What is the bug or the crash?

When running a simple batch process with only 12 rows, it freezes up and does not continue it. Even after waiting for over an hour, it is still frozen. Even after clicking the cancel button, it freezes up unless I click entirely out of QGis.

Sadly no error codes, but the issue appears after running a batch process even on another computer.
My friend also tried the same process on his computer with a different set of temp data, and he also ran into the same issue.

Steps to reproduce the issue

Join Attributes by Location (summary) --> Batch Processing --> County Intersect TMax

Versions

The current released version of QGIS is 3.32.0.

Supported QGIS version

• I'm running a supported QGIS version according to the roadmap.

New profile

• I tried with a new QGIS profile

Additional context

No response

[Mikuohatsune53]

Version:

QGIS version
3.32.0-Lima
QGIS code revision
311a8cb
Qt version
5.15.3
Python version
3.9.5
GDAL/OGR version
3.7.0
PROJ version
9.2.1
EPSG Registry database version
v10.088 (2023-05-13)
GEOS version
3.11.2-CAPI-1.17.2
SQLite version
3.41.1
PDAL version
2.5.3
PostgreSQL client version
15.2
SpatiaLite version
5.0.1
QWT version
6.1.6
QScintilla2 version
2.13.1
OS version
Windows 10 Version 2009

Active Python plugins
quick_map_services
0.19.29
raster_cutter
0.6
db_manager
0.1.20
grassprovider
2.12.99
MetaSearch
0.3.6
otbprovider
2.12.99
processing
2.12.99

[roya0045]

As a workaround, you can try to close qgis, a popup will open, click yes to cancel pending tasks, qgis should remain open, hit run again and it may finish this time.

[Mides]

The same problem occurs in 3.28.9.

[wikirikitiki]

Unfortunately I have the same problem. QGIS 3.24 is the last version that has no problem with batch processing.
Everything after (3.26 up to 3.32) and the batch processing freezes without crashing.

I tried with with a new QGIS profile but without success. I also tried the same with 3 other computers at my workplace (all have different specs, but same OS - Windows 10 Pro 20H2).

It doesn't matter how many "rows" are there, how big are the files or how resource heavy is process being used - the batch processing freezes anyway.

[nyalldawson]

I've tried extensively to reproduce this with no luck.

If it's still reproducible in current QGIS versions, can someone please post a workflow which is able to demonstrate the issue?

[Mides]

If you select layers loaded in Qgis, the problem occurs. If you load layers from a file, the problem no longer exists.

[nicogodet]

@nyalldawson It's very hard (impossible?) to provide a sample project to reproduce... This deadlock is very random.
You can try Steps to reproduce listed here: #54124
I do not garante it will happen (and it might be a Windows issue too) but when I wrote #54124, I was able to reproduce it everytime.

[roya0045]

@nyalldawson It's very hard (impossible?) to provide a sample project to reproduce... This deadlock is very random. You can try Steps to reproduce listed here: #54124 I do not garante it will happen (and it might be a Windows issue too) but when I wrote #54124, I was able to reproduce it everytime.

It also seems to happen more commonly when there are a lot of items in the batch (40+) and will stop at a random point. Using gdal translate to change tiff to jpg in batch is also an easy case. Just split a tiff into many subfile (I get mine fron GEE though), then change their extension with gdal translate. With 120 small files I get a stall around 20 to 40 items done-> Trigger the batch cancel-> keep the dialog open-> remove the first elements that were processed -> run again -> repeat until all done.

[wikirikitiki]

Sorry for replying so late. I'm not a dev so I have no idea it this is what you wanted Nyall. Please let me know if there's anything else I can do.

I've uploaded two videos to a Google Drive folder here

Both videos show QGIS version 3.32.2 on a Windows 10 machine (the os is up to date).

First video is called workflow_1.mp4.
It shows batch processing of "Clip raster by mask layer". I'm using 15 rasters (small file sizes) and 15 polygons to mask each raster with corresponding polygon. The video shows that batch processing got stuck at random moment and I had to cancel the batch processing and rerun it. Windows task manager shows that QGIS is fine, only the batch processing got stuck. Fortunately I was able to rerun it successfully the second time (which happens very rarely).

Second video is called workflow_2.mp4.
It shows batch processing of "Clip". I'm clipping one geopackage containing a lot of points and clipping that one file using multiple polygons. The video shows batch processing getting stuck multiple times at random moments.

I've tried the same today on QGIS 3.32.3 but the problem still occurs. This problem does not appear on QGIS 3.24 and earlier.

[nyalldawson]

Unfortunately I still can't reproduce . I'll have to leave this to someone else to debug.

[GTP85]

Hello, I have exactly the same problem on various batch processing models, when I try to work on 70, 100, or 800 files.

Thanks for the solution to use QGIS 3.24 (3.24.3 for me), it works fine with that version!

QGIS 3.32.3 (but same problem with an older version).
Windows 10 Pro
Intel(R) Core(TM) i5-10310U CPU @ 1.70GHz 2.21 GHz

[nicogodet]

@nyalldawson It might be a windows only
@roya0045 it happens on Linux for you?

[GTP85]

If you select layers loaded in Qgis, the problem occurs. If you load layers from a file, the problem no longer exists.

For me, the problem occurs with files (not loaded as QGIS layers). I didn't tried with layers already loaded in QGIS.

[roya0045]

@nyalldawson It might be a windows only @roya0045 it happens on Linux for you?

windows, so I assume it's wondows blocking too many subprocess or something to that effect.

[nyalldawson]

Can someone please test a build including #54845 and report whether the issue is fixed?

[GTP85]

Can someone please test a build including #54845 and report whether the issue is fixed?

I have no idea how to work with builds, but if someone can send me a "how-to", I could test that.

[nicogodet]

Download this artifact
https://github.com/qgis/QGIS/suites/17011932137/artifacts/971564009

Extract twice and run qgis.exe in bin folder

[nicogodet]

@nyalldawson Same issue with #54845

And gdalwarp was not even run.

[baswein]

Thanks for trying @nyalldawson!
It is still a problem in todays QGIS 3.35.00 from osgeo4w. I find it is hit or miss I have been running a bunch of batch processing with 90 layers over the last couple of months. I think less than 10% of the time it gets all the way through. Often it stalls multiple times during the processing. This is of course very annoying as I can't leave thigs to process over night and have to keep checking to see if it is stalled out. Is there anyway I can log what is happening to try to get to the bottom of it?

[Eksploracja]

According to this StackExchange thread, it seems that the aforementioned issue does not occur in the Ubuntu system https://gis.stackexchange.com/questions/464309/qgis-batch-processing-stops-part-way-through-indefinitely-for-no-apparent-reason

[sivansakthi]

Hello @agiudiceandrea,

I have done many times re-projecting the layers. Please refer the attached recorded videos as zip file
2024-04-01-17-33-38.zip

Please have a look and let me know what to do.

Regards,
Sivansakthi

[effjot]

Happens with version 3.34.6 on Windows 10, too. A few layers are ok, but with more than ≈10, processing randomly gets stuck.